Internal AI Engineering Stack: Analysis and Implementation Insights
Cloudflares internally-developed AI engineering stack integrates advanced tools and infrastructure to enhance developer productivity. Over eleven months, the organization built MCP servers, AI gateways, and agentic coding tools, resulting in a significant increase in developer velocity. This article examines the architecture, adoption metrics, and operational improvements achieved through this project.
Overview of the AI Engineering Stack
Cloudflares AI engineering stack combines multiple components, including MCP servers, access layers, and AI tooling. These elements were designed to integrate seamlessly with the companys existing development workflows. Tools such as OpenCode and Windsurf, compatible with the stack, play a critical role in enabling engineers to optimize coding processes.
The system operates on a foundation of Cloudflares Zero Trust authentication, ensuring secure access to internal resources. Additionally, centralized LLM (large language model) routing and cost tracking mechanisms were implemented for efficiency and transparency. These features are crucial for managing the high volume of requests processed daily.
Key Metrics Highlighting AI Adoption
The adoption of agentic AI tools has been substantial across Cloudflares workforce. In the past 30 days, 93% of the companys research and development team actively utilized these tools, amounting to 3,683 internal users. This represents 60% of the total employee base of approximately 6,100 individuals.
Key metrics include 4.795 million AI requests and 20.18 million AI gateway requests per month. Additionally, the stack routed 24.137 billion tokens through the AI gateway and processed 51.83 billion tokens on the Workers AI platform, demonstrating the scalability and efficiency of the system.
Impact on Developer Velocity
The AI tools significantly improved developer velocity, as evidenced by a sharp increase in weekly merge requests. The four-week rolling average climbed from 5,600 to over 8,700, peaking at 10,952 during the week of March 23. This represents nearly double the baseline observed in the previous quarter.
These improvements are attributed to the integration of AI coding assistants, which streamline code reviews, onboarding processes, and the propagation of changes across multiple repositories. This approach has enabled teams to deliver higher volumes of high-quality code within shorter timeframes.
Role of the Dev Productivity Team
The success of the AI engineering stack is largely due to the efforts of the Dev Productivity team. This team, which manages internal tooling such as CI/CD build systems and automation, played a pivotal role in deploying and maintaining the stack. Their work ensured that the AI tools were both functional and accessible to all relevant stakeholders.
Beyond deployment, the team also focused on rethinking standards for coding, reviewing, and collaboration. These changes were essential to fully leveraging the capabilities of the AI tools, resulting in more cohesive and efficient workflows.
Architecture and Tooling Layers
The AI engineering stack is structured into distinct layers, each corresponding to a specific function or tool. The engineer-facing tools layer includes both open-source and third-party coding assistants that integrate with Cloudflares internal systems. These tools are MCP-compatible, ensuring interoperability and ease of use.
At its core, the stack is powered by Cloudflares proprietary products, such as the AI Gateway and Workers AI. These components handle token routing, processing, and authentication, forming the backbone of the system. This architecture allows for scalability and adaptability as the organizations needs evolve.
Future Prospects and Continuous Improvements
Cloudflares AI engineering stack is not static it continues to evolve with ongoing enhancements. The company plans to refine its tooling and infrastructure further, focusing on areas such as cost efficiency, user experience, and performance optimization. These efforts aim to sustain the momentum achieved through this initiative.
By continuously adapting the stack to meet emerging challenges, Cloudflare ensures that its AI tools remain relevant and effective. This commitment to innovation underscores the companys dedication to maintaining a competitive edge in the tech industry.