Building Cloudflare's Data Platform and AI Agent
Cloudflare faced a significant challenge in managing the vast amounts of data generated across its network, spanning 330 cities in 120 countries. With billions of events processed every second, the company needed a way to efficiently access and analyze this data. To address these complexities, Cloudflare developed two tools: Town Lake, a unified data analytics platform, and Skipper, an AI agent designed to simplify data queries. These innovations enable users to access information through a single SQL interface and receive auditable answers in plain English.
Challenges of Data Sprawl in Hypergrowth Companies
During hypergrowth periods, companies often face data sprawl, where information is dispersed across multiple systems. At Cloudflare, this issue manifested in the form of disparate systems including Postgres databases, ClickHouse clusters, Kafka streams, BigQuery datasets, Google Cloud buckets, and R2 logs. Each of these systems had unique credentials, query languages, and retention policies, creating barriers to efficient data access for various stakeholders.
For instance, a product engineer investigating a customer issue might need to navigate multiple systems with inconsistent access protocols. This complexity hindered timely data retrieval and analysis, which is critical for decision-making in dynamic environments. The lack of a unified system resulted in unnecessary delays and inefficiencies.
Impact of Sampled and Stale Data
Cloudflare's analytics pipeline was designed to downsample data to handle high volumes-up to 700 million events per second. While this approach optimized dashboard loading times, it was unsuitable for critical processes like billing, which demand accurate and complete data. Sampled data often led to discrepancies and unreliable insights, affecting operational efficiency and decision-making.
Additionally, stale data exacerbated the issue, as analysts struggled to determine whether the information they accessed was current or outdated. This lack of freshness in data further complicated efforts to derive actionable insights, making it imperative to develop a solution that could ensure data timeliness and precision.
Dependence on External Data Vendors
Another challenge was the reliance on external vendors for parts of the internal reporting stack. This dependency introduced additional costs and risks, as the functionality of Cloudflare's internal systems was tied to external providers. Such reliance could lead to service disruptions, especially if the external vendor experienced issues.
To mitigate these risks, Cloudflare recognized the need for an in-house solution that would eliminate external dependencies and give the company full control over its data infrastructure. This would not only reduce costs but also enhance reliability and security, ensuring the seamless operation of its systems.
Development of Town Lake: A Unified SQL Interface
Town Lake was designed as a single SQL interface to integrate all of Cloudflare's data sources. By consolidating data from various systems into a unified platform, Town Lake addressed the challenges of data sprawl and eliminated the need for users to navigate multiple systems. This solution streamlined the data retrieval process, enabling users to access comprehensive insights quickly and efficiently.
The platform also ensured data consistency and accuracy by maintaining real-time updates across all integrated systems. This feature was particularly beneficial for critical applications like billing, where precise data is essential for generating accurate invoices.
Introduction of Skipper: An AI Data Agent
To complement Town Lake, Cloudflare developed Skipper, an AI-powered data agent that allows users to query data in plain English. Skipper translates natural language queries into SQL commands, enabling non-technical users to access complex datasets without requiring specialized skills. This functionality has democratized data access within the organization, empowering employees across departments to make informed decisions.
Skipper also ensures the accuracy and auditability of the responses it generates. By leveraging the unified data platform provided by Town Lake, Skipper delivers reliable answers in seconds, enhancing the overall efficiency of data analytics at Cloudflare.
Future Implications of the Unified Data Platform
The integration of Town Lake and Skipper has transformed Cloudflare's approach to data analytics. By addressing the challenges of data sprawl, sampled data, and external dependencies, these tools have significantly improved the organization's ability to harness its vast data resources. This development highlights the importance of building scalable and efficient data platforms in today's data-driven world.
Looking ahead, the success of Town Lake and Skipper sets a precedent for other organizations facing similar challenges. By adopting a unified data platform and leveraging AI-driven tools, companies can overcome the limitations of fragmented systems and unlock new opportunities for growth and innovation.