Context & History

The UK Competition and Markets Authority (CMA) opened a consultation in early 2026 on conduct requirements that target Google’s use of web content for its generative AI features. This follows the Digital Markets, Competition and Consumers Act 2024 (DMCA) that gave the CMA power to designate firms with Strategic Market Status (SMS) and impose detailed rules. In October 2025, Google was labeled an SMS holder for general search and search advertising, covering its AI Overviews and AI Mode. The consultation aims to give publishers tools to control how their content is used by AI crawlers, a move that many see as a first step toward a more level playing field.

Implementation & Best Practices

To respond effectively, publishers should follow a three‑phase roadmap: first, audit current bot traffic and identify which crawlers are accessing sensitive content second, apply technical controls such as updated robots.txt directives, Web Application Firewall (WAF) rules, and rate‑limiting policies third, engage with legal teams to align site policies with the CMA’s upcoming conduct requirements and negotiate fair data‑use agreements where possible.

Technical Controls for Crawler Separation

While robots.txt remains the simplest way to express crawling preferences, it relies on voluntary compliance. For stronger enforcement, publishers can configure a WAF to block or challenge unwanted AI bots. The web interoperability guide provides patterns for handling bot detection at the edge, and the rate limiting techniques article shows how to throttle request rates for non‑human agents without impacting real users. Combining these methods creates a layered defence that reduces the chance of Googlebot’s content being repurposed without consent.

Legal and Policy Measures

Publishers should review the CMA’s draft rules alongside the Digital Markets, Competition and Consumers Act 2024 to understand their rights and obligations. Updating privacy policies to explicitly state the conditions under which content may be used for AI training can provide a contractual basis for enforcement. In jurisdictions where the CMA’s conduct requirements become law, non‑compliance could lead to significant penalties, making proactive policy alignment essential.

Monitoring and Enforcement

Continuous monitoring is key. Tools that log bot signatures, request patterns, and content usage help verify whether a crawler respects the defined rules. Cloudflare’s AI Crawl Control data, for example, shows that Googlebot accesses far more URLs than competing bots, highlighting the need for ongoing vigilance. Publishing regular audit reports and sharing findings with industry groups can also pressure dominant platforms to adopt fairer practices.

Key takeaway: Separating AI crawling from search indexing protects publisher revenue, promotes competition, and aligns with emerging regulatory expectations.