Definition
When an AI agent encounters an HTTP error, the response format determines how much data the model must process. Traditional HTML error pages contain thousands of characters designed for human readers, forcing agents to parse unnecessary markup, styles, and prose. RFC 9457‑compliant responses replace that noise with a concise, machine‑readable contract expressed in markdown or JSON, allowing the agent to act on the error instantly while preserving precious tokens.
Why traditional HTML error pages waste tokens
HTML error pages embed full document structures: <head>, <style>, and explanatory text meant for a user staring at a browser. An AI model that receives such a payload must tokenize every tag, attribute, and line break before it can locate the actionable information. The token count can easily exceed a thousand for a single error, dramatically inflating the cost of each request.
Beyond token volume, the content of HTML pages is ambiguous for automated consumption. Phrases like You have been blocked lack explicit guidance on whether a retry is possible or if a different endpoint should be used. The model must infer intent from natural language, a process that introduces latency and potential misinterpretation.
When an agent makes dozens of calls within a workflow, repeated encounters with heavy HTML responses compound the cost. In high‑throughput scenarios-such as crawling large catalogs or orchestrating multi‑step transactions-the token overhead becomes a financial bottleneck.
Replacing HTML with a focused data contract removes the need for the model to discard irrelevant markup. The result is a direct, low‑overhead communication channel that aligns with the token‑budget constraints of modern language models.
The RFC 9457 specification and its relevance
RFC 9457, known as Problem Details for HTTP APIs, defines a JSON structure for describing errors in a uniform way. The format includes members such as type, title, status, and detail, with the ability to add custom fields for domain‑specific information. By adhering to this specification, a service ensures that any client capable of parsing JSON can understand the failure without custom logic.
Because the spec is intentionally minimal, extensions like error_code, retryable, and retry_after can be added safely. Clients that do not recognize these extensions simply ignore them, preserving backward compatibility while still gaining access to richer guidance when available.
Adopting RFC 9457 also aligns with best practices for API design, where error handling is treated as a first‑class citizen. The clear contract encourages developers to build deterministic retry loops, back‑off strategies, and escalation paths directly into their agents.
For readers unfamiliar with the specification, the Wikipedia entry provides a concise overview: Problem Details for HTTP APIs. Cloudflares implementation builds on this foundation, extending it with fields that reflect edge‑level policies.
How Cloudflare implements markdown and JSON contracts
Cloudflare inspects the Accept header of each request. When the header indicates a preference for text/markdown, application/json, or application/problem+json, the edge returns a payload that matches the requested media type. The underlying data follows the RFC 9457 shape, with additional keys that describe Cloudflare‑specific conditions.
The markdown variant renders the same fields in a human‑readable but still structured format. Each line begins with a YAML front‑matter block, allowing parsers to extract keys before the narrative text. This approach satisfies both model‑first pipelines that expect plain text and traditional logging tools that benefit from a readable summary.
In the JSON response, the type member points to Cloudflares documentation for the specific error code, while custom members such as owner_action_required signal whether the agent should halt or continue. The schema is deliberately stable, preventing breaking changes for downstream consumers.
Site owners need not make any configuration changes for this behavior to activate. The edge automatically serves the appropriate contract when an agent requests it, while browsers continue to receive the legacy HTML page unless they explicitly ask for a different format.
Related internal guidance can be found in the Cloudflare active‑defense scanner article and the SASE migration guide, both of which discuss how edge services can expose machine‑readable data to downstream systems.
Practical impact on token consumption
Empirical measurements on a 1015 rate‑limit error show that the HTML version contains roughly 1,200 characters, translating to more than 1,000 tokens for a typical language model. The markdown representation drops to under 30 characters, while the JSON payload sits around 45 characters. This reduction equates to a token savings of more than 98 percent per error encounter.
When an agent processes a chain of ten requests, each triggering a rate‑limit response, the cumulative token cost drops from over 10,000 tokens to fewer than 200. For high‑frequency agents that issue thousands of calls per minute, the financial implications become substantial, especially when operating under pay‑per‑token pricing models.
Beyond raw token count, the reduced payload speeds up parsing time. JSON parsers can deserialize a small object in microseconds, whereas HTML parsers must first strip tags, handle nested structures, and then locate the relevant text. The latency improvement further enhances the overall throughput of the agent.
The combination of lower token usage and faster parsing creates a feedback loop: agents can make more requests within the same cost envelope, enabling richer data collection and more sophisticated workflows without exceeding budget constraints.
Integration patterns for AI agents
Developers should adjust their HTTP client libraries to include an Accept header that prefers machine‑readable formats. A typical header might be Accept: application/problem+json, application/json, text/markdown. The client then branches based on the Content-Type of the response, handling JSON directly and treating markdown as a simple text block that can be split on line breaks.
When the response contains retryable: true and a retry_after value, the agent can implement an exponential back‑off algorithm that respects the server‑suggested wait time. If owner_action_required is true, the agent should cease automatic retries and raise an alert for human intervention.
Logging should capture the ray_id, timestamp, and zone fields, enabling correlation with Cloudflare support tickets if needed. Because these identifiers are stable across formats, the same logging infrastructure can handle both HTML and structured responses without modification.
For agents that operate in mixed environments-some services returning traditional HTML and others providing RFC 9457 payloads-a detection layer can examine the Content-Type header and delegate to the appropriate parser. This design keeps the codebase clean and avoids duplicated parsing logic.
Operational considerations and monitoring
While the token savings are compelling, teams should still monitor error rates to ensure that the underlying policies are not causing excessive blockages. Alerting on a surge of error_category: rate_limit entries can prompt a review of request patterns before they degrade user experience.
Metrics such as average retry_after duration and the proportion of responses marked owner_action_required provide insight into how often agents must defer to human operators. Tracking these figures over time helps refine throttling strategies and informs capacity planning.
Because the structured payload includes a stable type URI, automated systems can fetch the linked documentation to display contextual help to operators. This feature reduces the need for manual lookup and accelerates incident resolution.
Security teams should verify that the exposure of error details does not inadvertently reveal sensitive configuration. The RFC 9457 format intentionally limits information to what is needed for remediation, but organizations may still wish to mask fields like zone in public logs.
Future extensions and best practices
Cloudflare plans to extend the same contract to 4xx and 5xx edge errors, bringing uniformity across the entire error spectrum. Early adopters can prepare by designing their agents to handle unknown error_category values gracefully, defaulting to a safe halt when faced with unexpected conditions.
Best practice recommendations include: always sending an explicit Accept header, caching the type URI for quick reference, and implementing retry logic that respects the retry_after field. Agents should also record the full payload for audit trails, ensuring that any escalation to site owners contains the exact context of the failure.
By treating error responses as actionable instructions rather than decorative pages, developers can dramatically cut costs, improve reliability, and create a more predictable interaction model between AI agents and the web. The shift from HTML to structured contracts represents a practical evolution in how edge platforms communicate with autonomous systems.
Adopting RFC 9457‑compliant responses aligns with broader industry movements toward standardized API error handling, positioning organizations to benefit from ecosystem tools that already understand the format. As more services adopt the same approach, agents will be able to operate across diverse platforms with a single, consistent error‑parsing strategy.