Web Bot Auth is a method for automated clients such as crawlers and AI agents to identify themselves to websites with a cryptographic signature in the HTTP request. A site can then recognize requests from a known operator without relying on user agent names or IP lists alone.
At a glance
- Publisher
- IETF working group webbotauth (authors at Cloudflare and Google)
- Connects
- Bots and AI agents with websites
- Status
- Internet-Draft, Standards Track intended, not yet an RFC
- Version
- draft-ietf-webbotauth-httpsig-protocol-00 (2026-09-01)
- Specification
- IETF draft, builds on RFC 9421
Last checked:
What is the goal?
The IETF working group webbotauth works on standard methods for authenticating automated clients. Its charter names search crawlers, web archivers, link checkers, AI training crawlers and agents acting for users as in scope. It points out that origins want to manage access to their resources and that both bots and origins want protection against impersonation.
How does it work at Cloudflare?
Cloudflare describes four steps for bot operators.
- Generate an Ed25519 key pair and convert the public key to JWK format.
- Publish a signed key directory over HTTPS at
/.well-known/http-message-signatures-directory. - Register the bot with Cloudflare.
- Sign every request and send three headers.
| Header | Content |
|---|---|
Signature-Agent |
HTTPS address of the key directory |
Signature-Input |
Metadata such as tag="web-bot-auth", key ID, creation and expiry time |
Signature |
The signature itself |
The signing builds on HTTP Message Signatures (RFC 9421). Cloudflare supports only part of its components.
Who uses it?
Google tests Web Bot Auth and labels it experimental. It signs a subset of requests from its Google-Agent user agent and still recommends IP-based verification as a fallback. Cloudflare uses Web Bot Auth to verify crawlers in pay per crawl.
What is the status of the specification?
Cloudflare says the approach relies on IETF drafts, and Google calls it a draft specification. As of October 2026, the working group has not published an RFC. Details can therefore still change.
- The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file.Source: rfc-editor.org
- OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training.Source: openai.com
- Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs.Source: blog.google
- After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended.Source: developers.google.com
- Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended.Source: machinelearning.apple.com
- Cloudflare lets customers block AI bots, scrapers and crawlers with a single click.Source: blog.cloudflare.com
- With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training.Source: openai.com
- Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents.Source: developers.google.com
- Based on publisher feedback, Google clarifies the description of the Google-Extended token.Source: developers.google.com
- Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421).Source: blog.cloudflare.com
- Cloudflare changes its default to block AI crawlers unless site owners decide otherwise.Source: blog.cloudflare.com
- Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price.Source: blog.cloudflare.com
OpenAI only says “last summer” for its crawler permissions (post of May 7, 2024); the often-cited August 2023 date for GPTBot comes from press reports. ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent and CCBot are documented by their operators, but without a verifiable launch date. Last checked: .
Common mistakes
- Sending
Signature-Agentwithout quotes. Cloudflare requires the value as a structured string in double quotes, for example"https://example.com". - Signing unsupported parameters or components. According to Cloudflare, verification fails with the parameters
sf,bs,key,reqandname. Cloudflare does not support the components@query-paramsand@status. - Leaving the signature valid for too long. Cloudflare recommends short intervals to protect against replay attacks, and says a minute is often sufficient. The expiry must not pass before the request arrives.
- Using non-ASCII characters in signed components. Cloudflare lists them as a common cause of verification problems.
Sources
- Web Bot Auth – Cloudflare Docsdevelopers.cloudflare.com
- Web Bot Auth (webbotauth) Working Group – IETF Datatrackerdatatracker.ietf.org
- Authenticate requests with Web Bot Auth (experimental) – Google Crawling Infrastructure docsdevelopers.google.com
- RFC 9421: HTTP Message Signatures – IETFrfc-editor.org
- Web Bot Auth – Cloudflare Docs (Signing, Fehlerbehebung)developers.cloudflare.com
Last updated: