Pay per crawl is a model in which a crawler pays for each page it fetches. The term comes from a Cloudflare feature of the same name. Site owners decide for each crawler whether it may fetch their content for free, for a fee, or not at all.
At a glance
- Type
- Payment model for crawler access (Cloudflare feature)
- Used in
- Cloudflare, closed beta
- Values
- Allow, Charge, Block (rule per crawler)
- Since
- Announced July 2025
Last checked:
How does pay per crawl work at Cloudflare?
Cloudflare announced the feature in July 2025. The documentation describes it as a closed beta (as of October 2026), and access runs through a sign-up or an account executive.
A crawler that wants to pay sends its payment intent in request headers. Cloudflare’s blog names crawler-max-price for a price limit the crawler is willing to pay and crawler-exact-price for agreeing to the listed price. Without a payment offer, the server answers with HTTP 402 (Payment Required) and a crawler-price header with the price. A completed charge is confirmed with the crawler-charged header.
For each crawler the site owner picks one of three rules.
- Allow: free access
- Charge: access only against payment
- Block: no access
The price is a flat per-request price, set per zone. Cloudflare acts as the merchant of record, collects the payments and pays out the site owners.
- The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file.Source: rfc-editor.org
- OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training.Source: openai.com
- Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs.Source: blog.google
- After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended.Source: developers.google.com
- Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended.Source: machinelearning.apple.com
- Cloudflare lets customers block AI bots, scrapers and crawlers with a single click.Source: blog.cloudflare.com
- With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training.Source: openai.com
- Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents.Source: developers.google.com
- Based on publisher feedback, Google clarifies the description of the Google-Extended token.Source: developers.google.com
- Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421).Source: blog.cloudflare.com
- Cloudflare changes its default to block AI crawlers unless site owners decide otherwise.Source: blog.cloudflare.com
- Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price.Source: blog.cloudflare.com
OpenAI only says “last summer” for its crawler permissions (post of May 7, 2024); the often-cited August 2023 date for GPTBot comes from press reports. ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, OAI-SearchBot, ChatGPT-User, Meta-ExternalAgent and CCBot are documented by their operators, but without a verifiable launch date. Last checked: .
How are crawlers verified?
To stop other bots from posing as a paying crawler, Cloudflare uses Web Bot Auth. Crawlers register with Cloudflare, sign their requests with an Ed25519 key and send the headers signature-agent, signature-input and signature.
Which rules come first?
According to Cloudflare, WAF and Bot Management rules override pay per crawl. A crawler blocked there gets no access, even when it offers to pay.
Common mistakes
- Treating pay per crawl as protection against blocked bots. According to Cloudflare, WAF and Bot Management rules take precedence. A crawler blocked there has no access to the zone, even if it would pay.
- Expecting immediate availability. The documentation describes the feature as a closed beta. Access runs through a sign-up or an account executive for Enterprise customers.
Sources
- What is pay per crawl? – Cloudflare AI Crawl Control docsdevelopers.cloudflare.com
- Introducing pay per crawl – Cloudflare Blogblog.cloudflare.com
- Pay Per Use: when AI uses your work, you should get paid – Cloudflare Blog (Sep 30, 2026)blog.cloudflare.com
Last updated: