# Pay per crawl Category: [AI Bots & Crawlers](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/) › [Managing & measuring AI bots](https://seo-geo-glossary.com/en/glossary/category/ai-bots-crawlers/managing-measuring-ai-bots/) **Pay per crawl** is a model in which a crawler pays for each page it fetches. The term comes from a Cloudflare feature of the same name. Site owners decide for each crawler whether it may fetch their content for free, for a fee, or not at all. ## How does pay per crawl work at Cloudflare? Cloudflare announced the feature in July 2025. The documentation describes it as a closed beta (as of October 2026), and access runs through a sign-up or an account executive. A crawler that wants to pay sends its payment intent in request headers. Cloudflare's blog names `crawler-max-price` for a price limit the crawler is willing to pay and `crawler-exact-price` for agreeing to the listed price. Without a payment offer, the server answers with HTTP 402 (Payment Required) and a `crawler-price` header with the price. A completed charge is confirmed with the `crawler-charged` header. For each crawler the site owner picks one of three rules. - **Allow:** free access - **Charge:** access only against payment - **Block:** no access The price is a flat per-request price, set per zone. Cloudflare acts as the merchant of record, collects the payments and pays out the site owners. ## How are crawlers verified? To stop other bots from posing as a paying crawler, Cloudflare uses Web Bot Auth. Crawlers register with Cloudflare, sign their requests with an Ed25519 key and send the headers `signature-agent`, `signature-input` and `signature`. ## Which rules come first? According to Cloudflare, WAF and Bot Management rules override pay per crawl. A crawler blocked there gets no access, even when it offers to pay. ## Common mistakes - **Treating pay per crawl as protection against blocked bots.** According to Cloudflare, WAF and Bot Management rules take precedence. A crawler blocked there has no access to the zone, even if it would pay. - **Expecting immediate availability.** The documentation describes the feature as a closed beta. Access runs through a sign-up or an account executive for Enterprise customers. ## At a glance - **Type:** Payment model for crawler access (Cloudflare feature) - **Used in:** Cloudflare, closed beta - **Values:** Allow, Charge, Block (rule per crawler) - **Since:** Announced July 2025 - **Docs:** [Cloudflare: What is pay per crawl?](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) Last checked: October 8, 2026 ## AI bots: crawlers, tokens and controls - **September 2022:** The IETF publishes the Robots Exclusion Protocol as RFC 9309 on the Standards Track. It defines how crawlers interpret the rules in a robots.txt file. ([Source](https://www.rfc-editor.org/rfc/rfc9309.html)) - **2023:** OpenAI introduces web crawler permissions for AI. Site owners can use them to state whether their content may be used for AI training. ([Source](https://openai.com/index/approach-to-data-and-ai/)) - **September 28, 2023:** Google introduces Google-Extended. This robots.txt token controls whether content helps improve Bard and the Vertex AI generative APIs. ([Source](https://blog.google/technology/ai/an-update-on-web-publisher-controls/)) - **February 8, 2024:** After Bard is renamed, Google's documentation lists Gemini Apps as a product affected by Google-Extended. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token_1)) - **June 10, 2024:** Apple introduces the foundation models behind Apple Intelligence. Publishers can opt out of training use, according to Apple's documentation via Applebot-Extended. ([Source](https://machinelearning.apple.com/research/introducing-apple-foundation-models)) - **July 3, 2024:** Cloudflare lets customers block AI bots, scrapers and crawlers with a single click. ([Source](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/)) - **July 25, 2024:** With the SearchGPT prototype, OpenAI separates search from training. Sites can appear in search results even if they opt out of AI training. ([Source](https://openai.com/index/searchgpt-prototype/)) - **August 20, 2024:** Google adds Google-CloudVertexBot to its list of crawlers. It crawls sites at the owner's request to build Vertex AI Agents. ([Source](https://developers.google.com/search/updates#introducing-the-google-cloudvertexbot-crawler)) - **April 25, 2025:** Based on publisher feedback, Google clarifies the description of the Google-Extended token. ([Source](https://developers.google.com/search/updates#updated-the-description-of-the-google-extended-product-token)) - **May 15, 2025:** Cloudflare proposes that bots and agents cryptographically sign their requests, based on HTTP Message Signatures (RFC 9421). ([Source](https://blog.cloudflare.com/web-bot-auth/)) - **July 1, 2025:** Cloudflare changes its default to block AI crawlers unless site owners decide otherwise. ([Source](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/)) - **July 1, 2025:** Cloudflare launches pay per crawl as a private beta. Crawlers either pay per request or receive a 402 status code with a price. ([Source](https://blog.cloudflare.com/introducing-pay-per-crawl/)) ## Related terms - [CCBot](https://seo-geo-glossary.com/en/glossary/ccbot/index.md) - [Crawl-to-refer ratio](https://seo-geo-glossary.com/en/glossary/crawl-to-refer-ratio/index.md) - [Crawl budget](https://seo-geo-glossary.com/en/glossary/crawl-budget/index.md) - [Web Bot Auth](https://seo-geo-glossary.com/en/glossary/web-bot-auth/index.md) - [Blocking AI bots](https://seo-geo-glossary.com/en/glossary/blocking-ai-bots/index.md) ## Sources - [What is pay per crawl? – Cloudflare AI Crawl Control docs](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) - [Introducing pay per crawl – Cloudflare Blog](https://blog.cloudflare.com/introducing-pay-per-crawl/) - [Pay Per Use: when AI uses your work, you should get paid – Cloudflare Blog (Sep 30, 2026)](https://blog.cloudflare.com/pay-per-use/) --- Last updated: October 8, 2026 URL: https://seo-geo-glossary.com/en/glossary/pay-per-crawl/