The Robots Exclusion Protocol (REP) is the standard that lets site owners tell crawlers, through a robots.txt file, which parts of a site they may fetch. Martijn Koster defined the method in 1994. The IETF published it as RFC 9309 in September 2022.
What does the protocol define?
The file is named /robots.txt, sits in the top-level path of the host, is UTF-8 encoded and served as text/plain. It consists of groups. Each group starts with user-agent lines followed by allow and disallow rules. If several rules match, the longest match wins. Crawlers must support # for comments, * as a wildcard and $ for the end of a path.
RFC 9309 also defines how to treat the fetch itself. A 4xx response means the crawler may access any resource. If the file is unreachable because of server or network errors (5xx), the crawler must assume a complete disallow. Crawlers should follow at least five consecutive redirects and should not use a cached copy for more than 24 hours. The parsing limit must be at least 500 KiB.
The RFC states that these rules are not a form of access authorization. Crawlers are requested to honor them, and restricting access takes authentication.
RFC 9309 covers only robots.txt. The robots meta tag and the X-Robots-Tag header control indexing and snippets for individual pages, and Google documents them separately. Whether they count as part of the REP in a wider sense varies between sources (assessment).
| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |
robots.txt and noindex cancel each other out. If a URL is blocked by robots.txt, Google never sees the noindex, and the URL can still appear in results. Google doesn't support noindex inside the robots.txt file itself. Sources: Google Search Central: Introduction to robots.txt (last updated 2025-12-10); Google Search Central: Block Search indexing with noindex (last updated 2025-12-10); Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24); Google Search Central: How to specify a canonical URL (last updated 2026-07-10); Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04); Google Search Central: Remove information from Google (last updated 2025-12-10); Search Console Help: Removals and SafeSearch reports tool. Last checked: .
Example
User-agent: *
Disallow: /internal/
Allow: /internal/press/
FAQ
Is robots.txt still used? Yes. Google and other search engines still read it, and it has been an IETF standard since 2022.
Sources
- RFC 9309: Robots Exclusion Protocol – IETF (09/2022)rfc-editor.org
- robots.txt specification – Google Search Centraldevelopers.google.com
- About /robots.txt – robotstxt.orgrobotstxt.org
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications – Google Search Centraldevelopers.google.com
Last updated: