robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may request. It controls crawling. Whether a URL ends up in a search index is decided by other signals, such as noindex. The format was standardized as the Robots Exclusion Protocol in RFC 9309 in September 2022.
At a glance
- Type
- Plain text file at the root of the host (
/robots.txt) - Affects
- Crawling, not indexing
- Values
- At Google:
user-agent,allow,disallow,sitemap - Specification
- RFC 9309 (IETF, September 2022)
Last checked:
How does robots.txt work?
Before fetching pages, a compliant crawler requests /robots.txt from the host. The file applies only to that exact combination of protocol, host and port. Rules in https://shop.example.com/robots.txt have no effect on https://www.example.com.
A file consists of groups. Each group starts with one or more user-agent lines and lists allow and disallow rules for those crawlers. Google supports four fields: user-agent, allow, disallow and sitemap. It does not support crawl-delay. When rules conflict, Google applies the most specific one, meaning the rule with the longer path. If two rules are equally specific, Google uses the less restrictive one. Path values are case-sensitive.
According to Google’s documentation, Google generally caches robots.txt for up to 24 hours, ignores content beyond 500 KiB, and treats a 4xx response (except 429) as if no robots.txt existed.
| File | Standard or proposal? | Location | Purpose | Who reads it (documented) |
|---|---|---|---|---|
| robots.txt (this term) | IETF standard: RFC 9309 (September 2022) | / | tells crawlers which URLs they may fetch; per the RFC not a form of access authorization | Crawlers; Google's common crawlers always respect it for automatic crawls |
| llms.txt | Proposal (Jeremy Howard, 2024; version 2 from August 2026) | / or in a subpath | Markdown overview linking to Markdown versions of pages, for AI agents | per the proposal mainly coding agents; Google Search ignores it, says Google; Lighthouse only checks the fetch |
| llms-full.txt | no specification; not part of the llms.txt proposal, a convention of individual vendors (e.g. Mintlify) | / (Mintlify also serves /.) | entire documentation as Markdown in one file | no reader documented; Google Search doesn't use such files, says Google |
| ai.txt | Individual IETF Internet-Draft (June 2026, expires 2026-12-14), not a standard; predecessor by Spawning (2023) | /. (draft), / (Spawning) | AI usage preferences (training, scraping, indexing, caching), licensing, attribution | no reader documented in primary sources; Google Search doesn't use 'AI text files', says Google |
TDMRep (tdmrep.) | W3C Community Group Final Report (2024-05-10), not a W3C standard | /., HTTP header tdm-reservation or HTML meta tag | machine-readable reservation of text and data mining rights (Art. 4 DSM Directive) | parties performing text and data mining (per the specification) |
| security.txt | RFC 9116 (April 2022, informational, not standards track) | /. | contact for reporting security vulnerabilities; required fields Contact and Expires | Security researchers |
| ads.txt | IAB Tech Lab specification, version 1.1 (August 2022) | / on the root domain, subdomains as needed | declares the companies authorized to sell the site's ad inventory | advertising systems and buyers (ad system crawlers) |
| humans.txt | Initiative (humanstxt.org), no formal specification | / next to robots.txt, optionally linked via rel="author" | names the people who contributed to the website | people who want to know who is behind the website |
Only robots.txt, security.txt and ads.txt are based on a finalised specification (IETF or IAB Tech Lab). Google states that Google Search doesn't use new machine-readable files or 'AI text files' and ignores llms.txt files, which neither help nor harm Search visibility. TDMRep is a technical way to express the reservation under Art. 4 of Directive (EU) 2019/790. Not legal advice. Sources: RFC 9309: Robots Exclusion Protocol; Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); The /llms.txt file, v2 (llmstxt.org); llmstxt.org: Changes (v2, August 2026); Google Search Central: Optimizing for generative AI features (last updated 2026-07-10); Chrome for Developers: Lighthouse llms.txt audit (2026-05-05); Mintlify: llms.txt and llms-full.txt; IETF Internet-Draft draft-car-ai-txt-wellknown-00 (2026-06-12); W3C TDMRep Community Group: TDM Reservation Protocol, Final Report 2024-05-10; RFC 9116: A File Format to Aid in Security Vulnerability Disclosure; IAB Tech Lab: ads.txt; IAB Tech Lab: ads.txt Specification Version 1.1 (GitHub); humanstxt.org. Last checked: .
Example
User-agent: *
Disallow: /cart/
Disallow: /*?sessionid=
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
All crawlers are asked to skip the cart and any URL with a session parameter. GPTBot is asked to skip the entire site. The last line points to the XML sitemap.
Common mistakes
- Using Disallow to remove pages from Google. Google says a disallowed URL can still be indexed if other pages link to it. To keep a page out of the index, use
noindexor password protection, and leave the page crawlable so thenoindexcan be read. - Putting
noindexin robots.txt. Google stopped honoring this unofficial rule on September 1, 2019. - Treating robots.txt as access control. RFC 9309 states the rules are not a form of authorization. The file is public and can even reveal the paths it blocks.
- Shipping
Disallow: /from a staging environment to the live site during a relaunch. This is a common practice case, not a statement from Google’s documentation.
| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |
robots.txt and noindex cancel each other out. If a URL is blocked by robots.txt, Google never sees the noindex, and the URL can still appear in results. Google doesn't support noindex inside the robots.txt file itself. Sources: Google Search Central: Introduction to robots.txt (last updated 2025-12-10); Google Search Central: Block Search indexing with noindex (last updated 2025-12-10); Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24); Google Search Central: How to specify a canonical URL (last updated 2026-07-10); Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04); Google Search Central: Remove information from Google (last updated 2025-12-10); Search Console Help: Removals and SafeSearch reports tool. Last checked: .
FAQ
Where does robots.txt have to be located?
At the root of the host, reachable at /robots.txt. Crawlers do not look for it in subdirectories.
Do I need a robots.txt file?
No. If the server returns 404 for /robots.txt, Google’s crawlers assume there are no restrictions.
Is robots.txt still relevant with AI crawlers?
Yes. OpenAI, for example, documents the user-agent token GPTBot and states that disallowing it signals that a site’s content should not be used to train its foundation models. Following the rules is still up to each crawler.
More on the searchVIU blog
Sources
- Introduction to robots.txt – Google Search Centraldevelopers.google.com
- How Google interprets the robots.txt specification – Google Search Centraldevelopers.google.com
- RFC 9309: Robots Exclusion Protocolrfc-editor.org
- A note on unsupported rules in robots.txt – Google Search Central Blog, 2019developers.google.com
- Overview of OpenAI Crawlers – OpenAIdevelopers.openai.com
Last updated: