# robots.txt

Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [robots.txt & robots directives](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/robots-txt-directives/) · Also known as: Robots Exclusion Protocol, robots exclusion standard

**robots.txt** is a plain text file at the root of a host that tells crawlers which URLs they may request. It controls crawling. Whether a URL ends up in a search index is decided by other signals, such as `noindex`. The format was standardized as the Robots Exclusion Protocol in RFC 9309 in September 2022.

## How does robots.txt work?

Before fetching pages, a compliant crawler requests `/robots.txt` from the host. The file applies only to that exact combination of protocol, host and port. Rules in `https://shop.example.com/robots.txt` have no effect on `https://www.example.com`.

A file consists of groups. Each group starts with one or more `user-agent` lines and lists `allow` and `disallow` rules for those crawlers. Google supports four fields: `user-agent`, `allow`, `disallow` and `sitemap`. It does not support `crawl-delay`. When rules conflict, Google applies the most specific one, meaning the rule with the longer path. If two rules are equally specific, Google uses the less restrictive one. Path values are case-sensitive.

According to Google's documentation, Google generally caches robots.txt for up to 24 hours, ignores content beyond 500 KiB, and treats a 4xx response (except 429) as if no robots.txt existed.

## Example

```robots
User-agent: *
Disallow: /cart/
Disallow: /*?sessionid=

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml
```

All crawlers are asked to skip the cart and any URL with a session parameter. GPTBot is asked to skip the entire site. The last line points to the XML sitemap.

## Common mistakes

- **Using Disallow to remove pages from Google.** Google says a disallowed URL can still be indexed if other pages link to it. To keep a page out of the index, use `noindex` or password protection, and leave the page crawlable so the `noindex` can be read.
- **Putting `noindex` in robots.txt.** Google stopped honoring this unofficial rule on September 1, 2019.
- **Treating robots.txt as access control.** RFC 9309 states the rules are not a form of authorization. The file is public and can even reveal the paths it blocks.
- **Shipping `Disallow: /` from a staging environment** to the live site during a relaunch. This is a common practice case, not a statement from Google's documentation.

## FAQ

**Where does robots.txt have to be located?**
At the root of the host, reachable at `/robots.txt`. Crawlers do not look for it in subdirectories.

**Do I need a robots.txt file?**
No. If the server returns 404 for `/robots.txt`, Google's crawlers assume there are no restrictions.

**Is robots.txt still relevant with AI crawlers?**
Yes. OpenAI, for example, documents the user-agent token `GPTBot` and states that disallowing it signals that a site's content should not be used to train its foundation models. Following the rules is still up to each crawler.

## At a glance

- **Type:** Plain text file at the root of the host (/robots.txt)
- **Affects:** Crawling, not indexing
- **Values:** At Google: user-agent, allow, disallow, sitemap
- **Specification:** [RFC 9309 (IETF, September 2022)](https://www.rfc-editor.org/rfc/rfc9309.html)
- **Docs:** [Google: Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro)

Last checked: October 8, 2026

## Machine-readable site files compared

| File | Standard or proposal? | Location | Purpose | Who reads it (documented) |
|---|---|---|---|---|
| robots.txt | IETF standard: RFC 9309 (September 2022) | /robots.txt | tells crawlers which URLs they may fetch; per the RFC not a form of access authorization | Crawlers; Google's common crawlers always respect it for automatic crawls |
| llms.txt | Proposal (Jeremy Howard, 2024; version 2 from August 2026) | /llms.txt or in a subpath | Markdown overview linking to Markdown versions of pages, for AI agents | per the proposal mainly coding agents; Google Search ignores it, says Google; Lighthouse only checks the fetch |
| llms-full.txt | no specification; not part of the llms.txt proposal, a convention of individual vendors (e.g. Mintlify) | /llms-full.txt (Mintlify also serves /.well-known/llms-full.txt) | entire documentation as Markdown in one file | no reader documented; Google Search doesn't use such files, says Google |
| ai.txt | Individual IETF Internet-Draft (June 2026, expires 2026-12-14), not a standard; predecessor by Spawning (2023) | /.well-known/ai.txt (draft), /ai.txt (Spawning) | AI usage preferences (training, scraping, indexing, caching), licensing, attribution | no reader documented in primary sources; Google Search doesn't use 'AI text files', says Google |
| TDMRep (tdmrep.json) | W3C Community Group Final Report (2024-05-10), not a W3C standard | /.well-known/tdmrep.json, HTTP header tdm-reservation or HTML meta tag | machine-readable reservation of text and data mining rights (Art. 4 DSM Directive) | parties performing text and data mining (per the specification) |
| security.txt | RFC 9116 (April 2022, informational, not standards track) | /.well-known/security.txt | contact for reporting security vulnerabilities; required fields Contact and Expires | Security researchers |
| ads.txt | IAB Tech Lab specification, version 1.1 (August 2022) | /ads.txt on the root domain, subdomains as needed | declares the companies authorized to sell the site's ad inventory | advertising systems and buyers (ad system crawlers) |
| humans.txt | Initiative (humanstxt.org), no formal specification | /humans.txt next to robots.txt, optionally linked via rel="author" | names the people who contributed to the website | people who want to know who is behind the website |

Sources: [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html); [Google: Overview of Google crawlers and fetchers (last updated 2026-06-12)](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers); [The /llms.txt file, v2 (llmstxt.org)](https://llmstxt.org/); [llmstxt.org: Changes (v2, August 2026)](https://llmstxt.org/changes.html); [Google Search Central: Optimizing for generative AI features (last updated 2026-07-10)](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide); [Chrome for Developers: Lighthouse llms.txt audit (2026-05-05)](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt); [Mintlify: llms.txt and llms-full.txt](https://www.mintlify.com/docs/ai/llmstxt); [IETF Internet-Draft draft-car-ai-txt-wellknown-00 (2026-06-12)](https://www.ietf.org/archive/id/draft-car-ai-txt-wellknown-00.html); [W3C TDMRep Community Group: TDM Reservation Protocol, Final Report 2024-05-10](https://www.w3.org/community/reports/tdmrep/CG-FINAL-tdmrep-20240510/); [RFC 9116: A File Format to Aid in Security Vulnerability Disclosure](https://www.rfc-editor.org/rfc/rfc9116.html); [IAB Tech Lab: ads.txt](https://iabtechlab.com/ads-txt/); [IAB Tech Lab: ads.txt Specification Version 1.1 (GitHub)](https://github.com/InteractiveAdvertisingBureau/Supply-Chain-Validation/blob/main/ads.txt%20v1.1.md); [humanstxt.org](https://humanstxt.org/). Last checked: October 8, 2026

## Controlling crawling and indexing: the options compared

| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
| robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
| noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
| noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
| rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |

Sources: [Google Search Central: Introduction to robots.txt (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/robots/intro); [Google Search Central: Block Search indexing with noindex (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/block-indexing); [Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24)](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag); [Google Search Central: How to specify a canonical URL (last updated 2026-07-10)](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls); [Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04)](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes); [Google Search Central: Remove information from Google (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/remove-information); [Search Console Help: Removals and SafeSearch reports tool](https://support.google.com/webmasters/answer/9689846). Last checked: October 8, 2026

## Not to be confused with

- [llms.txt](https://seo-geo-glossary.com/en/glossary/llms-txt/index.md): is a proposal for a Markdown file that gives AI agents an overview of a site's key content with links. It does not control access; robots.txt still does that.

## Related terms

- [Disallow directive](https://seo-geo-glossary.com/en/glossary/disallow-directive/index.md)
- [Meta robots tag](https://seo-geo-glossary.com/en/glossary/meta-robots-tag/index.md)
- [XML sitemap](https://seo-geo-glossary.com/en/glossary/xml-sitemap/index.md)
- [AI crawler](https://seo-geo-glossary.com/en/glossary/ai-crawler/index.md)
- [llms.txt](https://seo-geo-glossary.com/en/glossary/llms-txt/index.md)

## More on the searchVIU blog

- [Why you shouldn’t block your entire staging environment via robots.txt](https://www.searchviu.com/en/robots-txt-staging-environment/)

## Sources

- [Introduction to robots.txt – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [How Google interprets the robots.txt specification – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [A note on unsupported rules in robots.txt – Google Search Central Blog, 2019](https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt)
- [Overview of OpenAI Crawlers – OpenAI](https://developers.openai.com/api/docs/bots)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/robots-txt/