# Disallow directive

Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [robots.txt & robots directives](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/robots-txt-directives/) · Also known as: robots.txt disallow, Disallow rule, Disallow directive

**Disallow** is a rule in a robots.txt file that tells a crawler not to fetch URLs whose paths begin with a given value. It is part of the Robots Exclusion Protocol (RFC 9309) and always sits inside a group that starts with a `User-agent` line. Disallow controls crawling. It does not control whether a URL is indexed.

## How does Disallow work?

A rule path starts with a slash. A rule for a directory should end with a slash. The path is compared with the start of each URL, and the comparison is case-sensitive, so `/file.asp` does not block `/FILE.asp`.

- An empty `Disallow:` blocks nothing and allows crawling of everything.
- The `*` wildcard matches any sequence of characters.
- The `$` sign marks the end of the URL, so `Disallow: /*.pdf$` matches URLs that end in `.pdf`.
- If several rules match, RFC 9309 says the most specific one applies, meaning the match with the most characters. For an equal match, the allow rule wins. Google's documentation gives the same outcome for conflicting rules: it applies the least restrictive one.

A crawler follows the group that matches its own user agent most specifically. Groups for the same user agent are merged first.

## Example

```robots
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help/

User-agent: Googlebot-Image
Disallow: /private-images/
```

Here every crawler skips the cart and all paths starting with `/search`, except `/search/help/`.

## Does Disallow remove a page from Google?

No. Google describes robots.txt as a way to manage crawler traffic and says it is not a mechanism for keeping a page out of Google. A blocked URL can still appear in results without a description if other pages link to it. A noindex rule helps only when the crawler can read it, because Google ignores robots meta tags and headers on pages blocked in robots.txt. To keep a page out of the index, allow crawling and use noindex, or protect the page with a password.

## Common mistakes

- **Blocking and expecting deindexing.** The page stays unseen, and so does its noindex.
- **Blocking files needed for rendering.** Google advises blocking scripts and stylesheets only if the page still renders without them.
- **Wrong case or missing slash.** The rule then matches different URLs than intended.
- **Treating robots.txt as security.** RFC 9309 states the protocol is no substitute for access control.

## Controlling crawling and indexing: the options compared

| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
| robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
| noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
| noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
| rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |

Sources: [Google Search Central: Introduction to robots.txt (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/robots/intro); [Google Search Central: Block Search indexing with noindex (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/block-indexing); [Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24)](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag); [Google Search Central: How to specify a canonical URL (last updated 2026-07-10)](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls); [Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04)](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes); [Google Search Central: Remove information from Google (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/remove-information); [Search Console Help: Removals and SafeSearch reports tool](https://support.google.com/webmasters/answer/9689846). Last checked: October 8, 2026

## Not to be confused with

- [noindex](https://seo-geo-glossary.com/en/glossary/noindex/index.md): tells search engines not to show a page in results and works only when Google is allowed to fetch the page. Disallow decides whether a URL is fetched at all.

## Related terms

- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Robots Exclusion Protocol](https://seo-geo-glossary.com/en/glossary/robots-exclusion-protocol/index.md)
- [noindex](https://seo-geo-glossary.com/en/glossary/noindex/index.md)
- [Meta robots tag](https://seo-geo-glossary.com/en/glossary/meta-robots-tag/index.md)
- [Crawl budget](https://seo-geo-glossary.com/en/glossary/crawl-budget/index.md)

## Sources

- [How to write and submit a robots.txt file – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt)
- [Introduction to robots.txt – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [Robots meta tag, data-nosnippet, and X-Robots-Tag specifications – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
- [RFC 9309: Robots Exclusion Protocol – IETF](https://www.rfc-editor.org/rfc/rfc9309.html)
- [How Google interprets the robots.txt specification – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/disallow-directive/