Disallow is a rule in a robots.txt file that tells a crawler not to fetch URLs whose paths begin with a given value. It is part of the Robots Exclusion Protocol (RFC 9309) and always sits inside a group that starts with a User-agent line. Disallow controls crawling. It does not control whether a URL is indexed.
How does Disallow work?
A rule path starts with a slash. A rule for a directory should end with a slash. The path is compared with the start of each URL, and the comparison is case-sensitive, so /file.asp does not block /FILE.asp.
- An empty
Disallow:blocks nothing and allows crawling of everything. - The
*wildcard matches any sequence of characters. - The
$sign marks the end of the URL, soDisallow: /*.pdf$matches URLs that end in.pdf. - If several rules match, RFC 9309 says the most specific one applies, meaning the match with the most characters. For an equal match, the allow rule wins. Google’s documentation gives the same outcome for conflicting rules: it applies the least restrictive one.
A crawler follows the group that matches its own user agent most specifically. Groups for the same user agent are merged first.
Example
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help/
User-agent: Googlebot-Image
Disallow: /private-images/
Here every crawler skips the cart and all paths starting with /search, except /search/help/.
Does Disallow remove a page from Google?
No. Google describes robots.txt as a way to manage crawler traffic and says it is not a mechanism for keeping a page out of Google. A blocked URL can still appear in results without a description if other pages link to it. A noindex rule helps only when the crawler can read it, because Google ignores robots meta tags and headers on pages blocked in robots.txt. To keep a page out of the index, allow crawling and use noindex, or protect the page with a password.
| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
robots.txt Disallow (this term) | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |
robots.txt and noindex cancel each other out. If a URL is blocked by robots.txt, Google never sees the noindex, and the URL can still appear in results. Google doesn't support noindex inside the robots.txt file itself. Sources: Google Search Central: Introduction to robots.txt (last updated 2025-12-10); Google Search Central: Block Search indexing with noindex (last updated 2025-12-10); Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24); Google Search Central: How to specify a canonical URL (last updated 2026-07-10); Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04); Google Search Central: Remove information from Google (last updated 2025-12-10); Search Console Help: Removals and SafeSearch reports tool. Last checked: .
Common mistakes
- Blocking and expecting deindexing. The page stays unseen, and so does its noindex.
- Blocking files needed for rendering. Google advises blocking scripts and stylesheets only if the page still renders without them.
- Wrong case or missing slash. The rule then matches different URLs than intended.
- Treating robots.txt as security. RFC 9309 states the protocol is no substitute for access control.
Sources
- How to write and submit a robots.txt file – Google Search Centraldevelopers.google.com
- Introduction to robots.txt – Google Search Centraldevelopers.google.com
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications – Google Search Centraldevelopers.google.com
- RFC 9309: Robots Exclusion Protocol – IETFrfc-editor.org
- How Google interprets the robots.txt specification – Google Search Centraldevelopers.google.com
Last updated: