Crawler directives are instructions that tell search engine bots how to fetch and process a website. The term is a common SEO umbrella term and does not appear in Google’s documentation. It usually covers robots.txt, the robots meta tag and the X-Robots-Tag HTTP header. Some glossaries add the canonical tag, which Google describes as a strong signal.
Which crawler directives exist?
robots.txt defines which URLs a crawler may fetch. Google describes it as a way to manage crawl traffic. It is not a mechanism for keeping a page out of Google, because a blocked URL can still appear in results when other pages link to it.
The robots meta tag in the <head> and the X-Robots-Tag HTTP header control indexing and display. Valid rules include noindex, nofollow, nosnippet, none and max-snippet. The header also works for files such as PDFs, images and videos.
| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |
robots.txt and noindex cancel each other out. If a URL is blocked by robots.txt, Google never sees the noindex, and the URL can still appear in results. Google doesn't support noindex inside the robots.txt file itself. Sources: Google Search Central: Introduction to robots.txt (last updated 2025-12-10); Google Search Central: Block Search indexing with noindex (last updated 2025-12-10); Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24); Google Search Central: How to specify a canonical URL (last updated 2026-07-10); Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04); Google Search Central: Remove information from Google (last updated 2025-12-10); Search Console Help: Removals and SafeSearch reports tool. Last checked: .
How do they interact?
If robots.txt blocks a page, Google cannot read indexing rules on that page, so a noindex there is ignored.
Example
# robots.txt
User-agent: *
Disallow: /cart/
<meta name="robots" content="noindex, nofollow">
Sources
- Introduction to robots.txt – Google Search Centraldevelopers.google.com
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications – Google Search Centraldevelopers.google.com
- How to specify a canonical URL with rel="canonical" and other methods – Google Search Centraldevelopers.google.com
Last updated: