# Crawler directives

Category: [Crawling & Indexing](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/) › [robots.txt & robots directives](https://seo-geo-glossary.com/en/glossary/category/crawling-indexing/robots-txt-directives/) · Also known as: robots directives

**Crawler directives** are instructions that tell search engine bots how to fetch and process a website. The term is a common SEO umbrella term and does not appear in Google's documentation. It usually covers robots.txt, the robots meta tag and the X-Robots-Tag HTTP header. Some glossaries add the canonical tag, which Google describes as a strong signal.

## Which crawler directives exist?

**robots.txt** defines which URLs a crawler may fetch. Google describes it as a way to manage crawl traffic. It is not a mechanism for keeping a page out of Google, because a blocked URL can still appear in results when other pages link to it.

The **robots meta tag** in the `<head>` and the **X-Robots-Tag** HTTP header control indexing and display. Valid rules include `noindex`, `nofollow`, `nosnippet`, `none` and `max-snippet`. The header also works for files such as PDFs, images and videos.

## How do they interact?

If robots.txt blocks a page, Google cannot read indexing rules on that page, so a `noindex` there is ignored.

## Example

```robots
# robots.txt
User-agent: *
Disallow: /cart/
```

```html
<meta name="robots" content="noindex, nofollow">
```

## Controlling crawling and indexing: the options compared

| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
| robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
| noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
| noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
| rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |

Sources: [Google Search Central: Introduction to robots.txt (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/robots/intro); [Google Search Central: Block Search indexing with noindex (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/block-indexing); [Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24)](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag); [Google Search Central: How to specify a canonical URL (last updated 2026-07-10)](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls); [Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04)](https://developers.google.com/crawling/docs/troubleshooting/http-status-codes); [Google Search Central: Remove information from Google (last updated 2025-12-10)](https://developers.google.com/search/docs/crawling-indexing/remove-information); [Search Console Help: Removals and SafeSearch reports tool](https://support.google.com/webmasters/answer/9689846). Last checked: October 8, 2026

## Related terms

- [robots.txt](https://seo-geo-glossary.com/en/glossary/robots-txt/index.md)
- [Meta robots tag](https://seo-geo-glossary.com/en/glossary/meta-robots-tag/index.md)
- [X-Robots-Tag](https://seo-geo-glossary.com/en/glossary/x-robots-tag/index.md)
- [noindex](https://seo-geo-glossary.com/en/glossary/noindex/index.md)
- [nofollow](https://seo-geo-glossary.com/en/glossary/nofollow/index.md)

## Sources

- [Introduction to robots.txt – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots/intro)
- [Robots meta tag, data-nosnippet, and X-Robots-Tag specifications – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
- [How to specify a canonical URL with rel="canonical" and other methods – Google Search Central](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls)

---

Last updated: October 8, 2026
URL: https://seo-geo-glossary.com/en/glossary/crawler-directives/