The meta robots tag is an HTML element in the <head> of a page that tells search engines how to handle that page. It has no connection to the company Meta or to physical robots. The most common rule is noindex, which keeps a page out of search results. The tag sits next to the robots.txt file in the toolbox, with a different job, because robots.txt controls whether a URL is crawled and the meta robots tag controls indexing and display.
Which values does Google support?
noindexexcludes the page from results.nofollowasks Google not to follow the links on the page.noneequalsnoindex, nofollow.nosnippetshows no text or video preview.max-snippet:[number],max-image-preview:[none|standard|large]andmax-video-preview:[number]limit the preview.indexifembeddedlets a noindex page be indexed when it is embedded in another page.notranslateandnoimageindexswitch off translation and image indexing.unavailable_after:[date]removes the page after a date.
all is the default and needs no tag. Google states that noarchive is no longer used. Rules can be combined in one tag with commas or placed in several tags. If rules conflict, the more restrictive one applies, so nosnippet beats max-snippet:50.
| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
noindex in the robots meta tag (this term) | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |
robots.txt and noindex cancel each other out. If a URL is blocked by robots.txt, Google never sees the noindex, and the URL can still appear in results. Google doesn't support noindex inside the robots.txt file itself. Sources: Google Search Central: Introduction to robots.txt (last updated 2025-12-10); Google Search Central: Block Search indexing with noindex (last updated 2025-12-10); Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24); Google Search Central: How to specify a canonical URL (last updated 2026-07-10); Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04); Google Search Central: Remove information from Google (last updated 2025-12-10); Search Console Help: Removals and SafeSearch reports tool. Last checked: .
Example
<meta name="robots" content="noindex, nofollow">
<meta name="googlebot" content="max-snippet:160">
The robots name addresses all crawlers. The googlebot name addresses Google’s web crawler only. Files that have no HTML head, such as PDFs, take the same rules in an X-Robots-Tag HTTP header instead.
Common mistakes
- Blocking the page in robots.txt. Google never fetches the page, so it does not see the tag.
- Relying on noarchive. Google no longer uses it.
- Using nofollow on important pages by accident. The page-level value covers every link on the page.
Sources
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications – Google Search Centraldevelopers.google.com
- <meta name="robots"> – MDN Web Docsdeveloper.mozilla.org
Last updated: