- Crawl-delay
Crawl-delay is a robots.txt directive that asks a crawler to wait between requests.
- Crawler directives
Crawler directives are instructions that tell search engine bots how to fetch and process a website.
- data-nosnippet
data-nosnippet is an HTML attribute that marks a section of text on a page as off limits for snippets in search results.
- Disallow directive
Disallow is a rule in a robots.txt file that tells a crawler not to fetch URLs whose paths begin with a given value.
- max-snippet
max-snippet is a robots rule that sets the maximum length of a text snippet in search results, counted in characters.
- Meta robots tag
The meta robots tag is an HTML element in the <head> of a page that tells search engines how to handle that page.
- noarchiveHistorical
noarchive is a robots rule that used to stop Google from showing a cached link to a saved copy of a page in search results.
- nofollow
nofollow is a value that tells search engines not to follow a link or all links on a page.
- noimageindex
noimageindex is a robots rule that tells Google not to index the images on the page where it appears.
- noindex
noindex is a rule that tells search engines not to show a page or file in their search results.
- nosnippet
nosnippet is a robots rule that tells a search engine not to show a text snippet or video preview for a page.
- Paywalled content
Paywalled content is content that is fully visible only after payment, subscription or registration.
- Robots Exclusion Protocol
The Robots Exclusion Protocol (REP) is the standard that lets site owners tell crawlers, through a robots.txt file, which parts of a site …
- robots.txt
robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may request.
- unavailable_after
unavailable_after is a robots rule that gives Google a date and time after which a page should no longer appear in search results.
- X-Robots-Tag
The X-Robots-Tag is an HTTP response header that passes indexing and serving rules for a URL to search engines.