A web scraper is a program that automatically extracts data from web pages and converts it into a structured format such as a spreadsheet or database. In SEO the term usually comes up when other bots copy content from a website or generate heavy load on it.
At a glance
- Type
- Program that automatically extracts data from web pages
- Affects
- Content of publicly accessible web pages
Last checked:
How does a scraper work?
A scraper fetches pages, reads the HTML and pulls out specific elements such as prices, text or reviews. Wikipedia lists techniques such as HTML parsing, DOM parsing with browser tools like Selenium or Playwright, and text pattern matching. Many scrapers run on scripts or libraries, others come as no-code tools.
Crawling and scraping often occur together. Wikipedia describes crawling as the fetching part and scraping as fetching plus extraction. No formal standard defines the line between them.
Scraping is a technique with legitimate uses such as price comparison, market research and monitoring. For site owners it becomes a topic when requests strain the server or content gets reused without permission.
How do site owners handle scrapers?
A robots.txt file only works for bots that follow it. RFC 9309 says its rules are not a form of access authorization and that the protocol is not a substitute for valid content security measures. A scraper that ignores robots.txt is not stopped by it.
Common countermeasures include rate limiting, IP blocking, CAPTCHAs and bot management. Cloudflare classifies bots by behavior as search, agent or training, which helps to separate wanted from unwanted access.
Example
A shop with 20,000 product pages is fetched within one hour by a single IP address with an identical user agent, without requests for images or CSS. In the log file this pattern points to a scraper (Assumption).
Common mistakes
- Using robots.txt as protection. According to RFC 9309, the Robots Exclusion Protocol is not a substitute for valid content security measures. Protecting content takes measures such as authentication.
- Listing sensitive paths in robots.txt. According to RFC 9309, listing paths in robots.txt exposes them publicly and makes them discoverable.
FAQ
Is scraping the same as crawling? The terms overlap. Crawling means fetching pages, scraping means extracting data from them.
Does robots.txt stop scrapers? Only those that follow it. Technical blocks and bot management work independently of it.
Sources
- RFC 9309: Robots Exclusion Protocol – IETFrfc-editor.org
- Bots and AI classification – Cloudflare Docsdevelopers.cloudflare.com
- Web scraping – Wikipedia (secondary)en.wikipedia.org
Last updated: