Deutsch
Servers, HTTP & caching

Web application firewall

Also known as: WAF

A web application firewall (WAF) inspects the HTTP and HTTPS traffic that reaches a website or web application and blocks suspicious requests. OWASP describes it as an application firewall for HTTP applications that applies rules to HTTP conversations. For SEO it matters because it can slow down or lock out crawlers as well.

How does a WAF work?

A WAF sits in front of the application, much like a reverse proxy. It analyzes request methods, headers, query strings and body content against rules. Typical targets are attacks such as SQL injection and cross-site scripting, malicious bots and floods of requests. Rules can follow an allowlist model, which permits only approved patterns, or a blocklist model, which stops known bad patterns. A WAF can run in the cloud, on premises or as a hybrid.

A classic network firewall filters by IP address, port and protocol. A WAF works at layer 7, the application layer, and looks at what the request contains. Many organizations run both.

CDN: edge server in front of the origin serverUsers and Googlebot request pages from the nearest edge server of the CDN. It holds the WAF and bot rules, edge functions that can change the response, and the cache. If a copy is cached, the edge server responds directly. If not, it fetches the page from the origin server. If the WAF blocks a crawler, Google recommends status code 503 or 429.UserGooglebotEdge server at the nearest locationWAF and bot ruleslet requests through or blockEdge functionschange the response (edge SEO)Cacheis a copy of the page stored?blockOrigin server(origin)cache missCache hit · the edge server responds directly, the origin is not asked.Cache miss · the edge server fetches the page from the origin and stores it.First request · the cache is still empty, the origin serves every URL at least once.Block · if the WAF blocks a crawler, Google recommends 503 or 429.
Google allows higher crawl rates for sites backed by a CDN. On the first request for a URL the cache is still empty, so the origin server delivers it. The order inside the edge server is simplified and differs between providers. Sources: Google Search Central Blog: Crawling December – CDNs and crawling (Dec 24, 2024); MDN: CDN (glossary); Cloudflare: Workers HTMLRewriter (example of edge functions).

WAFs and crawlers

Many WAF products include bot rules. According to reports, those rules can hit Googlebot or an SEO crawler and lead to error responses. Vendors list false positives, legitimate requests that get blocked, as a known challenge that needs tuning.

Good bot rules separate real crawlers from imitations. Cloudflare verifies bots through published IP lists with reverse DNS or through a cryptographic signature (Web Bot Auth). You can check Googlebot the same way (see bot verification).

Example

A log excerpt (sample data) shows what an overly strict rule looks like:

Text
192.0.2.15 - - [06/Oct/2026:10:12:01 +0000] "GET /products/ HTTP/1.1" 403 199 "-" "ExampleBot/1.0"

The crawler receives status 403 instead of the page. Grouping status codes by bot in the log makes such patterns visible.

Common mistakes

  • Recognizing Googlebot by user agent alone. Google describes verification through reverse DNS or published IP ranges, because third parties can claim to be Google. Allowing only the user agent lets imitations through as well.
  • Slowing crawlers with 401 or 403. Google says not to use these codes to limit the crawl rate. The 4xx codes except 429 have no effect on it, and Google does not use the content of such URLs.

FAQ

What is WAF in cyber security? A filter for web traffic that applies rules to HTTP requests to defend web applications.

What is the difference between a WAF and a firewall? A WAF checks request content at the application layer. A network firewall controls connections at lower network layers.

More on the searchVIU blog

Sources

  1. Web Application Firewall – OWASP Communitycommunity.owasp.org
  2. What is a web application firewall (WAF)? – Ciscocisco.com
  3. Verified bots – Cloudflare Docsdevelopers.cloudflare.com
  4. What is a WAF? – Prophaze (secondary)prophaze.com
  5. How HTTP status codes affect Google's crawlers – Google Crawling Infrastructuredevelopers.google.com
  6. Verifying requests from Google crawlers – Google Crawling Infrastructuredevelopers.google.com

Last updated: