Password protection, in the website sense, means locking a site or directory behind a username and password, most often with HTTP Basic Authentication on the web server. A request without credentials gets a 401 status instead of the page. This entry does not cover password security for user accounts.
How does Basic Auth work?
When a client requests a protected resource without credentials, the server answers with 401 and a WWW-Authenticate header. The browser asks for a username and password and sends them in the Authorization header of the next request. The scheme, defined in RFC 7617, joins username and password with a colon and encodes the result in Base64.
Base64 is a reversible encoding, not encryption. MDN and RFC 7617 therefore call for HTTPS, otherwise the credentials travel in readable form.
Password protection and SEO
According to RFC 9110, 401 means the request lacks valid authentication credentials. Crawlers without credentials therefore receive 401 and Google does not use content from URLs that return a 4xx status such as 401, and it removes already indexed URLs from the index. In its documentation on removing content, Google lists password protection as a way to let the right users view a page while preventing Googlebot and other crawlers from accessing it. Google’s robots.txt introduction adds that robots.txt is not a mechanism for keeping a page out of Google and names noindex or password protection for that purpose. Staging sites and test servers are the typical use.
| Method | Prevents crawling? | Keeps the URL out of Search? | What Google says | Typical use |
|---|---|---|---|---|
robots.txt Disallow | yes | no, the URL can be indexed without its content if other pages link to it | used mainly to avoid overloading the server with requests; not a mechanism for keeping a page out of Google | Limiting crawler load, excluding areas from being fetched |
noindex in the robots meta tag | no, Google has to be able to fetch the page | yes | only works if the page isn't blocked by robots.txt; Google then drops the page even if other sites link to it | HTML pages that stay reachable but shouldn't show up in search results |
noindex in the X-Robots-Tag HTTP header | no, Google has to be able to fetch the URL | yes | any robots meta tag rule can also be sent as a header; same prerequisite as the meta tag | Files without an HTML head, such as PDFs, images or videos |
rel="canonical" | no | not directly; a strong signal for which of several duplicate URLs becomes canonical | helps consolidate signals such as links into the preferred URL; don't use robots.txt or noindex for canonicalization | Duplicate and very similar pages, for example from parameters |
| Status code 410 (or 404) | no, but Google gradually fetches the URL less often | yes, a previously indexed URL is removed | treats 404 and 410 the same way | Page permanently removed with no replacement |
| Password protection (login) (this term) | yes, crawlers can't reach the content | yes | one of the ways Google lists for permanent removal; authorised users see the page, crawlers don't | Private or confidential content |
| Search Console: Removals | no, Google can keep crawling the URL | hides the URL from search results for about six months | temporary only; 404/410, password protection or noindex make it permanent | Urgent cases, until a permanent measure takes effect |
robots.txt and noindex cancel each other out. If a URL is blocked by robots.txt, Google never sees the noindex, and the URL can still appear in results. Google doesn't support noindex inside the robots.txt file itself. Sources: Google Search Central: Introduction to robots.txt (last updated 2025-12-10); Google Search Central: Block Search indexing with noindex (last updated 2025-12-10); Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (last updated 2026-03-24); Google Search Central: How to specify a canonical URL (last updated 2026-07-10); Google Crawling Infrastructure: How HTTP status codes affect Google's crawlers (last updated 2026-02-04); Google Search Central: Remove information from Google (last updated 2025-12-10); Search Console Help: Removals and SafeSearch reports tool. Last checked: .
Example
AuthType Basic
AuthName "Staging access"
AuthUserFile /path/to/.htpasswd
Require valid-user
The .htpasswd file lists users with hashed passwords and should sit outside the publicly served directory.
Common mistakes
- Using Basic Auth over plain HTTP.
- Leaving the protection active after launch, so the live site returns 401 to everyone, including Googlebot.
- A syntax error in
.htaccess, which causes a 500 error (practical experience). - A page that should appear in search is protected by accident. Search Console reports the status “Blocked due to unauthorized request (401)”: the page was blocked to Googlebot by a request for authorization.
Sources
- HTTP authentication – MDN Web Docsdeveloper.mozilla.org
- RFC 7617: The 'Basic' HTTP Authentication Scheme – IETFrfc-editor.org
- Introduction to robots.txt – Google Search Centraldevelopers.google.com
- RFC 9110: HTTP Semantics, 15.5.2 401 Unauthorized – IETFrfc-editor.org
- Remove information from Google Search – Google Search Centraldevelopers.google.com
- Page indexing report – Search Console Help (Blocked due to unauthorized request (401))support.google.com
Last updated: