Telling a crawler where it may go is different from protecting information that should remain private. Start by naming the actual goal before changing technical rules. A public page you want excluded from search and a confidential document require different decisions and verification.
Reconstructed archive: July 2019. Written in November 2026 using contemporary historical sources. The archive date organizes this series; this text is not a recovered original.
The July 2019 clarification
Google’s July 2, 2019 announcement addressed unsupported robots.txt rules, including noindex, and described retiring their handling that September. It distinguished crawling rules from other approaches to indexing and access. The historical change is a reminder to use documented controls for their intended purpose.
A present-day access decision sheet
For each document or page, name the intended audience and the actual protection required. Keep the document's contents out of a general audit sheet when they are sensitive; a controlled identifier and responsible owner may be enough.
- Public and searchable: visitors should be able to read the intended information, and the search settings should reflect that decision.
- Public but excluded from search: anyone with the link may still be able to open it. Review the supported indexing directive and its interaction with crawling.
- Restricted to approved people: the system needs appropriate authentication and authorization. A difficult-to-guess URL or missing search result is not an access-control test.
Illustrative example: a customer-specific document belongs behind an approved access process. Ask the responsible administrator to test access using the permitted and unpermitted roles without copying private contents into screenshots or support messages. Record the expected result, observed result, date and owner. If public exposure is discovered, involve the responsible security or privacy contact promptly; removing a search result alone does not fix the exposure.
Our explanation of robots.txt versus noindex covers public search controls. Private information needs an access decision in addition to any search setting.
Use current implementation guidance
Have the responsible developer review the supported mechanism and any interactions with other rules. A crawling restriction can affect whether another instruction is seen. Broad changes deserve representative testing so important public content is not accidentally restricted.
Verify the stated goal
Check access protection for private material and the relevant indexing behavior for public pages with search restrictions. Retain the decision and responsible owner. The lasting lesson is to match each control to the actual business need instead of using one familiar file as a substitute for every visibility or privacy requirement.
Explore website design, digital marketing strategy, or contact Guardian Solutions to discuss your next step.
Historical source: Google announcement dated 2019-07-02.