The short answer
Keeping a page out of search, asking crawlers not to fetch it and restricting who can open it are different jobs. Robots.txt and noindex do not protect confidential information. Choose the control based on the intended outcome, and use real access protection when only authorised people should see the content.
Three goals that need different controls
A private proposal is uploaded to the website. Someone adds a rule to robots.txt and calls it hidden. Elsewhere, an old public page is blocked from crawling because the team wants it removed from search results.
Both actions sound reasonable until you ask what the visitor or search engine can still do. Privacy, crawling and indexing need separate decisions. Confusing them can leave private content accessible or stop a search engine from reading the very instruction intended to remove a page.
Start with who should be allowed to open the page
If a document contains confidential customer or business information, the first question is access. A person who knows the URL should not be able to open it without the required permission. A crawler preference is not an access check.
Ask for a demonstration in a signed-out browser or with an account that should not have access. The exact protection depends on the system, but the outcome should be observable: unauthorised visitors cannot retrieve the private content. Do not rely on an obscure filename.
A worked example
Suppose a business needs a private client proposal and a separate public event page that should no longer appear in search.
Where it goes wrong
Block both URLs in robots.txt and describe them as private and removed.
What to do instead
Protect the proposal with actual access checks. Choose and verify the appropriate indexing behaviour for the public page, allowing the search engine to read the instruction where required.
The controls match two distinct outcomes. Confidentiality no longer depends on a search crawler following a preference.
Then decide what should happen in search
Robots.txt primarily tells cooperating crawlers which URLs they may fetch. A blocked URL can still appear in Google based on other information. Noindex tells a supporting search engine not to index a page it can fetch and process.
Those controls can conflict. If crawling is blocked, Google cannot read a noindex instruction on the page. Adding both without considering that sequence can undermine the intended result. Ask the implementer to explain the combined behaviour, rather than count the number of restrictions.
Verify the intended outcome after the change
Access protection can be checked directly. Search removal involves a search engine's processing and may not appear immediately. Separate those outcomes in the handover so 'configuration added' is not mistaken for 'removed from every result'.
For public pages, also consider whether removal is the right decision. An outdated page may need an accurate update or a relevant replacement. Choose the control after deciding what should happen to people who still follow an old link.
Take this into your next conversation
Questions to ask your agency
- Should anyone with the URL be able to open this?
- Are we controlling access, crawling or indexing?
- Can the search engine read the removal instruction?
- What evidence proves each intended outcome?
Common questions
Is an unlisted URL private?
No. Links can be shared or discovered. If the content requires permission, enforce that permission when someone requests it.
Will noindex remove a result immediately?
Do not assume so. Search engines need to process the instruction. Report the configuration and the observed search state separately.
Source notes
Platform guidance behind this topic:
From explanation to implementation
This guide is published by NAVIFLARE, a website, CRM and AI workflow practice. Source notes identify the platform guidance; worked examples explain a method rather than claim a client result.
Explore website design and developmentYour next step
Identify the pages involved and write down who should open them and whether they should appear in search. That gives a concrete starting point for discussing the correct website controls.
Discuss page access and search
