Robots.txt vs. Noindex on Treatment Center Websites: Why a Blocked Page Still Shows Up in Google

A treatment center asks us why its admissions thank-you page, an old detox landing page, or a staging copy of the site is showing up in Google when “we blocked it in robots.txt.” The answer is almost always the same: robots.txt was asked to do a job it was never designed to do. Robots.txt controls crawling. The noindex rule controls indexing. They are different mechanisms, and using the wrong one, or both at once, produces exactly the result the site owner was trying to avoid.

This post explains how each mechanism works according to Google’s own documentation, the timing rules that catch teams off guard, and a check you can run on your site today.

Robots.txt Controls Crawling, Not Indexing

A robots.txt file tells crawlers which URLs they may request. When Googlebot sees a Disallow rule for a URL, it does not fetch that page. That is the whole job. It does not tell Google to forget the URL exists.

Google’s Search Console documentation is explicit about the consequence. In the Page indexing report help, the status “Indexed, though blocked by robots.txt” describes a page that was indexed despite the block, because Google can still index a URL using information from pages that link to it. Google never reads the blocked page, so any snippet in search results will be very limited. For a treatment center, that can mean an outdated program page, a discontinued location, or a form confirmation URL appearing in results with no useful description and no way for Google to see that it should be removed.

Links come from more places than people expect: old press mentions, directory listings, email campaigns that were forwarded and posted, paid landing pages shared on social media, and internal links that nobody removed when the page was retired.

Noindex Controls Indexing, but Only If Google Can Read It

The noindex rule is the mechanism for keeping a page out of Google Search. It can be set as a robots meta tag in the page’s <head> (<meta name="robots" content="noindex">) or as an X-Robots-Tag: noindex HTTP response header, which also works for PDFs and image files. According to Google’s documentation on blocking search indexing with noindex, when Googlebot crawls the page and sees the rule, Google drops that page from search results entirely, regardless of whether other sites link to it.

The catch is in the same document: for noindex to work, the page must not be blocked by robots.txt. If Googlebot is not allowed to fetch the page, it never sees the tag. Google also states that putting noindex inside the robots.txt file itself is not supported.

This is the most common mistake we find in audits: a page carries a correct noindex tag and a robots.txt Disallow rule, added by two different people at two different times, each trying to be thorough. The robots.txt rule guarantees the noindex is never read. The fix is counterintuitive: remove the robots.txt block so Google can crawl the page and see the instruction to drop it.

Google is also candid that dropping a page is not instant. Google has to recrawl the page to see a new noindex, and depending on how important the page is, that can take months. The URL Inspection tool in Search Console lets you request a recrawl of a specific URL to speed that up.

The Timing Rules That Surprise Site Owners

Google’s robots.txt behavior is documented in detail in How Google interprets the robots.txt specification. Several of those rules are specific enough to plan around, and they are where well-meaning changes go wrong.

Google generally caches a site’s robots.txt file for up to 24 hours, so an edit is not seen the moment it is saved. If the robots.txt file returns a server error (a 5xx status), Google stops crawling the entire site for the first 12 hours while it keeps retrying; after that, it uses the last good copy of the file for up to 30 days. If the robots.txt file returns a 4xx error other than 429, Google treats the site as having no robots.txt at all, which means no crawl restrictions. Google also ignores anything past the first 500 kibibytes of the file.

Each of those numbers maps to a real failure pattern:

• A server outage on robots.txt: a hosting problem that returns 5xx errors for the robots.txt file can pause Google’s crawling of the whole site, not just that one file.
• A firewall that blocks Googlebot: a security plugin or web application firewall that returns 403 Forbidden for robots.txt makes Google behave as though every URL is open to crawling, the opposite of what a restrictive file intended.
• A staging launch that copied the wrong file: a site migrated from a staging environment with Disallow: / still in place blocks crawling of the live site, and because the file is cached, fixing it does not take effect instantly.
• A bloated file: rules written past the 500 KiB limit are silently ignored, which usually happens on sites that generated a separate rule for every filtered or parameterized URL instead of consolidating patterns.

Robots.txt and Noindex Are Not Privacy Controls

This matters more for behavioral health providers than for most industries. Robots.txt is a public file: anyone can open yourdomain.com/robots.txt in a browser. Listing paths such as an intake-form directory or a patient portal in robots.txt does not hide them; it advertises where they are. Well-behaved crawlers follow the file voluntarily, and nothing technically forces any crawler to obey it.

Neither robots.txt nor noindex is an access control, and neither is a HIPAA safeguard. Any page or file containing protected health information belongs behind authentication, on systems covered by the appropriate agreements, not on a public URL that merely asks search engines to look away. If a page should never be seen by the public, the fix is a login or removing the file, not a crawler directive. If it has already appeared in Google, Search Console’s Removals tool can temporarily hide it while the underlying page is secured or deleted.

Which Tool to Use for Common Treatment Center Pages

The decision comes down to one question: do you want the URL out of Google’s index, or do you want Google to stop spending requests on it?

• Form confirmation and thank-you pages: use noindex and leave them crawlable, so the tag can be read and the page stays out of results.
• Retired program or location pages: redirect to the closest live equivalent if one exists; if not, return a 404 or 410 rather than blocking the old URL in robots.txt.
• Staging and development sites: put them behind password protection. A crawler directive alone is not enough, because staging URLs leak through links and shared previews.
• Internal search results and endless filter combinations: robots.txt is the right tool here, because the goal is preventing wasted crawling of near-infinite URL variations, not removing specific pages from results.
• PDFs such as old brochures or admission packets: use the X-Robots-Tag: noindex response header, since a PDF has no HTML head for a meta tag.

On WordPress sites, the “Discourage search engines from indexing this site” checkbox under Settings, Reading is implemented in current versions as a noindex robots meta tag. It is a common culprit when a newly launched site never appears in Google: the box was ticked during development and nobody unticked it at launch. We check it on every website launch we run.

A Fifteen-Minute Check You Can Run Today

1. Open https://yourdomain.com/robots.txt in a browser. Read every Disallow line and ask whether each path is meant to be uncrawled or meant to be out of search results. If it is the second, the rule is the wrong tool.
2. Check the status code the file returns. From a terminal, run curl -I https://yourdomain.com/robots.txt. You want 200. A 403 or 5xx response means Google is not reading the rules you think it is reading.
3. In Search Console, open Indexing, then Pages, and look for “Indexed, though blocked by robots.txt.” Every URL listed there is a page someone tried to hide with the wrong mechanism.
4. For any page that should be removed, delete its robots.txt rule, confirm the noindex tag is present using URL Inspection’s live test, and request indexing so Google recrawls and sees the tag.
5. Under Settings in Search Console, open the robots.txt report to confirm which version of the file Google last fetched and whether it reported any fetch errors.

Crawl and index directives are a small part of technical SEO, but they fail quietly. A misconfigured file rarely produces an error message; it produces pages that should not rank, or pages that should rank and never get crawled. If you want a second set of eyes on your site’s crawl setup, our SEO services start with exactly this kind of audit. The same clean indexing foundation is what AI search visibility depends on, because a page Google cannot crawl or has been told not to index cannot be cited by Google’s AI features either.