robots.txt vs. noindex: Which One Actually Keeps a Page Out of Google

A treatment center wants its admissions thank-you page out of Google. Someone adds a Disallow line to robots.txt, and a few weeks later the page is still in search results, now with no description under it. Nothing is broken. robots.txt was never meant to do that job. It controls crawling, noindex controls indexing, and mixing up the two is one of the most common technical SEO mistakes we find when we audit a site.

The Short Answer

robots.txt and noindex do different jobs, and using both on the same URL makes the noindex useless. robots.txt tells Googlebot which URLs it may fetch. A noindex rule, placed in a meta robots tag or an X-Robots-Tag HTTP header, tells Google not to show a page it has already fetched. Google’s documentation on blocking indexing with noindex says that for noindex to work, the page must not be blocked by robots.txt. If the crawler can’t fetch the page, it never sees the noindex rule, and the URL can still show up in results when other pages link to it. To keep a page out of Google, let it be crawled and serve noindex. To save crawl activity on URLs that don’t matter, use robots.txt.

What robots.txt Actually Controls

robots.txt is a plain text file at the root of your domain. It lists paths that particular crawlers may or may not request. Google’s own introduction to robots.txt says it is used mainly to avoid overloading a site with requests, and that it is “not a mechanism for keeping a web page out of Google.”

The reason is simple. Google can learn that a URL exists without ever visiting it, because other pages link to it, sometimes with descriptive anchor text. A disallowed URL can therefore be indexed from those signals alone. Because Google was never allowed to read the page, the listing often has no useful snippet. In Search Console’s Page Indexing report, this shows up as the status Indexed, though blocked by robots.txt.

Some details from Google’s robots.txt specification that matter in practice:

• Supported fields: Google reads user-agent, allow, disallow and sitemap. Other fields, including crawl-delay, are not supported, so a crawl-delay line does nothing for Googlebot.
• File size limit: Google enforces a 500 KiB limit on robots.txt and ignores anything after that point.
• Caching: Google generally caches robots.txt for up to 24 hours, so a change you make this morning may not take effect until tomorrow.
• No noindex inside robots.txt: a Noindex: line in robots.txt is not a supported rule. It is ignored.

What Happens When robots.txt Itself Fails

This part of the spec rarely comes up until a hosting problem forces it. Google’s robots.txt specification says a robots.txt file that returns a server error does not fail open. If Googlebot finds a robots.txt file but gets a 5xx error when fetching it, Google stops crawling the whole site for the first 12 hours while it keeps retrying. If it still can’t get a fresh copy, it uses the last good cached version for up to 30 days. A 4xx error works the other way: every 4xx status except 429 is treated as if no robots.txt file exists, which means no crawl restrictions at all.

So a firewall rule or security plugin that starts returning 403 for robots.txt quietly lifts every restriction you wrote. A misconfigured server that returns 500 for the same file can stop Google from crawling your service pages for half a day. Both tend to appear after a security plugin, CDN or firewall change, and neither is visible to someone browsing the site. The only way to catch them is to request robots.txt directly and look at the status code.

What noindex Actually Controls

noindex is an indexing instruction. You can deliver it in two ways:

• Meta robots tag: <meta name="robots" content="noindex"> in the page’s <head>. Most WordPress SEO plugins set this with a per-page checkbox.
• X-Robots-Tag header: X-Robots-Tag: noindex sent in the HTTP response. This is the only option for non-HTML files such as PDFs, which can’t carry a meta tag.

Once Googlebot crawls the page and reads the rule, Google drops the page from search results, even if other sites link to it. The key word is crawls. The rule only works if Google can fetch the page and read the response.

Why Pages Stay in Google After You Try to Remove Them

When a client tells us a page “won’t go away,” the cause almost always falls into one of these patterns:

• Both rules on one URL: the page has a noindex tag and is also disallowed in robots.txt. The disallow stops Google from ever seeing the noindex, so the URL stays indexed from links alone.
• Disallow used as a delete button: someone blocked a retired page in robots.txt to “remove” it. Google stops refreshing it, so the old listing lingers instead of being dropped.
• Staging settings copied to production: a development site is launched with its noindex or disallow rules still in place, and the whole live site slowly falls out of results. WordPress’s “Discourage search engines from indexing this site” setting is the usual culprit.
• The rule isn’t in the HTML Google received: the noindex is added by JavaScript after load, or a caching layer serves an older copy of the page without it.

The fix for the first two is the same, even though it feels backwards: remove the robots.txt block, leave the noindex in place, and let Google crawl the page so it can read the instruction. After Google has recrawled the URL and dropped it, you can add the disallow back if you still want to limit crawling.

A Behavioral Health Caveat: robots.txt Is Public

Treatment providers sometimes treat robots.txt as a privacy control. It isn’t. Anyone can open yourdomain.com/robots.txt in a browser. Listing /patient-portal/ or /intake-uploads/ there does nothing to protect those areas. It just advertises where they are, and it only asks well-behaved crawlers to stay out.

Anything that could contain protected health information belongs behind authentication, not behind a robots.txt line or a noindex tag. Google’s own robots.txt documentation points to password protection for content that should not be reachable from search. noindex is for pages that are safe for anyone to see but have no value in search results: admissions form confirmation pages, internal staff resources, printable duplicates of existing pages, and thin tag or archive pages.

What This Means for AI Search

The same crawl-versus-index split applies to Google’s AI features. Google’s guidance on AI features and your website says robots.txt rules for Googlebot are how site owners control crawling for Search, and that nosnippet, data-nosnippet, max-snippet and noindex are the controls for limiting what appears from a page. No separate AI-only tag exists.

We are regularly asked about Google-Extended. Google’s list of common crawlers describes it as a robots.txt token that manages whether your content may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. It also says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. Blocking it will not remove you from AI Overviews, and allowing it will not get you into them. If AI search visibility is the goal, the work is the same as for any other search result: crawlable pages that answer the question clearly. That is how we approach answer engine optimization.

A Ten-Minute Check You Can Run Today

1. Open your robots.txt. Go to yourdomain.com/robots.txt. Confirm it loads with a 200 status, not a 403 or 5xx. Read every Disallow line and ask whether that path is something you’re trying to keep out of results. If it is, the disallow is the wrong tool.
2. Check the Page Indexing report. In Search Console, open Indexing > Pages and look for Indexed, though blocked by robots.txt. Every URL listed there is one Google indexed without being allowed to read it.
3. Check your noindexed pages. In the same report, open Excluded by ‘noindex’ tag. Make sure no service page, location page or clinician bio is on that list by accident.
4. Inspect one URL. Run a page you’ve noindexed through the URL Inspection tool and view the crawled HTML. The noindex tag should be in the HTML Googlebot received, not added afterward by a script.

If step 1 turns up a disallowed path you meant to deindex, remove the disallow, confirm the noindex is present, and request indexing in URL Inspection so Google recrawls it sooner.

Getting It Right Without Guessing

Crawl and index controls fail silently. A wrong line in robots.txt produces no error page, just missing or lingering search results that nobody connects back to a text file. In technical audits for our SEO services, these conflicts are among the first things we check, and we check again at launch on web design projects, where staging rules most often slip through.