Crawl Budget on Small Sites: What Google’s Thresholds Say and What the Crawl Stats Report Shows Instead

Crawl budget is one of the most over-sold ideas in SEO. It shows up in proposals for websites with a few hundred pages and gets blamed whenever a new service page is slow to appear in Google. For most behavioral health, treatment and B2B industrial websites we work on, Google’s own documentation agrees with us: crawl budget is not the problem. The Crawl Stats report in Search Console usually shows what is.

Everything below was checked against Google’s documentation on October 2, 2026. If you have old bookmarks, note that Google has moved its crawl budget and robots.txt pages from Search Central to a separate crawling infrastructure section; the old addresses redirect.

What Google means by crawl budget

Google defines a site’s crawl budget as the set of URLs that Google can and wants to crawl. Two things decide it. The first is the crawl capacity limit, sometimes called hostload: how many parallel connections Googlebot will hold open to your server, and for how long, without overloading it. It drops when response times climb or the server returns 5xx or 429 errors. The second is crawl demand: how much Google wants to crawl your URLs, based on how popular they are, how often they change, and how many of the URLs it knows about are worth fetching at all.

The thresholds Google publishes, and why they matter

Google’s crawl budget guide, part of its crawling infrastructure documentation, says it is written for large sites with more than 1 million unique pages that change about once a week, medium or larger sites with more than 10,000 unique pages that change daily, and sites where a large share of URLs sit in the “Discovered – currently not indexed” status in Search Console. Google adds that these numbers are rough estimates, not exact thresholds. It also says that if your pages are usually crawled the same day they are published, you do not need the guide at all.

The Crawl Stats report documentation in Search Console Help goes further. It says that if your site has fewer than a thousand pages, you should not need to use the report or worry about crawling at this level of detail.

Here is the paragraph we wish every treatment center owner read before signing an SEO proposal: Google’s documentation says crawl budget management is aimed at sites with more than 1 million unique pages that change weekly, or more than 10,000 unique pages that change daily, and Search Console Help says a site with fewer than 1,000 pages should not need its Crawl Stats report at all. A behavioral health website with a few dozen service, program, and location pages sits two to three orders of magnitude below the level where Google says crawl budget becomes a real constraint. When that site has indexing problems, the cause is almost always page quality, duplication, or internal linking, not a crawl allowance that has run out.

Where small-site crawl problems actually come from

Small sites do have crawl problems. They are just not budget problems. These are the causes we see most often on WordPress sites:

• Inflated URL inventory: WordPress can generate tag archives, author archives, date archives, attachment pages and internal search result URLs that turn a 60-page site into hundreds of thin, near-duplicate URLs Google has to evaluate.
• Near-identical location pages: dozens of city pages that swap only the city name give Google little reason to index all of them, and they tend to land in “Crawled” or “Discovered – currently not indexed” rather than ranking.
• Bot protection that blocks Googlebot: a firewall, CDN rule, or security plugin that rate-limits or challenges unfamiliar crawlers can return errors to Googlebot that a human visitor never sees.
• Redirect chains left over from redesigns: a URL that redirects to another redirect before reaching the final page costs a separate request for every hop.
• Soft 404s and orphaned pages: pages that say “not found” while returning a 200 status keep getting recrawled, while important pages with no internal links pointing to them are discovered late or not at all.

None of these is fixed by “optimizing crawl budget.” Each is fixed by cleaning up what you publish, how pages link, and how your server responds, which is where our SEO services engagements usually start.

What the Crawl Stats report actually shows

Even on a small site, the Crawl Stats report is the fastest way to confirm whether Googlebot can reach your server. It lives under Settings, then Crawl stats, and only works on a Domain property or a root-level URL-prefix property. A few details in Google’s documentation change how you read it:

• Requests, not URLs: the report counts every request Google made, so a URL fetched twice counts twice, and response percentages are based on requests rather than unique pages.
• Every redirect hop counts: if page one redirects to page two, which redirects to page three, the report records three separate requests.
• A 90-day host status window: host status tells you whether Google hit a significant availability problem in the past 90 days, and whether the most recent one was within the last week or earlier.

Host status is broken into three categories: robots.txt fetching, DNS resolution and server connectivity. Each chart has a dotted red line marking the problem threshold. Google’s example is DNS resolution failing for more than 5% of requests on a given day.

The one crawl failure that can stop everything: robots.txt

If there is a single crawl issue worth worrying about on a small site, it is an unreachable robots.txt file. A missing robots.txt that returns a 404 is fine; Google treats that as having no crawl restrictions. A robots.txt that returns a server error is different.

According to Google’s robots.txt specification, when Google finds a robots.txt file but cannot fetch it because of a 5xx error, it stops crawling the site for the first 12 hours while it keeps retrying. For the next 30 days it uses the last good copy it has. If the errors continue past 30 days and the site is otherwise reachable, Google behaves as if there is no robots.txt file; if the site has broader availability problems, Google stops crawling it. Timeouts and DNS failures on that request count as server errors.

This is the inconvenient truth behind many “Google stopped picking up our new pages” calls: a hosting change, firewall rule or security plugin started blocking or timing out Googlebot’s robots.txt requests, and nobody noticed because the site looked fine in a browser. It is also why we check robots.txt and server response behavior during every website build or redesign, not just after launch.

A ten-minute check you can run today

With owner or full user access to Search Console, this takes about ten minutes:

1. Open Search Console, select a Domain property or root URL-prefix property, then go to Settings and open Crawl stats.
2. Look at Host status. If all three categories are clear for the last 90 days, Googlebot is reaching your server reliably, and crawl capacity is not your bottleneck.
3. If host status shows a problem, click into it and check which category crossed the red line: robots.txt fetching, DNS resolution or server connectivity. Compare the dates with any hosting, DNS, CDN or plugin changes.
4. In the “By response” table, check the share of requests returning anything other than 200, 301 or 304. A meaningful share of 5xx responses or “robots.txt not available” is worth a ticket to your host.
5. Open https://yourdomain.com/robots.txt in a private browser window and confirm it loads quickly and returns plain text, not an error page, a challenge screen or a login prompt.
6. Finally, open the Page Indexing report. If many pages sit in “Discovered – currently not indexed” or “Crawled – currently not indexed,” look for archives, attachment pages, parameter URLs and near-duplicate location pages.

What to do instead of “optimizing crawl budget”

Google’s advice for sites that genuinely have a crawl budget problem is good hygiene for every site: consolidate duplicates, return 404 or 410 for pages that are permanently gone, fix soft 404s, keep sitemaps current, and avoid redirect chains. Google also notes that blocking pages in robots.txt does not shift budget to your other pages unless Google is already hitting your crawl capacity limit, which a small site almost never is.

For a treatment provider, that translates into practical decisions: fewer, stronger program and location pages instead of templated duplicates; a clean redirect map after every redesign; robots.txt and server monitoring that someone actually checks; and an internal linking structure that puts admissions, insurance and program pages within a click or two of the homepage. The same clarity helps AI search features understand and quote your pages, which is how we approach answer engine optimization, with no special file or markup Google says it does not need.

If someone says your 80-page website has a crawl budget problem, ask to see the Crawl Stats report. If host status is clear, the conversation belongs on content and site structure.