For small sites, crawl budget doesn't matter — Google has plenty of capacity to visit every page often. For sites with tens or hundreds of thousands of URLs, crawl budget is the whole game.

What "crawl budget" actually is

Crawl budget is Google's term for two combined limits:

  • Crawl rate limit — how fast Google can hit your server without slowing it down.
  • Crawl demand — how interested Google is in your pages right now.

Multiply those, and you get how many URLs Google will fetch from your site in a given day.

Symptoms of a crawl budget problem

  • You have 50,000 URLs in your sitemap; Search Console shows only 8,000 indexed.
  • New pages take weeks to appear (or never do).
  • Search Console "Pages" report shows lots of "Discovered — currently not indexed."

The five biggest budget wasters

1. Faceted navigation

E-commerce filters that create infinite URL variants (?color=red&size=L&sort=price) burn crawl budget on duplicate-content variants. Block them with robots.txt or canonical them to the base category.

2. Internal search result pages

If your site has /?s=keyword style URLs from internal search, Google can find and crawl thousands of them. Block /?s= in robots.txt.

3. Session IDs in URLs

If your URLs carry session or tracking parameters, Google sees them as separate pages. Strip parameters with canonicals or use Search Console's URL Parameters tool.

4. Soft 404s

Pages that say "no results" but return HTTP 200 trick Google into crawling and indexing useless content. Return a real 404 status code for empty pages.

5. Slow server

If your server takes 3 seconds to respond, Google halves its crawl rate so it doesn't take you down. Fix server speed and Google will crawl more.

What helps

  • Clean, deduplicated sitemap with only canonical URLs you actually want indexed.
  • Internal linking from high-authority pages to your new content.
  • Indexing API submissions for new URLs (skip the crawl queue entirely).
  • Server speed under 500ms TTFB.

Stop waiting for Google's schedule

The Indexing API doesn't increase your overall crawl budget, but it lets you bypass the queue for individual URLs you care about. Pair it with the cleanup above and you'll see noticeably faster indexing inside 30 days.