Blog · 2026-08-30

Your Site May Be Burning Crawl Budget: Soft 404s and the Cloudflare Pages SPA Fallback Trap

Returning 200 plus a full homepage for every bad URL is not "at least there is something there", it is crawlers spending their quota on garbage URLs. One 404.html fixes it, and the fix surfaces a batch of problems you could not see before.

Indexing on a new site refuses to climb and you assume the content is not good enough. There is another possibility: every URL that does not exist, dead old backlinks, mistyped internal links, paths a scanner is probing, returns 200 on your site and serves a full copy of the homepage. Crawlers come every day and spend the whole quota on that garbage, while the pages that matter never make the queue.

The Symptom: curl a Path That Does Not Exist and See What Comes Back

We had a 60-odd page content site that had long suffered from insufficient crawling. During the health check we curled a few paths that definitely do not exist: /random-xyz, /rss.xml, /llms.txt. All returned HTTP 200, with the complete homepage HTML as the body, 87KB, canonical pointing at /.

That is a soft 404: the URL does not exist and the server says it does. It has two insidious properties:

• Monitoring stays green. Uptime checks and click tests all get a 200, so nothing ever alerts.
• It fools you too. You want to confirm whether the site has an llms.txt, curl returns 200, so you assume it exists. What you actually received was the homepage. This caught us once for real.

The cause is simple: when there is no 404.html in the root of the build output, Cloudflare Pages falls back to single-page application mode automatically, returning index.html with a 200 for any path that does not match. For an actual SPA that is a feature. For a multi-page static site it is a disaster, and a silent default.

Why This Burns Crawl Budget

"The homepage canonical points at /, so bad URLs will not get indexed as duplicate content. What is the problem?"

Indexing is indeed blocked. Crawling is not. The canonical tag lives in the HTML, so Googlebot has to download the entire 87KB homepage before it can read it. Every bad URL is a full download. And bad URLs come from more places than you would think: external links someone mistyped, old paths left over from a redesign, and the /feed and /.env style paths scanners always probe.

Worse, returning 200 tells the search engine "this URL is alive", so it stays in the queue and gets recrawled repeatedly, while a real 404 gradually gets recrawled less often. Google's crawl budget management guide lists soft 404s explicitly among the things that waste crawl resources. On a large site that is noise. On a site already short of crawl capacity, it is pouring the quota that could save you straight down the drain.

The Fix: Drop In a 404.html and Make the Platform Leave SPA Mode

The fix is disproportionately cheap: put a real 404.html in the root of the build output, Cloudflare Pages detects it and switches back to static site mode, and unmatched paths return a genuine HTTP 404.

We built the 404 page into the generator rather than hand-writing an orphan file, using the site's own CSS and navigation and linking to the main pages, so a real person who lands there has somewhere to go.

Measured results:

• Bad URLs: 200 / 87KB → 404 / 34KB
• All 67 real pages on the site still return 200, with zero collateral damage
• Repeated the run with Googlebot's user agent, and got the same 404

Verification covers three angles: bad URLs now 404, good URLs did not break along with them, and the crawler identity sees what you see. Checking only the first and calling it done is a gamble.

The Day After the Fix, the Dead Links Showed Themselves

The day after the soft 404 was fixed, our own people clicked the navigation bar and hit a 404. Tracing it turned up two long-standing dead links in the homepage navigation: one pointing at a page that was never built at all, and another with an extra directory level, where the correct path is /how-to-pick/ but the link read /guide/how-to-pick/.

Why had nobody noticed? Because in the soft 404 era, clicking them returned 200 and the homepage, which looks like "a bit odd but it works". Once the 404 was fixed, those links appeared as actual 404s in front of human eyes for the first time. The root cause is typical too: the homepage was maintained by hand while the inner page navigation was generated, and the two drifted apart as they each evolved.

So do not treat a fixed 404 as a closed case. It is the start of old problems becoming visible. Over the following days, expect to dredge up a batch of previously invisible broken links.

Make the Check a Deploy Gate, and Prove It Fires First

A problem human eyes cannot catch should not keep depending on human eyes. We added a dead link gate to the deploy pipeline: it scans every internal href in the build output, and if one points at no actual file and no redirect rule catches it, the deploy is blocked.

The key part is negative testing: deliberately put a known dead link back in and confirm the gate actually blocks it. A gate you have never seen stop anything is just decoration that makes people feel safe.

The same full-site scan turned up another problem invisible to the naked eye: 27 page titles ran over 30 full-width characters (including the homepage at 33.5), while Traditional Chinese search results truncate at roughly 28 to 30 characters, so everything past that was written for nothing. We compressed them page by page to under 30 characters, keeping the brand and the core keyword phrase and cutting only the trailing filler, which brought the over-limit count to zero.

Closing

What makes soft 404s hard is not the fix, one 404.html handles that, it is that they are asymptomatic the whole way through: monitoring is green, clicks produce a page, and even your diagnostic commands get fooled. If your site runs on Cloudflare Pages or any platform with an SPA fallback, curl a path that does not exist right now and see what comes back. If it is 200, you have found one of the reasons your indexing is not moving. Once fixed, turn the dead link scan into a deploy gate, then use a negative test to prove it works. We have handled plenty of these "looks fine so nobody fixes it" problems. If you want to know what is still hiding on your site, come talk to us.

We Turned This Kind of Technical Debt Into a Service

A soft 404 will not take your site down. It will quietly burn through your crawl budget, until one day you notice new pages never get indexed.

• Site architecture and technical SEO: status codes, canonical tags, sitemap and robots aligned in one pass
• Index monitoring: indexing status page by page, identifying pages that were crawled but not indexed
• Content production: spend the crawl budget on pages that have content

Specs are on iGaming SEO Site Build.