Published August 2, 2026. Updated: August 6, 2026, to add additional July reference points.
A follow-up to the original framework, published July 16, 2025, itself an extension of an earlier article on Cloudflare’s Pay Per Crawl, published two weeks prior, which first sketched the nine-factor value table and raised a question this post finally has enough evidence to answer: how many of those factors evaluate a page, and how many actually evaluate the site underneath it?
A year ago, I argued that indiscriminate crawling was ending and that Cloudflare’s Pay Per Crawl, mounting infrastructure costs, and tightening legal frameworks would push search engines and AI companies from “crawl everything” to “crawl selectively.” At the time, the strongest evidence was directional: rising crawler traffic, a collapsing referral-to-crawl ratio, and publishers starting to block AI bots by default.
Twelve months on, that direction hasn’t just held; it’s been confirmed by Google’s own systems, documented in Google’s own updated guidance (detailed by Barry Schwartz in Search Engine Roundtable), and now, for the first time, tested in a European courtroom. This isn’t a revision of the thesis. It’s the thesis catching up to evidence that didn’t exist when the original was published.
Three things happened that the original framework didn’t, and, in one case, couldn’t anticipate:
- Google itself confirmed selective indexing is active, not theoretical. The May 2025 Indexing Purge, validated by Adam Gent from Indexing Insight, confirmed that my original article covered as breaking news is now a settled pattern, and Google’s own July 2026 update to its Crawl Budget documentation independently confirms mechanics the original could only infer from Mueller’s public statements and my own observations and presumptions.
- A new economic driver emerged that the original framework didn’t have an industry-recognized focus for: information gain, and a counterintuitive cost dynamic where falling per-token AI costs are increasing the pressure toward selectivity, not easing it.
- A legal ruling gave this framework a dimension it never had: a German court found Google directly liable for AI Overview inaccuracies, opening a plausible legal, not just economic, rationale for source selectivity.
Why This Framework Now Becomes Critical
For most of the web’s history, crawling was treated as a neutral, near-limitless activity. More pages discovered was generally better. Compute was the search engine’s problem, not the publisher’s. That assumption no longer holds, and the reason isn’t the one most people reach for first.
Here’s the part that gets missed: per-token AI inference costs are collapsing, not rising. LLM inference costs have dropped roughly 1,000× in three years — GPT-4-equivalent performance cost about $20 per million tokens in late 2022 versus roughly $0.40 per million tokens in early 2026. Read in isolation, that suggests compute scarcity is easing. It isn’t.
Gartner’s analysis is explicit: as token consumption rises faster than token costs fall, overall inference costs are expected to increase — “Chief Product Officers should not confuse the deflation of commodity tokens with the democratization of frontier reasoning.” The volume side of the equation is what’s actually exploding: token prices have fallen roughly 280x over two years, but total enterprise AI spend has risen 320% in the same period — driven by agentic AI workflows that trigger 10-20 LLM calls per user task, RAG architectures that inflate context windows 3-5x, and always-on monitoring agents that consume compute continuously.
Google’s own earnings numbers confirm the pattern at the source. Two days before this article was updated, Alphabet reported Q2 2026 earnings showing exactly the dynamic this framework has been arguing: total costs and expenses rose 21% year-over-year to $79.03 billion, with R&D up 32% to $18.22 billion, driven by AI-related hiring, compensation, and depreciation. Capital expenditures doubled year-over-year to $44.92 billion, enough to push free cash flow negative for the quarter, and Alphabet raised its 2026 capex guidance to $195-205 billion, up from $180-190 billion just one quarter earlier, citing accelerated AI infrastructure deployment.
This isn’t undirected spending. Alphabet has been explicit that the goal is protecting Search’s margin from AI compute costs: the company upgraded AI Overviews and AI Mode to Gemini 3 specifically to prevent higher computing costs from eroding Search’s profitability, and reports having already cut core AI response costs by more than 30% since making that move. The scale forcing the issue is staggering even by AI-industry standards — Gemini models now process 22 billion API tokens per minute, against a Gemini App user base of 950 million monthly actives.
Put plainly: Google is fighting the same battle this framework describes, in public, with real dollars. Falling per-token cost and rising total spend aren’t in tension — they’re the same story, and Google’s own capex guidance is the clearest evidence yet that the company expects that story to keep playing out through 2027, not resolve itself.
This is the mechanism that makes selective ingestion an economic inevitability rather than a preference: cheaper tokens didn’t reduce the incentive to be selective about what feeds a model — they raised it. Every additional low-value document pulled into a context window, or RAG pipeline, is now multiplied across vastly more inference calls than it would have been two years ago. A RAG-enhanced enterprise query typically consumes 3-5x more tokens than a simple query on the same underlying model. A single redundant source doesn’t cost one crawl. It gets re-processed, re-embedded, and re-retrieved indefinitely. Waste no longer stays a one-time cost. It compounds.
The Missing Mechanism: Information Gain
Most crawl-worthiness arguments stop at engagement, authority, and freshness. What they miss is a more mechanical filter underneath all three: does this document add anything the model doesn’t already have?
Google’s information gain patent describes exactly this kind of scoring: an information gain score for a given document indicates additional information that is included in the document beyond the information contained in documents that the user previously viewed. Mechanically, search results may be ranked at least in part based on their respective information gain scores, and one or more documents may be excluded or significantly demoted based on the new information gain scores. In plain terms, the logic is about boosting some pages in rankings based on how much information they would add to a searcher, and demoting them if they don’t add much.
This is the mechanism behind Gate 3 in the Web Crawling Framework model — the gate that decides whether a page that’s already cleared crawl capacity and crawl demand actually survives.
A page can pass every other test — fast server, clean sitemap, real backlinks, decent traffic — and still fail the one that decides whether it’s worth re-processing at scale: does it say something the corpus doesn’t already know? In an economy where redundant tokens carry real, compounding reprocessing cost, a page that restates consensus is a liability to ingest, not an asset. “Skyscraper content” that comprehensively covers the same ground as the top 10 results is a losing strategy for exactly this reason.
The Evidence Trail
May 2025 — The Indexing Purge. Since May 26th, 2025, over 25% of monitored URLs by Indexing Insights shifted to “crawled – previously indexed,” breaking Google’s long-observed 130-day rule and actively deindexing pages recrawled only 90-130 days prior — with 15%-75% of monitored pages removed at individual sites. The removal had zero measurable impact on clicks or impressions; one site that lost 75% of its indexed pages still saw a positive trend the following core update. Google’s John Mueller confirmed it wasn’t a bug: “Our systems make adjustments in what’s crawled & indexed regularly. That’s normal and expected… crawl capacity & crawl demand… also play a role in indexing.”
January 2026 — The Technical Ceiling Drops. Google reduced its HTML crawl limit from 15MB to 2MB, while Gary Illyes disclosed that infrastructure speed matters more than site scale, stating “if you are making expensive database calls, that’s going to cost the server a lot.” Database latency, not page count, is now the named constraint.
July 2026 — The Documentation Catches Up. Google’s crawl budget page was updated “to improve clarity, terminology consistency, and flow,” and now states outright that not every crawled page will necessarily be indexed after crawling; each page must be evaluated, consolidated, and assessed for suitability. It also confirms every site starts with the same default, conservative crawl capacity limit, shared across all crawlers — meaning demand from one crawler directly reduces capacity for others — and adds “content uniqueness” explicitly to the list of factors governing crawl-resource allocation.
Mid-July 2026 — Google Confirms the Mechanism, On the Record
During the weekly Google Search Off the Record Podcast in July, the team discussed key elements of these gates. The first is Episode 112 on July 16, 2026: How to Read the Indexing Report, and Episode 113 on July 30th: Should You Block Your Search Pages?
Episode 112 — Redundancy Defined
John Mueller, describing redundancy in his own words, unprompted, with zero patent-speak — this is Gate 3 / Information Gain stated plainly by the person who runs Search relations: “there’s so much other stuff that is just as good. So why would we add it to the index?” Immediately followed by an AI-content-specific version: “maybe most of your website is AI generated and it worked for a while, it might be that people look at this AI-generated site and they’re like, ‘Well, I can tell this is AI generated. There’s nothing unique or valuable that is available here for me.‘”
This is enormously useful because it’s a plain-language confirmation of the patent mechanism, from Google, independent of the patent filing itself — two separate lines of evidence now converge on the same concept.
Also from Episode 112 — direct confirmation that quality gates Capacity/Demand together, not just Gate 3: “if we have strong concerns about the overall quality… it doesn’t make much sense for our systems to spend a lot of time on the website. So we’ll probably crawl a lot less, we’ll index a lot less.” This is Mueller stating that Gates 1/2 and Gate 3 aren’t independent. A quality problem throttles crawling and indexing simultaneously, which sharpens the site-level tax argument I have built in the addendum below.
Also from Episode 112 — the myth-debunk you’ll want alongside the “million page” myth: Google’s own developer documentation site has roughly 5% of pages indexed, and Mueller states plainly this is fine — there’s no target ratio. Directly reinforces “don’t chase a number, understand the pattern,” and is a nice real-world counterpart to your existing Bill Hunt case-study numbers.
Episode 113 — the “it’s not a penalty, it’s inefficiency” framing, stated explicitly
This is the cleanest articulation yet of my framework’s central distinction: “it’s not necessarily something where we would say, ‘We think your site is spam’… It’s just like, you’re being very inefficient.” This is Google, on the record, drawing exactly the line the entire article has been drawing between economic deprioritization and quality penalty.
Also from Episode 113 — a new failure mode for Gate 1 not yet in the article: internal site-search pages create “infinite crawl spaces” (unbounded parameter/query URL generation), and a 500 error triggered by crawl overload causes Google to reduce crawling sitewide, not just on the offending URL cluster — direct primary-source confirmation of the shared-capacity mechanic already in Gate 1, plus a concrete, common real-world cause (search/filter URLs) that we had not detailed.
Also from Episode 113 — a genuinely new angle: the permission layer as attack surface. Bad actors exploit unblocked internal search pages to inject spam terms and get indexed riding a legitimate site’s authority. This is another element of the Permission Layer as it reframes robots.txt compliance from “efficiency hygiene” to “liability prevention,” which pairs interestingly with the Munich section’s corroboration-risk argument.
August 2026 — The Signal Convergence
Within days of the July documentation rewrite, Google added a further, distinct update: an explicit recommendation to use HTTP 304 responses, telling site owners that returning a cached-content signal for unchanged pages saves crawl resources on both sides — “if a page hasn’t changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources.” The same update also formalized the shared-capacity language covered above.
This isn’t a new idea; it’s Google formally endorsing a mechanism Cloudflare and Bing had already built dedicated tooling around. Cloudflare’s Crawler Hints exist specifically to tell search engines when content hasn’t changed, avoiding a repeat crawl. Bing’s IndexNow protocol works the inverse case by proactively pinging search engines the moment content has materially changed, rather than waiting to be recrawled. Three separate infrastructure players Google, Cloudflare, and Microsoft have now converged, independently, on the same underlying signal: don’t make me guess whether this changed; tell me. That convergence is the clearest evidence yet that this framework describes an industry-wide response to shared economics, not one company’s isolated cost-cutting decision.
The Three-Gate Model
Together, this evidence describes qualification as three sequential gates. Failing any one is fatal, regardless of performance at the others.
| Gate | Tests | Failure Mode |
| 1. Crawl Capacity | Can your server handle the crawl without degrading? | Connections throttled; less of your site gets looked at, period |
| 2. Crawl Demand | Does the system want to spend budget here — inventory, popularity, staleness? | Pages sit undiscovered or unrefreshed |
| 3. Information Gain | Does this page add something the corpus doesn’t already have? | Crawled, indexed, then excluded or demoted — May 2025 in miniature |
No amount of information gain matters if a page never clears Gates 1 and 2 if the system never reaches it. And no amount of speed or freshness saves a page that says nothing new. All three gates have to open.
Re-Introducing the Framework
If crawling and indexing are no longer neutral, cost-free acts, then every page effectively has to earn its way through evaluation — economically, technically, and now increasingly legally. The nine factors below are the practical framework for that evaluation: the dimensions search engines and AI systems appear to weigh, in some combination, when deciding whether a page is worth the compute it will cost across its lifetime of reuse.
| Factor | Value to AI/Search Systems | Cost / Risk Implication |
| Information Gain | Adds knowledge beyond the existing corpus | Determines post-indexing survival, independent of traffic |
| User Demand | Frequently queried in search or AI prompts | Justifies recurring crawl/reprocessing cost |
| Frequency of Change | Signals new information is likely | Reduces redundant crawling (Google’s HTTP 304 guidance targets this directly) |
| Corroboration / Source Reliability | Cross-verifiable by independent sources | Reduces synthesis liability risk for AI-generated answers |
| Token Efficiency | Clean, dense, low-noise content | Reduces compounding reprocessing cost across RAG/agentic calls |
| Page Authority | Trustworthiness in AI-generated answers | Reduces hallucination and legal exposure |
| Structured Signals | Schema, headings, canonical tags | Eases parsing, lowers per-crawl compute cost |
| Commercial Impact | High-value use cases (health, finance, transactions) | Increases ROI for paid crawl or licensing |
| Legal / Licensing Clarity | Signals legal access via robots.txt, headers, terms | Reduces exposure; simplifies ingestion workflows — see Permission Layer & Access Gating under Primary Filters for the enforcement mechanics now sitting underneath this signal |
We have documented each of these factors and what specifically signals it, how it’s likely weighted, and what a publisher can actually do about it. That’s the deeper breakdown this framework sets up. We offer a factor-by-factor look at what “earning inclusion” now requires, starting from the two gates almost nobody optimizes for, capacity and demand, before the newest and least understood one: information gain.
In practice, this is the test worth running before publishing anything: if you removed your page from the index entirely, would the searcher’s understanding of the topic actually change? If the answer is no, if everything in the page is already fully covered, in the same depth, by content already ranking, the page is a redundancy cost to any system that ingests it, no matter how well it’s written or optimized. This is close to the logic SEO practitioners like Marie Haynes have independently arrived at from the practitioner side: content that isn’t additive to what a searcher has already seen may simply not be worth adding, regardless of its individual quality.
A Necessary Addendum: Not Every Gate Operates at the Same Level
The scorecard above treats all nine factors as page-level inputs, scored per URL. That’s a simplification worth correcting explicitly, because Google’s own documentation draws a sharper line than the scorecard implies and the distinction changes how a publisher should actually prioritize fixes.
Google’s crawling infrastructure defines a site as a unique hostname — www.example.com and code.example.com are treated as separate sites with separate crawl budgets. That means Gates 1 and 2, Capacity and Demand, aren’t computed per-URL at all. They’re computed once, per hostname, and then applied uniformly across every page that hostname serves. Gate 3, Information Gain, is the outlier: it’s necessarily page-specific, since it’s scored against what already exists in the corpus on that exact topic, not against the domain as a whole.
| Factor | Operating Level | Why |
|---|---|---|
| Licensing Clarity | Site | robots.txt and Cloudflare’s Search/Agent/Training classification are set once and apply to every URL under the hostname |
| Ingestion Efficiency | Site | This is Gate 1 itself — hostload/crawl capacity is calculated per-hostname |
| Frequency of Change | Site (baseline) + Page (specific) | Google names update frequency as a site-level demand input; staleness is still tracked per-document |
| Authority / Crosslink Value | Site (baseline) + Page (marginal) | Domain-level link equity sets the floor; individual backlinks push a page above it |
| Structured Signals | Site (pattern) + Page (instance) | Consistent schema across a site builds pattern-level trust, even though each tag lives on one page |
| Commercial Impact | Page, occasionally Site | A page-level factor, except on domains where the entire vertical is high-stakes (health, finance) |
| Topic Scarcity | Page | Scarcity is corpus-relative to that specific topic, not the domain |
| User Demand | Page, with a site-level halo (branded search volume) | |
| Information Gain | Page only | Inherently comparative — cannot exist as a domain-wide property |
The practical consequence: a site with weak Licensing Clarity (see below) or poor Ingestion Efficiency doesn’t handicap one page — it applies a domain-wide tax that every URL under that hostname inherits before Gate 3 is even reached. This is why a generally strong site can still lose individual pages to Gate 3 failure (the May 2025 purge, applied selectively) while a generally weak site can have every page throttled at Gate 1 or 2, regardless of how good any single page is. It’s also the sharper version of the smaller-publisher concern raised a year earlier, in the piece that first sketched this framework: a small site rarely accumulates enough site-level signal to clear the first two gates decisively, no matter how strong any one page’s content is — the toll gets charged at the hostname, before the page ever gets evaluated on its own merits.
A note on Licensing Clarity’s weight here: this factor carries more legal significance than the table row suggests. The earlier rant on Cloudflare’s Pay Per Crawl argued that the 2025 Meta ruling found no functioning market for licensing AI training content exists yet, and that the absence of a market isn’t evidence none is needed; it’s evidence one hasn’t been built. Cloudflare’s Pay Per Crawl, and by extension any site’s Licensing Clarity signals, may be exactly the infrastructure that argument was missing: a working, trackable mechanism that future plaintiffs could point to if an AI firm ignores it. That’s a separate legal thread from the Munich synthesis-liability risk covered above — one about market harm and licensing, the other about output accuracy, and both now sit underneath the same “Licensing Clarity” row in the table above.
Real-World Validation: Dan Taylor’s Crawl Economics Column
Two weeks before the 304 recommendation landed, Dan Taylor published a well-articulated case-based account of exactly this framework playing out against scaled AI content — described from the failure side rather than the mechanism side, which makes it a useful practitioner cross-check. Dan’s framing of Gate 2 tracks almost exactly: when a site floods in with hundreds or thousands of new URLs, Google evaluates the site based on perceived inventory (total URL volume versus what’s actually useful), demand (how much users and Google actually care about the topics), and popularity/staleness (the baseline authority to justify the processing cost), and if demand and popularity don’t justify the spike in perceived inventory, Google will throttle resource allocation, even after an initial burst-crawl.
Dan also documents the Gate 3 failure mode in practice, with a mechanism worth naming directly: new content gets a temporary freshness-driven visibility boost regardless of quality, but once that newness wears off, the page must stand on its own against Google’s quality threshold, and Google’s recrawl window (the same ~130-140 day rule, sometimes as little as 75 days, referenced earlier in this post) shrinks dramatically for aggressive programmatic AI content specifically. His summary line makes the connection explicit without borrowing the gate terminology: “the indexing ecosystem rewards information gain, technical efficiency, and genuine demand,” and mass programmatic content fails “when it treats search optimization as a simple checklist rather than a resource management problem.”
Worth citing this specifically as the articles clearest independent confirmation that the three-gate model isn’t a retrospective theory built to fit the May 2025 purge — it’s a mechanism practitioners are actively diagnosing in live client failures, in real time, using different vocabulary but describing the identical sequence: volume without demand throttles Gate 2, and content without information gain fails Gate 3 the moment the freshness boost expires.
The Crawl Budget scope objection
Shortly after posting this, a frequent reader, long-time contrarian, sent me a note that Crawl Budget is only for large sites. They pointed out correctly that at the top of the Crawl Budget Guide states plainly it’s “an advanced guide intended primarily for” large sites and clearly defines them and, for all practical purposes, tells smaller sites outright: “If your site doesn’t have a large number of pages that change rapidly… you don’t need to read this guide.” If I may, that very exclusionary statement is prefaced by “these recommendations are generally good practices.”
That’s a real statement, and worth quoting accurately rather than glossing over. But it’s answering a different question than the one this framework is asking. Google’s scope note is about whether you need to actively manage a crawl budget constraint — for a 1,000-page site that gets fully crawled within a day of publishing, capacity essentially never binds, so there’s nothing to manage. It is not a statement that the three gates don’t apply below that threshold. The gates are the mechanism by which any page enters the index, full stop. Google says as much in the same document, without a size qualifier:
“For Google Search, not every page that is crawled will necessarily be indexed. After crawling, each page must be evaluated, consolidated, and assessed to determine its suitability for the index.”
That sentence carries no page-count threshold. Neither does the information gain patent, nor the May 2025 purge, which hit sites of every size. google
The honest version of this argument: a 1,000-page site and a 10-million-page site are running the identical three-gate process. The difference isn’t kind; it’s order of magnitude.
A small, well-run site rarely bumps against Gate 1 (its capacity limit is never fully taxed) and clears Gate 2 quickly (its inventory is small enough that Google can just crawl all of it), so the whole process is invisible and resolves in a day.
A large or poorly-run site hits Gate 1 constraints constantly and can spend its entire crawl budget failing Gate 2 on duplicate or low-value URLs before ever reaching content worth evaluating for Gate 3. Scale doesn’t create the gates — it just makes them visible. A small site’s redundant page still fails Gate 3 the same way a large site’s does; it’s simply competing in a much smaller queue, so the failure is quieter and easier to mistake for “crawl budget doesn’t apply to me.”
This is worth stating as its own explicit line in the guide, since it directly counters the “unless you have a million pages, don’t worry about this” framing that my earlier quote already pushes back on from a different angle: crawl budget is a scale problem. The three-gate model underneath it is not — it’s the operating system every URL runs through regardless of how many others are running through it alongside it.
