If you’ve been watching Google Search Console and seeing more URLs pile up under “Crawled – currently not indexed,” you’re not imagining a trend. And the fix isn’t always technical.
Marie Haynes recently published a sharp breakdown of this exact issue — Why Your Pages Are Stuck In Crawled-Currently Not Indexed & What To Do About It on Search Engine Journal, pulling in commentary from Google’s own John Mueller and Martin Splitt. The short version: a lot of pages sitting in that bucket aren’t broken. They’re just redundant. Well-written, accurate, technically sound, and completely unnecessary, because Google already has ten other pages saying the same thing.
That distinction matters, and it’s worth understanding why it’s happening now, not just five years ago.
Links used to be enough. They’re not the whole story anymore.
For most of Search’s history, the model was simple: crawl broadly, store cheaply, let links sort out what deserves to rank. Links were a proxy for trust, and indexing itself was inexpensive enough that source #11, #50, or #200 on a topic got a seat in the index even if it said nothing new. Ranking was the filter. Indexing wasn’t.
That’s shifted. Google and every AI lab building answer engines now run content through synthesis: AI Overviews, AI Mode, RAG pipelines. That’s a fundamentally more expensive step than storing a URL. It means reading the page, extracting what it contributes, and deciding whether it actually changes the synthesized answer. Once an engine has already built a solid answer from ten sources, an eleventh source saying the same thing adds zero information to that answer — but costs exactly as much to crawl, embed, and maintain as it always did.
That’s a bad trade at web scale. And it’s a big part of why “good content” isn’t automatically “indexed content” anymore.
Links aren’t devalued here as they still establish trust and authority, especially for competitive or YMYL topics. What’s changed is the gate before ranking even happens: does this page deserve to be read and processed at all? That gate increasingly runs on information gain, not link count.
Why “gap analysis” tools can quietly make this worse
A lot of GEO/AEO tools work by comparing your content to whatever’s currently cited in an AI Overview, then handing you a list of “gaps” — topics or phrases the cited sources mention that you don’t.
The problem: that gap list is built entirely from what’s already out there. Close it, and you match the existing consensus — you don’t add to it. You become a well-optimized version of source #11. If everyone in a niche runs the same class of tool against the same cited sources, everyone converges toward the same checklist, and the whole category homogenizes. More comprehensive, sure. But comprehensive in exactly the same way as every competitor.
Gap analysis still has a use — confirming you’re not missing baseline, accurate coverage. It just can’t be where your differentiation comes from. It structurally can’t surface what only you have: first-party data, direct experience, a defensible contrarian take, specificity nobody else bothered to capture.
A simple way to check this yourself
Following Marie’s approach, the actual test isn’t a static rubric — it’s a live comparison. Two places worth checking:
1. Top-ranking / cited competitors. Search your target query, open the AI Overview or top few ranking pages, and ask an AI model (Gemini, Claude, ChatGPT) to evaluate them against Google’s own helpful-content questions: Does this offer original information, research, or analysis? Insight beyond the obvious? If it draws on other sources, does it add real value rather than just rewriting them? Then paste in your own page and ask where it falls short against that same live bar — no fixes yet, just diagnosis.
2. Your own domain. This is the secondary test worth adding: run the same check within your own site. If you’ve published ten articles on adjacent topics over the years, there’s a good chance some of them are saying the same thing in different words — redundant with each other, not just with competitors. That’s the same commodity problem, just self-inflicted. A quick same-domain pass “does this new page say anything my existing pages don’t already say” catches internal redundancy before you even get to the competitive landscape.
Simple prompt logic for both:
Compare [this page] against [the top-ranking pages for this query]
/ [these other pages on our own site].
Does it add original information, research, analysis, or first-hand
experience that the comparison pages don't already have? Or does it
mostly restate what's already covered?Run it against competitors to check external redundancy. Run it against your own archive to check internal redundancy. Both answer the same underlying question: is this page worth the compute to keep indexed?
The bottom line
Nobody’s saying links stopped mattering. What’s changed is the layer underneath them — the point where an engine decides whether a page is even worth reading in the first place. As that decision increasingly runs through synthesis rather than a link graph, information gain becomes the currency that keeps a page in the index at all. Everything else, links included, only gets to matter after that.
Credit to Marie Haynes for the original reporting and framework this post builds on.
