The platform flagged forty pages as uncited. The content was fine. An edge rule was answering 403 to AI crawlers before any page was read, and nothing in the monitoring stack was set up to notice. Crawler access is the first layer between a page and an answer, and it is the one least often checked.
What did the operator see?
Take a constructed example, not a measured result. A GEO monitoring platform reports, over a fixed set of sixty queries, that a site is cited in 3 and that forty pages it expected to be retrieved are absent. The insight panel suggests rewriting summaries and restructuring headings on those forty pages.
Nothing is wrong with the pages. They are well structured, dated and specific. The operator's first reasonable suspicion is content, and it costs two weeks to test: twelve pages rewritten, the same query set re-run, citations still 3. Crawler access was not on the list of suspects, because no dashboard in the stack reports it.
What was suspected first, and why was it reasonable?
The content. Platforms in this category tell the operator to fix content, and it is the layer an editorial team can change. Profound's homepage describes opening up content AI crawlers cannot currently read, which is an acknowledgement that the problem can sit below the page. Scrunch's homepage describes getting a site ready for AI, and its plan tables list AI agent and bot traffic as a monitored signal.
The measurement that ruled content out was the edge log. Twelve rewritten pages, zero additional citations, and zero additional crawler requests reaching the origin. If the crawlers never reach the page, the page cannot be the cause.
Which layer was failing?
Walk the stack one layer at a time. Start at the resolver and the door.
- DNS and TLS: resolving and valid
robots.txt: permits the major AI crawlers- Edge bot rule: challenges or blocks unrecognised automated clients
- Origin: healthy, responds
200to any request that reaches it - Rendering: server-rendered, readable without scripts
The edge rule is where it ends. The site sat behind a CDN with a bot-management setting turned on at the start of the year to cut scraper load. The rule classified several AI crawlers as automated clients and returned 403 or a challenge page. robots.txt said yes. The edge said no. The file was the wrong layer to consult. An edge rule can also apply by path or by rate: a crawler that fetches forty pages in a minute trips a rate rule that a person never would. Log the rule name with each block, or the review is guesswork.
Correlation is not mechanism, so say it plainly. The block and the missing citations were both present. The measurement that isolates the mechanism is the edge log filtered to the AI crawler user agents: over thirty days, 1,940 requests, of which 1,802 were blocked and 138 passed. Illustrative numbers, but the shape is the general one.
What is the mechanism behind poor crawler access?
A CDN evaluates the request before the origin sees it. If a bot rule matches, the origin never logs the attempt, so the origin logs look quiet and the operator reads that as no interest. The platform's monitoring measures answers, not access, so it sees only the consequence.
Crawler access therefore fails silently in both directions. The edge blocks without telling the origin, and the origin's silence is read as demand being low. The failing layer has no dashboard of its own.
What is the fix, and what does it not address?
Allow the crawlers you have decided to admit. That restores crawler access and nothing more. Read each provider's current documentation for the user-agent strings and the published address ranges, and write the rule against both, since a user-agent string alone is trivially claimed by anyone. Roll it out in the order the evidence supports: admit the crawler classes that feed retrieval first, hold the training class until a policy decision is made, and record which was chosen and why. Keep the rule narrow, log every match, and review it monthly. Re-run the query set two and four weeks after the change, since retrieval sets take time to refresh and an immediate flat result is not evidence against the fix. After the change, the constructed thirty-day figures were 2,050 requests with 1,990 passed, 97 percent.
It does not address whether the pages then get cited. Crawler access is necessary and not sufficient: the pages still need structure, relevance and authority. Fixing access moves a page from impossible to possible, and nothing more.
It also does not address the ownership gap that allowed the rule to sit unreviewed. The bot setting was changed by infrastructure for a good reason, and the GEO work sat with marketing, who never saw the change. Checkpoint GTM treats access as the first item in its programme, checked before any content work and re-checked on a schedule, by the same team that writes the pages. For a company where infrastructure and content sit in different departments, that is the stronger arrangement. A company with one owner for both can run the same check itself.
What check on crawler access would have caught it earlier?
One check, weekly, ten minutes. Filter the edge firewall or bot log to the AI crawler user agents and read the action column. Alert if the blocked share exceeds 10 percent of requests, and record any rule change on the same page as the platform's citation report.
Do not test by presenting a crawler's identity from a browser; read the logs instead. The wrong layer was the file everyone consulted, and the right one was the log nobody opened. This one took two wasted weeks and the cause was a setting, which is the least interesting possible answer.