How to audit whether AI crawlers can access your website

AI access is a runtime question

A robots.txt file can look permissive while a CDN, bot manager, JavaScript challenge, or authentication boundary still blocks the real crawler. The only reliable audit reproduces requests with governed crawler identities and records the actual response chain.

This is why agent access should be measured separately from broad SEO health. Strong technical SEO is a useful foundation, but it does not prove that an AI agent can enter, traverse, and understand the site.

The five layers to test

A useful access audit moves from the network edge toward meaning. Each layer needs its own evidence and failure classification.

• Policy: robots groups, user-agent directives, crawl delays, and conflicting rules.
• Delivery: DNS, TLS, CDN, WAF, rate limits, challenge pages, and HTTP status behavior.
• Navigation: canonical links, pagination, redirects, sitemaps, and reachable page graphs.
• Rendering: server HTML, JavaScript-only content, structured data, and hidden commerce state.
• Comprehension: stable entities, policies, product variants, availability, and machine-readable context.

Evidence before recommendations

Every finding should include the exact request, response status, relevant headers, affected paths, reproduction command, and expected behavior. A configuration guess is not proof.

For changes with production impact, the report should also identify the mutation risk, validation command, canary strategy, and rollback path before any fix is proposed.

What a good result looks like

The final report should tell an operator which agents are allowed, what each agent can actually read, where navigation breaks, which fixes can be automated, and which require explicit policy. That creates an operational backlog rather than another vanity score.