GEO Glossary
LLM Crawlability
LLM crawlability describes whether a service used by an AI system can retrieve a web page and read its useful content. Access is a technical prerequisite for retrieval, not proof that the page will be indexed, cited or recommended.
In depth
Investigate the exact URL that should answer a customer question. Check the final HTTP response, redirects, robots rules, index directives and the content actually returned. A public browser session is not sufficient evidence: a firewall may challenge an automated request, or the server may send an empty application shell before JavaScript runs. Conversely, a successful request using a crawler user-agent string does not prove the real crawler can access the site from its own network. Use server logs and the relevant search inspection tools to close that gap.
For Google, distinguish crawl access from indexing permission. A robots.txt exclusion controls crawling; a noindex directive tells Google to exclude a page after it can retrieve that instruction. Check both the HTML robots meta tag and the X-Robots-Tag response header. Also inspect the canonical destination: a technically accessible URL may point search engines toward a different page.
A practical example is a public product guide that returns HTTP 200 but inherits noindex from a staging template. More links or extra paragraphs do not remove that exclusion. Correct the directive on the intended public page, verify the deployed response and request inspection. Keep account pages and private reports outside the search index. After access is fixed, test whether the content actually answers the target question; accessibility alone says nothing about its usefulness.
Frequently Asked Questions
- Does HTTP 200 mean a page is indexable?
- No. The response can still contain noindex, point to another canonical URL, or lack useful content. Evaluate the response headers and rendered page together.
- Should I block a noindex page in robots.txt?
- Google needs to crawl a page to see its noindex directive. Blocking that crawl can prevent the directive from being read. Private information should be protected with authentication rather than relying on robots rules.
- Can I guarantee an AI citation by allowing crawlers?
- No. Retrieval access and source selection are separate. Measure actual citations and inspect the answer context instead of treating a successful technical check as visibility.
Diagnose the right layer
| Observation | Next check |
|---|---|
| 403 or a challenge page | Firewall rules and verified crawler access |
| 200 with noindex | Whether the page should be public and searchable |
| 200 with useful, indexable content | Indexing status, relevance and actual answer observations |
Sources & further reading
- Technical requirements — Google Search Central
- Block indexing with noindex — Google Search Central
- Introduction to robots.txt — Google Search Central
Next step
Check your AI visibility with Prompt Monitoring
Start with the free AI crawlability check to see whether AI crawlers can read your site.