State of AI crawler access, October 2026 (dataset)

Baseline edition of State of AI crawler access by QueryRev Research. The dataset holds the robots.txt rules for 11 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, Bytespider) and for the catch-all group, and the presence and structure of llms.txt, for a fixed panel of 9,973 domains: the first 10,000 places of Tranco list N26XW (created 14 October 2026), less 2 invalid entries and 25 domains of companies that build AI models. The domains were read on 15 October 2026 with the QueryRevBot crawler, which follows robots.txt as RFC 9309 describes.

Files: domains.csv (one row per domain, 41 columns), figures.csv (each figure with its count and three denominators, overall, by rank band and by country-code ending) and figures.json (the counts behind the study page). The three denominators are the panel (9,973), the domains whose robots.txt could be read (6,294) and the domains whose robots.txt names an AI crawler (1,339).

Limits: 3,679 domains gave no readable robots.txt and are marked as not read, not as having no file. robots.txt is a request, not enforcement. The scope is the 11 crawlers listed, not every AI crawler. One vantage point.

The panel is fixed so that later editions can be compared with this one. Study page, method and limits: https://queryrev.com/research/ai-crawler-access?utm_source=zenodo&utm_medium=social&utm_campaign=nov26-zn-01#method

Cite as: QueryRev Research. "State of AI crawler access, October 2026." QueryRev AI, 15 October 2026. https://queryrev.com/research/ai-crawler-access