Block training, keep answers? robots.txt rules for AI bots on 24,419 domains (September 2026)
Authors/Creators
Description
Trakis Study 1. Of the sites that keep an AI company's training bot out in robots.txt, how many also keep out the same company's bots that answer people: its search, and the pages its assistant opens when a person asks?
The robots.txt files of 24,419 domains from the Tranco list of 13 October 2026 (the top 10,000; 10,000 drawn at random from ranks 100,000 to 1,000,000; every .ro domain in the list), as Common Crawl archived them in its September 2026 crawl (CC-MAIN-2026-39). No request went to the sites. 13,938 were readable, and each was judged for 14 crawlers with the open-source robots-parser library.
Main results, all samples together: GPTBot is kept out of the whole site by 11.4% of readable sites, OAI-SearchBot by 3.5%. Of the 1,331 sites that block GPTBot by name, 17.1% also block OAI-SearchBot. 609 of them (45.8%) do it with Cloudflare's managed robots.txt, which names no answer bot; of the 722 that wrote the rule themselves, 31.3% also block OAI-SearchBot and 47.0% ChatGPT-User.
Files: totals.csv and pairs.csv for every sample; top-10k.csv, site by site, for the 10,000 highest-ranked domains only; code.zip, the scripts that reproduce the three tables byte for byte from Common Crawl's archive. README.md gives the method, the definitions, the checks and the limits.
Files
code.zip
Files
(1.8 MB)
| Name | Size | |
|---|---|---|
|
md5:5bebd813647399f8b088a193541c2376
|
22.9 kB | Preview Download |
|
md5:4d57ab8e2edd1ca5fd5ffb3481f8a403
|
1.8 kB | Preview Download |
|
md5:be48a32c04c0fcc2671a55b4450bb2ec
|
5.2 kB | Preview Download |
|
md5:5faa276eddb300144c62442a1cef9339
|
1.7 MB | Preview Download |
|
md5:9367372125f797bb9620a2c86ad85c75
|
3.7 kB | Preview Download |
Additional details
Related works
- Is supplement to
- Publication: https://trakis.io/research/block-training-keep-answers (URL)