AI Crawler Activity on Small Business Websites: Evidence from Server Logs of 924 WordPress Sites, September to October 2026

Most published measurements of AI crawler traffic come from large content delivery networks and describe the web as a whole, where a small number of heavily crawled properties dominate the totals. We report what AI crawlers do on small and mid-sized websites, using server-side request logs from 924 WordPress sites logged on at least 25 of the 30 days from 6 September to 5 October 2026. On the median site, AI crawlers made 3.1 page requests for every Googlebot request (bootstrap 95% CI 2.8 to 3.4), and 79% of sites received more requests from AI crawlers than from Googlebot. Weighting each site equally, AI crawlers made 40.7% of identified bot page requests, level with all search engines combined (40.8%). Meta-ExternalAgent was the most active single AI crawler (21.6% of the average site's AI requests), while OpenAI's three crawlers together held the largest operator share (27.2%). On the average site, 69.9% of AI crawler requests were for model training, 18.8% for AI search and 11.3% user-triggered. Crawl volume is highly concentrated: the busiest 1% of sites received 75% to 87% of the requests from each of the four largest training crawlers, against 42% for Googlebot, which is why pooled and per-site statistics diverge sharply. On sites that received ChatGPT visitors, OpenAI's crawlers made a median 33.7 page requests per visitor, far below network-wide pooled ratios. On a fixed cohort of 477 sites, the AI-to-Googlebot ratio held between 2.2 and 2.4 to 1 in every month from June to September 2026. We describe the robustness checks behind each figure and the limits of a plugin-based sample.

A continuously updated version is published at https://www.lovedby.ai/blog/ai-crawler-statistics