Key Takeaways
- •GPTBot grew from 5 percent to 30 percent of AI crawler traffic on Cloudflare's network between May 2024 and May 2025.
- •Meta's bots generate 52 percent of AI crawler traffic on Fastly's network, more than Google and OpenAI combined.
- •About 80 percent of AI bot crawling feeds model training, not live search or user requests.
- •Anthropic crawled 38,065 pages per referral visit in July 2025, while Google's ratio was 5.4 to 1.
- •GPTBot made 569 million monthly fetches on Vercel's network, about one eighth of Googlebot's 4.5 billion.
- •No major AI crawler renders JavaScript, so client-side-only content is invisible to them.
- •PerplexityBot requests grew 157,490 percent and ChatGPT-User requests grew 2,825 percent in the year to May 2025.
- •62 to 75 percent of top US and UK news sites now block the main AI training crawlers.
- •More than one million Cloudflare customers block AI crawlers, and blocking became the default for new Cloudflare domains on July 1, 2025.
- •User-action AI crawling, where agents fetch pages on a user's behalf, grew more than 15x during 2025.
Who Is Crawling the Web
Bots now account for close to a third of all web traffic, and a handful of AI companies dominate the AI crawler share, though each network measures a different slice of the web.
- 01
About 30 percent of global web traffic comes from bots, according to Cloudflare's 2025 crawler analysis.
Source: Cloudflare, 2025
- 02
30 percent of AI-only crawler traffic on Cloudflare's network came from OpenAI's GPTBot in May 2025, up from 5 percent a year earlier.
Source: Cloudflare, 2025
- 03
21 percent of AI crawler traffic came from Anthropic's ClaudeBot and 19 percent from Meta-ExternalAgent in Cloudflare's May 2025 data.
Source: Cloudflare, 2025
- 04
96 percent growth in Googlebot requests between May 2024 and May 2025 pushed Google to about half of all combined search and AI crawler traffic on Cloudflare.
Source: Cloudflare, 2025
- 05
52 percent of AI crawler traffic on Fastly's network came from Meta's bots, more than Google's 23 percent and OpenAI's 20 percent combined.
Source: Fastly, 2025
- 06
Almost 80 percent of all AI bot traffic observed by Fastly from mid-April to mid-July 2025 was crawling, with real-time fetcher bots making up the rest.
Source: Fastly, 2025
- 07
Nearly 90 percent of AI crawler traffic observed by Fastly hit North American sites, with Europe, Asia, and Latin America seeing far smaller shares.
Source: Fastly, 2025
Crawl Volumes and Behavior
AI crawlers already fetch pages at a scale comparable to a meaningful fraction of Googlebot, but they crawl inefficiently and none of them execute JavaScript.
- 08
569 million requests per month came from GPTBot on Vercel's network in late 2024, with Anthropic's Claude crawler at 370 million.
Source: Vercel, 2024
- 09
Nearly 1.3 billion combined monthly fetches from GPTBot, Claude, AppleBot, and PerplexityBot equaled about 28 percent of Googlebot's 4.5 billion on Vercel's network.
Source: Vercel, 2024
- 10
24.4 million monthly fetches came from PerplexityBot on Vercel, a small fraction of GPTBot's volume.
Source: Vercel, 2024
- 11
None of the major AI crawlers from OpenAI, Anthropic, Meta, ByteDance, or Perplexity rendered JavaScript in Vercel's testing, so client-side content is invisible to them.
Source: Vercel, 2024
- 12
34.82 percent of ChatGPT's crawler fetches on Vercel hit 404 pages, and Claude showed a similar 34.16 percent, pointing to inefficient URL selection.
Source: Vercel, 2024
- 13
305 percent growth in GPTBot requests between May 2024 and May 2025 lifted its share of combined search and AI crawler traffic from 2.2 percent to 7.7 percent.
Source: Cloudflare, 2025
- 14
2,825 percent growth in ChatGPT-User requests and 157,490 percent growth in PerplexityBot requests occurred between May 2024 and May 2025 on Cloudflare's network.
Source: Cloudflare, 2025
- 15
More than 39,000 requests per minute was the peak rate Fastly observed from real-time fetcher bots such as those serving ChatGPT and Perplexity.
Source: Fastly, 2025
- 16
Over 15x growth in user-action AI crawling during 2025 was recorded in Cloudflare's Radar Year in Review, driven by agents fetching pages on behalf of users.
Source: Cloudflare, 2025
Training vs Search vs User Actions
Most AI crawling still feeds model training rather than live search, which matters because training crawls give site owners nothing back.
- 17
About 80 percent of AI bot crawling in the year to mid-2025 was for model training, versus 18 percent for search and 2 percent for user actions.
Source: Cloudflare, 2025
- 18
82 percent was the training share in the six months to July 2025, up from 72 percent a year earlier, while search crawling fell to 15 percent.
Source: Cloudflare, 2025
- 19
Nearly three quarters of user-action AI bot traffic in July 2025 came from OpenAI's ChatGPT-User agent alone.
Source: Cloudflare, 2025
- 20
Nearly half of all observed AI crawling activity in early August 2025 came from just two bots, ClaudeBot and GPTBot.
Source: Cloudflare, 2025
Crawl-to-Referral Ratios
AI companies take far more pages than they send back as visitors, and the gap between crawling and referral traffic is the core economic complaint publishers have.
- 21
38,065 pages were crawled by Anthropic for every referral visit it sent in July 2025, down 86.7 percent from 286,930 to 1 in January.
Source: Cloudflare, 2025
- 22
1,091 pages were crawled by OpenAI per referral in July 2025, nearly unchanged from 1,217 to 1 in January.
Source: Cloudflare, 2025
- 23
194.8 pages per referral was Perplexity's July 2025 ratio, up 256.7 percent from 54 to 1 in January but still the best among major AI answer engines.
Source: Cloudflare, 2025
- 24
5.4 pages per referral was Google's crawl-to-refer ratio in July 2025, showing how much more traffic classic search returns than AI platforms.
Source: Cloudflare, 2025
- 25
18 pages crawled per visitor sent is Google's current ratio according to Cloudflare CEO Matthew Prince, compared with 2 to 1 a decade ago, while he put OpenAI at 1,500 to 1.
Source: Axios via Techmeme, 2025
- 26
2,500 pages per referral was Anthropic's ratio for news and publishing sites in August 2025, versus 152 to 1 for OpenAI and 32.7 to 1 for Perplexity in the same industry.
Source: Cloudflare, 2025
Blocking and robots.txt Trends
Blocking AI crawlers went from a niche stance to the default for large publishers within two years, and Cloudflare now blocks AI crawlers by default for new domains.
- 27
48 percent of the most widely used news websites across ten countries were blocking OpenAI's crawlers by the end of 2023, while 24 percent blocked Google's AI crawler.
Source: Reuters Institute, 2023
- 28
79 percent of top US news sites blocked OpenAI's crawlers at the end of 2023, versus just 20 percent in Mexico and Poland.
Source: Reuters Institute, 2023
- 29
35.7 percent of the top 1,000 global websites were blocking GPTBot by August 2024, up from about 5 percent when the crawler launched in August 2023, per Originality.ai tracking.
Source: PPC Land (Originality.ai data), 2024
- 30
22.1 percent of top 1,000 sites blocked Common Crawl's CCBot and 13.6 percent blocked Google-Extended in the same August 2024 tracking.
Source: PPC Land (Originality.ai data), 2024
- 31
75 percent of 100 top US and UK news sites blocked CCBot in 2025, with ClaudeBot at 69 percent, PerplexityBot at 67 percent, GPTBot at 62 percent, and Google-Extended at 46 percent.
Source: BuzzStream, 2025
- 32
79 percent of top news sites blocked at least one AI training bot in BuzzStream's 2025 study, and only 18 percent blocked none at all.
Source: BuzzStream, 2025
- 33
More than one million Cloudflare customers chose to block AI crawlers after the company introduced a one-click option in September 2024, before blocking became the default for new domains on July 1, 2025.
Source: Cloudflare, 2025
- 34
About 14 percent of the top 10,000 domains had robots.txt rules targeting AI bots as of Cloudflare's 2025 analysis, and GPTBot, ClaudeBot, and CCBot were the most frequently fully disallowed agents.
Source: Cloudflare, 2025
How to Cite This Page
APA: Best AEO Tools. (2026). AI Crawler Statistics 2026. Retrieved from https://bestaeotools.com/statistics/ai-crawler-statistics
MLA: "AI Crawler Statistics 2026." Best AEO Tools, 2026, https://bestaeotools.com/statistics/ai-crawler-statistics.
Chicago: Best AEO Tools. "AI Crawler Statistics 2026." Accessed July 2026. https://bestaeotools.com/statistics/ai-crawler-statistics.
Frequently Asked Questions
It depends on the network measuring it. On Cloudflare, OpenAI's GPTBot led AI-only crawlers with 30 percent of traffic in May 2025, followed by ClaudeBot at 21 percent and Meta-ExternalAgent at 19 percent. On Fastly's network, Meta's bots dominated with 52 percent of AI crawler traffic, versus 23 percent for Google and 20 percent for OpenAI.
Sources
Every statistic on this page links to the publication where the number appears. Data comes from 11 sources.
- Cloudflare: https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/
- Fastly: https://www.fastly.com/press/press-releases/new-fastly-threat-research-reveals-ai-crawlers-make-up-almost-80-of-ai-bot
- Vercel: https://vercel.com/blog/the-rise-of-the-ai-crawler
- Cloudflare: https://blog.cloudflare.com/radar-2025-year-in-review/
- Cloudflare: https://blog.cloudflare.com/crawlers-click-ai-bots-training/
- Cloudflare: https://blog.cloudflare.com/ai-crawler-traffic-by-purpose-and-industry/
- Axios via Techmeme: https://www.techmeme.com/250620/p2
- Reuters Institute: https://reutersinstitute.politics.ox.ac.uk/how-many-news-websites-block-ai-crawlers
- PPC Land (Originality.ai data): https://ppc.land/top-websites-increasingly-block-ai-web-crawlers-amid-privacy-concerns/
- BuzzStream: https://www.buzzstream.com/blog/publishers-block-ai-study/
- Cloudflare: https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/
Related Statistics
AI Citation Statistics 2026
Which pages, formats, and domains AI engines actually cite, how often citations change, and what correlates with getting picked.
Answer Engine Optimization Statistics 2026
AI referral traffic growth, conversion quality, budget shifts, and the AEO tool market, with every number linked to its source.
Perplexity Statistics 2026
Query volume, funding, the Comet browser, and citation behavior, with every number linked to the publication where it appears.
AI-Generated Content Statistics 2026
How much new content is AI-written, how it performs in Google, how marketers use it, and whether detectors actually work.
Want to Show Up in AI Answers?
Compare the tools that track and improve your visibility in ChatGPT, Perplexity, and Google AI Overviews.
Compare AEO Tools