Cloudflare Takes Stronger Stance Against AI Scraping, Blocking Crawlers by Default

Cloudflare has taken a significant step in blocking AI scrapers by default, shifting the balance of power in the ongoing struggle between web publishers and AI companies. As of July 1, 2025, Cloudflare automatically blocks AI crawlers for its customers, and introduces a Pay Per Crawl program to allow publishers to charge AI companies for access to their content. AI crawlers, which scrape web data at high frequencies, can overwhelm servers, causing disruptions or even downtime. This has particularly affected news outlets, which have been vocal about wanting compensation for the use of their content. Cloudflare’s enhanced blocking tools now help identify even ‘shadow’ scrapers that are harder to detect. The company employs a mix of behavioral analysis, fingerprinting, and machine learning to distinguish harmful bots from legitimate ones. Many AI companies have previously bypassed standard anti-scraping measures like robots.txt, but Cloudflare’s new default could hinder such practices. Additionally, the introduction of the Pay Per Crawl program aims to force AI companies into negotiations with content creators, shifting the power dynamic and potentially leading to better licensing deals. Though participation in the program is still in beta, some startups, like ProRata’s Gist.AI, are already onboard. The full impact of Cloudflare’s decision will depend on how widely it is adopted and whether major AI companies engage in the new payment model.