Cloudflare Introduces Default Blocking of AI Crawlers and Pay-Per-Crawl Option for Publishers

Cloudflare, a major internet architecture provider, has announced that it will now block known AI web crawlers by default. This move is aimed at preventing AI crawlers from accessing content without permission or compensation. New domain owners will be asked whether they wish to allow AI scrapers, and Cloudflare will also provide a ‘Pay Per Crawl’ program, allowing publishers to set fees for AI companies to access their content. This program is currently available only to a select group of publishers, but Cloudflare plans to expand it further. The company has already been helping domain owners block AI bots for some time, allowing websites to restrict access to AI crawlers, even those that do not follow the robots.txt file. Additionally, Cloudflare has introduced an ‘AI Labyrinth’ feature to deter crawlers. Many major publishers, including The Associated Press, The Atlantic, and Fortune, have supported Cloudflare’s new approach to AI crawling. Cloudflare CEO Matthew Prince highlighted the growing trend of AI chatbots being used instead of search engines, and stressed the importance of protecting original content while fostering innovation in AI technology.