Cloudflare Puts a Tollgate in Front of AI Bots: the End of Free Scraping of Other People's Content

Cloudflare, through whose servers a large part of the world's internet traffic passes, has announced a rule that hits the big players in artificial intelligence directly. From September 15, 2026, it will by default block so-called "mixed" web robots from sites that display ads - the bots that simultaneously search, act as AI agents, and gather content to train models. The rule applies to new clients, to new sites of existing clients, and to all free users.

Behind the technical language is a simple fight over money. For years, AI companies have been scraping other people's texts, images and data to feed their models - without paying anything to those who created the content. Cloudflare is now putting up a tollgate. The accompanying tools say it openly: Pay Per Crawl, a marketplace through which sites can charge AI bots for every data grab, and Pay Per Use, a model that charges when content creates value. The first partners are Ceramic.ai and You.com.

The figures the company cites explain why this happened at all. Bot traffic recently overtook human traffic on the internet for the first time - more than half of the world visiting sites are no longer people. Sharper still: over 50 percent of AI-robot traffic goes to re-downloading pages that haven't changed at all. In other words, publishers' servers are paying to let someone else's machines read the same thing over and over, with no benefit to anyone except the AI company.

"Now that most internet traffic is non-human, we have to go further and act faster to create a sustainable ecosystem," said Matthew Prince, co-founder and CEO of Cloudflare. The goal, he says, is to force mixed robots to separate search from AI agents and training - three things bundled together until now precisely so no one could tell who takes what.

There's also a detail that reveals who's really in the crosshairs. According to Cloudflare, Google Search has access to about twice as much information as its competitors - because its robot searches and gathers for training at the same time, so it's hard to block one without losing the other. This is a move that hands small publishers a weapon against incomparably more powerful opponents. Whether it withstands the pressure of companies with budgets bigger than entire economies remains to be seen - but for the first time, someone who controls the internet's pipes has stood on the side of those who write the content, not those who take it for free.