Cloudflare to introduce pay-per-crawl for AI bots
The biggest news in tech this week (which isn’t over yet) is, without a doubt, that Cloudflare is about to introduce a pay-per-crawl model for AI bots—huge in many ways, as let’s not forget that approximately 20% of internet traffic is routed through Cloudflare. I have many thoughts right now, and it will take some time for them to settle. A good analysis and explanation of why this move is needed and is a good first step can be found in yesterday’s article by Dries Buytaert’s, The Web’s Broken Deal with AI Companies, which I recommend everyone read. ...