Cloudflare pushes AI crawlers toward paid access to publishers

TechCrunch reports that Cloudflare is giving AI companies until September 15 to separate search crawlers from crawlers used for training models and powering agents. Publishers will be able to block AI-oriented scraping by default unless they approve access or negotiate payment. The change escalates the fight over whether AI companies can keep treating the open web as free input while publishers lose referral traffic.

Key details: Cloudflare wants AI firms to separate search crawlers from AI training and agent crawlers, Publishers can default-block AI crawlers unless they approve access, The deadline for crawler separation is September 15.

Why it matters: Crawler pricing and blocking rules are becoming the practical enforcement layer for the AI copyright and publisher-economics fight.