Follow your favourite podcasts, listen offline and in the car with CarPlay and Android Auto, and always pick up where you left off. Free to try.
Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default
Today Cloudflare's new crawler defaults take effect, sorting automated visitors into three declared purposes, Search, Agent, and Training, and blocking the last two by default on ad supported pages. Chris and Laura dig into the mixed use crawler problem that puts Googlebot, Applebot, and Bingbot on notice, the crawl to referral ratios showing some AI companies taking thousands of pages for every visitor they send back, and the Pay Per Use marketplace meant to be the carrot behind the stick. They also ask the harder question: is a single private infrastructure company setting the terms of machine access to the open web a fix, or a new problem wearing a friendlier face?
Hosted by Chris and Laura.
The DX Today Podcast brings you daily deep dives into the most consequential stories in the AI ecosystem.
