Cloudflare lets sites separate search crawlers from AI bots

Cloudflare now lets website owners distinguish among search crawlers, AI training bots and agents acting on behalf of users, according to Engadget. Since September 15, new Cloudflare domains default to allowing search crawlers while blocking AI training bots and agents on pages that carry ads.

The controls arrive as AI shopping and browsing tools run into walls on major retail and financial sites. Meta's Muse AI assistant has hit human-verification checks at Walmart, which said the checks were not aimed at the assistant, while Amazon deliberately blocks Muse from shopping on its site, according to TechCrunch.

Shopify has taken a different route: its checkout tools let compatible browser agents update checkout details and submit orders once the buyer confirms.

Agent access draws scrutiny from banks

Financial institutions have raised their own concerns. During JPMorgan Chase's October earnings call, CFO Jeremy Barnum said some agents use shared credentials and screen scraping to transfer funds, describing those methods as not secure.

The legal picture around agents remains unsettled. A federal appeals court vacated a preliminary injunction against Perplexity in August 2026, though its dispute with Amazon over the Comet browser is still unresolved. Cloudflare had previously introduced controls against AI web crawlers.