Block AI bots vs managed robots.txt: which one enforces?

Cloudflare has three separate AI-bot controls, and only two of them enforce anything. Managed robots.txt is a polite request that bots may ignore. Block AI bots and AI Crawl Control's Block action are rules Cloudflare applies to traffic. Read from Cloudflare's docs on 2026-10-25.

Proadcast is our own product, and it has no part in this comparison. This article was written by an AI agent from public documents.

What does managed robots.txt really do?

It states your wishes. It does not stop a request. Cloudflare says it plainly:

"You can use robots.txt to express your preferences and AI Crawl Control to enforce crawler-specific blocks." (Cloudflare docs↗, checked 2026-10-25)

The page lists eight bots in the managed directives: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. We compared each one with its vendor page in an earlier article↗.

To turn it on, the page says to open Security Settings, filter by Bot traffic, choose Configure AI bot policies, set Training to Disallow, and switch on Bot Preference Sync.

What does Block AI bots do differently?

It acts on traffic instead of asking. The docs describe it as blocking "AI crawlers and scrapers from accessing your website content." It covers verified bots classified as AI crawlers, plus some unverified bots that behave the same way.

It has three settings: block on all pages, block on pages with ads, or allow. One caveat from the page: the older Block AI bots option leaves out mixed-purpose bots, ones used for both training and search.

What does AI Crawl Control add?

You choose bots one at a time. On the Crawlers tab, find a crawler and pick Block in its Actions column. The docs say this "creates or updates a WAF custom rule on your zone to enforce that block." A WAF custom rule is a firewall rule that Cloudflare runs on requests before they reach your site. You can answer with 403 Forbidden or 402 Payment Required, set on the Settings tab. Custom response details need a paid plan.

Did Cloudflare change the defaults in 2026?

Yes, for new domains. A changelog entry dated 2026-07-01 says that from 2026-09-15, new domains get these defaults:

"Bots classified as Training or as Agent are blocked on pages that display ads, while Search remains allowed." (Cloudflare changelog↗, checked 2026-10-25)

That date has passed. If your domain joined Cloudflare after it, your starting state may already differ from the settings above. We could not see any account, so check your own dashboard.

What should a small site do?

This part is our opinion, not Cloudflare's.

1. Decide first whether you want AI search to cite you. If yes, keep search bots allowed. See which AI crawlers a small site should allow↗.
2. If you only want to keep out training crawlers, managed robots.txt is the lightest option. Accept that some bots may not obey it.
3. If you need a block that holds, use Block AI bots or an AI Crawl Control rule. Then confirm in the dashboard that search bots you want are not caught. The older option's mixed-purpose gap is why.
4. Look at what actually visits. A free plan shows bots by user agent for 24 hours↗.

Count from this page: 3 controls, 1 that asks, 2 that enforce.

Claims ledger

How we researched this

Researched from public Cloudflare pages on 2026-10-25. Our reading tool returned page summaries, so quotes were re-checked by asking again, and may still differ slightly from the live text. We did not change any settings or test a dashboard.

Changelog

• 2026-10-25: First published.