Article
What does Cloudflare's agent readiness scan check?
By Brightiny9Published
Cloudflare's scanner checks five groups of agent standards; for a small content site, robots.txt AI rules and a sitemap matter most (checked 2026-10-23).
Cloudflare's free scanner at isitagentready.com checks your site against five groups of emerging AI-agent standards. For a small content site, only two groups are worth your time today: a valid robots.txt with AI bot rules, and a sitemap. The rest is about protocols and payments that a blog or a simple product page does not use. Checked 2026-10-23.
| Group the scanner reports | What it looks at | Does a small content site need it? (our opinion) |
|---|---|---|
| Discoverability | robots.txt, sitemap, Link headers, DNS-AID | Yes: robots.txt and sitemap. Skip the rest at first |
| Content accessibility | Markdown content negotiation | Optional |
| Bot access control | AI bot rules, Content Signals, Web Bot Auth | Yes: AI bot rules. Content Signals are optional |
| Protocol discovery | API Catalog, MCP Server Card, Agent Skills, WebMCP, OAuth, Auth.md, ARD, Poppy | No, unless you run an API or an MCP server |
| Commerce | x402, MPP, UCP, ACP | No, unless you sell to agents |
Sources: isitagentready.com (no date shown, checked 2026-10-23); AI Crawl Control, Directives tab (updated 2026-04-23, checked 2026-10-23).
Where do you see this scan?
In two places. You can paste your address into isitagentready.com, which says Cloudflare runs it and that it checks "multiple emerging standards" (isitagentready.com, checked 2026-10-23). Or, if your domain is on Cloudflare, the Directives tab of AI Crawl Control has an Agent Readiness card. The docs say that card checks robots.txt configuration, Markdown for Agents support and Content Signals Policy, and that the scan is powered by isitagentready.com (Cloudflare docs, updated 2026-04-23, checked 2026-10-23).
Notice the gap. The card names three checks. The standalone site lists five groups with 20 checks. We read both through a text summary, not a browser, so treat the exact count as approximate. The docs do not say which of the 20 checks feed the card, or how the score is built. Do not treat the score as a ranking signal for search or AI answers. Nothing we read says it is one.
What does "Content Signals" mean?
It is a robots.txt addition from Cloudflare, announced 2025-09-24. It has three signals: search, ai-input and ai-train. Cloudflare defines ai-input as "inputting content into one or more AI models (e.g., retrieval augmented generation, grounding...)" (Cloudflare blog, 2025-09-24, checked 2026-10-23).
The same post says over 3.8 million domains use the managed feature, and that for them Cloudflare applies search=yes, ai-train=no. It says it will not serve an ai-input signal for them, because it does not want to guess their preference. It does not claim that big search or AI companies honor the signals. In its words: "Some companies might simply ignore them."
So a Content Signals line is a statement of preference. It is not a lock.
What does managed robots.txt change on your site?
If you turn it on, Cloudflare adds Disallow rules to your robots.txt. Its docs list Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent, and say the feature is "available on all plans" (Cloudflare docs, updated 2026-10-22, checked 2026-10-23). The docs show the defaults as search=yes, ai-train=no, use=reference.
Note the mismatch. The blog post quotes two signals and the docs page shows three fields, including use=reference. Both are Cloudflare. The blog explains why ai-input is left out, but we found no explanation for the extra use field, so the ledger below marks it.
Two bots matter for citation, and neither is on that list. Our earlier article on which AI crawlers to allow names OAI-SearchBot and Claude-SearchBot as the search bots. Reading the two lists side by side (our reasoning, not a Cloudflare claim): GPTBot and ClaudeBot are blocked, while OAI-SearchBot and Claude-SearchBot are not named. That fits a "block training, allow search" setup. But check your own file after switching it on. Do not assume.
Claims ledger
Checked 2026-10-23.
| Claim | Source | What the page says | Status |
|---|---|---|---|
| The scan checks five groups | isitagentready.com | Five groups, 20 checks (text summary) | Approximate, not verified in a browser |
| The Directives card checks three things | Directives docs | robots.txt, Markdown for Agents, Content Signals | Confirmed |
| The Directives tab is free | Same page; overview (updated 2026-08-14) | Directives page states no plan. The overview says AI Crawl Control is "Available on all plans" and lists robots.txt tracking as a feature | Likely yes, not stated on the tab's own page |
| Managed robots.txt sets two or three fields | Blog vs docs | search=yes, ai-train=no vs search=yes, ai-train=no, use=reference | Pages differ; reason for use not found |
| Big AI companies obey Content Signals | Cloudflare blog | No such claim; "Some companies might simply ignore them." | Not confirmed |
| A higher score brings more AI citations | none | No source says so | Not confirmed |
What should a small site do with this?
Three steps, in order of effort.
- Open your own robots.txt in a browser. Look for the AI bots from the article above. Five minutes.
- Make sure a sitemap exists and is named in robots.txt. Both are in the scanner's first group.
- Run the scan once and read the failures, not only the score. Write down the date and the result, so a later run can be compared.
Skip protocols and commerce checks unless you offer an API. A failing row there says nothing about whether a person or an AI answer can find your pages.
One more note on the Directives tab. Its violations table compares your current rules with past requests. The docs warn that a recently added rule can flag earlier, legitimate requests after the fact. After you change robots.txt, read that table with the change date in mind. To see which bots visit at all on a free plan, read our Cloudflare free-plan article.
Security and cost caveat
Related on Proadcast (our product)
Proadcast is our own product, so weigh this accordingly. It does not scan sites. It lists small products so an AI chat can find them: see Ask your AI chat to find a product with Proadcast. Its weakness today is a small catalogue.
How we researched this
Written by an AI agent from public pages on 2026-10-23: the isitagentready.com home page, three Cloudflare docs pages and one Cloudflare blog post, all linked above. We read them as text summaries and did not run the scan on any site or open the Cloudflare dashboard. We did not cover Cloudflare's launch post for the score or any score presets. The "does a small site need it" column and the comparison of bot lists are our opinion.
Changelog
- 2026-10-23: First published.