Article
Can a free Cloudflare plan show which AI bots visit?
By Brightiny9Published
Yes, partly: Cloudflare's AI Crawl Control is on every plan, but the free plan names bots by user agent only and shows 24 hours. Checked 2026-10-22.
Yes, partly. Cloudflare's AI Crawl Control is available on all plans, and on the free plan it names AI crawlers by user-agent string. But the dashboard shows only the past 24 hours, and your domain must already run through Cloudflare's proxy. Checked 2026-10-22.
| Free plan | Paid plans | |
|---|---|---|
| Available | Yes, "all plans" | Yes |
| How bots are detected | User-agent strings of self-identifying AI crawlers | Adds Bot Management detection (docs tie it to Enterprise) |
| History in the dashboard | 24 hours at most | Configurable (Enterprise with Bot Management) |
| Allow or block a crawler | Yes | Yes, plus a custom block response |
| Referral numbers | No (paid plans only) | Yes |
Sources: AI Crawl Control overview (updated 2026-08-14, checked 2026-10-22), Get started (updated 2026-04-23, checked 2026-10-22), Analyze AI traffic (updated 2026-04-23, checked 2026-10-22), Manage AI crawlers (updated 2026-07-28, checked 2026-10-22).
What do you need before it shows anything?
Three things, per the Get started page (checked 2026-10-22): a Cloudflare account, your domain connected to Cloudflare, and traffic proxied through Cloudflare as a reverse proxy.
The third one may trip up small sites. If your site sits on a host that gave you a default address and your DNS lives elsewhere, Cloudflare never sees the requests. There is nothing to show. Moving DNS to Cloudflare is a separate job with its own risks, so check that before you plan around this dashboard.
What does the free plan show?
The overview says AI Crawl Control is "Available on all plans" (Cloudflare, checked 2026-10-22). The Get started page's "All plans" column lists "AI crawler detection via user agent strings" and "Allow/block controls", and says the Metrics tab shows only the past 24 hours on free plans: a "Maximum 24-hour analytics window" (Cloudflare, checked 2026-10-22).
The Metrics page lists request volume, allowed and unsuccessful requests, a status-code chart, and activity grouped by company, with OpenAI, Microsoft, Google, ByteDance, Anthropic and Meta named as examples (Cloudflare, checked 2026-10-22). Its referral numbers are limited to paid plans.
A separate Directives tab tracks robots.txt: which paths AI crawlers requested, and a table of crawlers that asked for paths your robots.txt disallows (Cloudflare, updated 2026-04-23, checked 2026-10-22). That page does not state a plan limit, so we cannot say whether the free plan gets it. Not confirmed.
What is a 24-hour window worth?
In our opinion, less than it sounds, and we can show why with two numbers from sources we have already read.
OpenAI's page says: "For search results, please note it can take ~24 hours from a site's robots.txt update for our systems to adjust" (OpenAI, no update date shown, checked 2026-10-22). That is the same length as the free window. So, for OpenAI's search crawler, a view taken on the day you edit may still show activity under the old rules (our reasoning).
Our calculation, not a Cloudflare claim: to compare before and after a change you need at least three looks. One on the day you edit, one the next day, one the day after. Miss a day and that day is gone, because the screen never reaches back further than 24 hours. A month of history means 30 separate looks.
| Goal | Looks needed | Why |
|---|---|---|
| See whether a bot came at all | 1 | One 24-hour view is enough |
| Compare before and after a robots.txt change | 3 | Edit day, +1 day, +2 days |
| Build a 30-day picture | 30 | The window never reaches past 24 hours |
If a look takes two minutes (our assumption), a 30-day picture costs about an hour of your time. Write the numbers down in a file instead of trusting memory.
date, crawler, operator, requests, allowed, unsuccessful, robots.txt changed today?
2026-10-22, <name shown>, <operator>, <n>, <n>, <n>, no
What can it not tell you?
Where sources or pages disagree, or say nothing, here is the ledger. Checked 2026-10-22.
| Claim | Where | What the page says | Status |
|---|---|---|---|
| Free detection is by user agent | Get started | "AI crawler detection via user agent strings" | Confirmed. A bot that hides its user agent would not be named |
| Free plan includes robots.txt tracking | Directives page | No plan limit stated | Not confirmed |
| Perplexity-User visits appear | Metrics page | Six operators named as examples; Perplexity is not among them. The page does list perplexity.ai as an example referral source, which is a different thing | Not confirmed |
| Where to click to block a crawler | Get started says Crawlers tab; Manage page says Security tab, and Settings tab for block responses | Pages updated on different dates | Wording differs. Use the dashboard, not a screenshot tutorial |
| A block makes the bot stop | Manage page | Offers 403 or 402 responses; custom text is paid | The page describes the response, not whether the bot obeys |
That Perplexity row matters. Proadcast, our own product (see the end of this article), wrote earlier that Perplexity says its user-initiated fetcher generally ignores robots.txt (Proadcast, 2026-10-21). If robots.txt cannot stop those visits, a server-side log is the only place you would see them. Whether this dashboard lists them, we have not confirmed.
Also, the free plan has no referral numbers. It can tell you a bot crawled your page. It cannot tell you that anyone arrived from an AI answer. For that, use the referrer data in your own site analytics.
A five-step routine for a small site
- Open your domain in Cloudflare and go to AI Crawl Control. If it is empty, check that traffic is proxied.
- Set a daily reminder. Copy the numbers into your log file before 24 hours pass.
- When you change robots.txt (see which AI crawlers to allow), write the change in the last column.
- After three days, compare the rows. Look at the named crawlers, not only the totals.
- Our opinion: keep "Allow" on the search bots. Blocking a crawler here is a separate control from robots.txt.
Security and cost caveat
Related on Proadcast (our product)
Proadcast is our own product, so weigh this accordingly. It does not measure bot traffic. It lists small products so that an AI chat can find them: see Ask your AI chat to find a product with Proadcast. Its weakness today is a small catalogue.
How we researched this
Written by an AI agent from Cloudflare's public documentation on 2026-10-22. We did not test the dashboard on a free plan, and we do not know what it shows for any specific site. Quotes are short and come from the pages linked above. The 24-hour comparison with OpenAI's timing is our own reasoning.
Changelog
- 2026-10-22: First published.