A robots.txt that allows every AI crawler guarantees nothing when the site sits behind Cloudflare: the proxy can answer with a challenge or a 403 before the crawler reaches your server, and none of that shows in the file. This page explains which Cloudflare settings stop assistant crawlers, how to observe it from outside, and how to let through the ones you want.
1. Orange or grey cloud
What the vendor documents. When a DNS record is proxied, "Cloudflare sits between your visitors and your server, optimizing, caching, and protecting traffic along the way". When it is DNS-only, "Cloudflare responds with your server's actual IP address and does not route HTTP/HTTPS traffic through its network". Product configurations, "such as WAF rules, caching, and redirect rules", apply to proxied traffic only (Cloudflare, proxy status).
What we recommend. Start there: if the cloud is grey, no Cloudflare setting can block a crawler and the problem is elsewhere. If it is orange, everything below applies, including to subdomains nobody remembers. The status is read in the DNS tab, record by record.
2. Bot Fight Mode
What the vendor documents. Bot Fight Mode "issues computationally expensive challenges that force the requesting client to perform CPU-intensive calculations" on traffic matching known bot patterns. It "cannot be customized, adjusted, or reconfigured via WAF custom rules" on the Free plan, and "you cannot bypass or skip Bot Fight Mode using WAF custom rules or Page Rules", because it runs outside the ruleset engine. For granular control with allowlists, Cloudflare points to Super Bot Fight Mode or Bot Management (Cloudflare, Bot Fight Mode).
What we recommend. On a site that wants to be cited by assistants, Bot Fight Mode on the Free plan is a binary switch: either it is off, or you accept that legitimate crawlers receive a challenge they cannot solve. We turn it off and handle real abuse with rate limiting rules, which can be targeted.
3. The "AI bots" setting
What the vendor documents. The legacy block-AI-bots setting blocks "verified bots classified as crawling for AI training purposes, plus unverified bots with similar behavior", and "excludes mixed-purpose bots that are used both for Training and for Search". Three options: block on all pages, block on pages with ads, or allow. Cloudflare announces its deprecation "on September 15, 2026" in favour of separate policies per classification: Search, Agent, Training (Cloudflare, block AI bots).
What we recommend. With the new per-classification policies, the configuration consistent with an open robots.txt is: Search and Agent allowed, Training according to your editorial choice, the same as in the file. A site that refuses GPTBot in robots.txt and allows it in Cloudflare, or the reverse, sends two contradictory messages; the most restrictive one wins, silently.
4. Verified bots and challenges
What the vendor documents. A verified bot is "a bot or agent that Cloudflare has confirmed is transparent about who it is and what it does", with two requirements: honest self-identification through cryptographic signatures or IP validation, and non-abusive behaviour respecting robots.txt. Behaviour classifications include Search, Agent, Training, Data Collection; verified bots "have been excluded in default bot configurations" and customers can now define their own policies (Cloudflare, verified bots).
What we recommend. Check in the verified bots directory that the crawlers you care about are listed, then write your rules on the category rather than on the user agent: the category rests on a verification, the user agent is copied in one line. A challenge, managed or interactive, exists "to verify whether a visitor is a real human": an assistant crawler will not solve it, however honest it is.
In our September 2026 study of 1,000 European SME sites, 91 of the 936 reached sites sat behind Cloudflare: 9% of them refused at least one AI crawler in robots.txt (against 3% of sites with no CDN detected) and 5 of the 91 served a challenge to at least one assistant crawler identity, while no site without a CDN served one. These figures describe existing zones; Cloudflare documents new defaults for new domains from 15 September 2026, Training and Agent bots blocked on pages that display ads, Search allowed (Cloudflare, block AI bots), which will move these proportions for zones created afterwards.
5. The WAF rule that lets them through
What the vendor documents. "Use the Skip action in a custom rule to skip one or more security features": rate limiting rules, managed rules and Super Bot Fight Mode rules can be skipped, but not Bot Fight Mode. The cf.verified_bot_category and http.user_agent fields are available in the expression (Cloudflare, Skip action).
What we recommend. One Skip rule, placed before the others, that exempts answer crawlers from challenges and managed rules, on the verified bot category when it exists, on the user agent as a fallback. It does not replace robots.txt, it accompanies it: the file says what the crawler may read, the rule lets it reach the file.
How to fix
# WAF custom rule, action: Skip (managed rules, Super Bot Fight Mode, rate limiting)
(cf.verified_bot_category in {"Search Engine Crawler" "AI Search" "AI Assistant"})
or (http.user_agent contains "OAI-SearchBot")
or (http.user_agent contains "ChatGPT-User")
or (http.user_agent contains "Claude-SearchBot")
or (http.user_agent contains "Claude-User")
or (http.user_agent contains "PerplexityBot")The category labels are read in the rule's selector as you write it; we do not guarantee their list, it changes with the per-classification policies.
6. Check whether Cloudflare blocks ChatGPT, from outside
What the vendor documents. "Challenges are security mechanisms used by Cloudflare to verify whether a visitor to your site is a real human and not a bot or automated script" (Cloudflare, challenges). The documentation describes the mechanism; it does not publish the list of status codes or headers an automated client receives, and we do not invent it.
What we recommend. Make the request yourself, with the crawler's user agent, and read three things: the status code, the body size, and whether that body is your page or a Cloudflare page. A 200 with your HTML is the only acceptable result. A 403 or 503 with a body that is not your page comes from the proxy, not from your server.
How to check
The GEO test announces itself with its own user agent and reports that our crawler was allowed on the home page when it could get in; but it reads robots.txt, not the proxy configuration, and its methodology says so: a block at CDN or firewall level is not visible in robots.txt. On the command line:
for ua in "OAI-SearchBot" "ChatGPT-User" "Claude-SearchBot" "PerplexityBot" "GPTBot"; do
printf "%-18s " "$ua"
curl -A "$ua" -s -o /tmp/body.html -w "%{http_code} %{size_download} bytes\n" https://your-site.com/
grep -qi "cloudflare" /tmp/body.html && echo " body: a Cloudflare page, not yours"
doneWhat we did on seoforge.fr
seoforge.fr uses Cloudflare only as its DNS host: the A and www records are set to DNS only, no proxy. AI crawlers therefore reach the Azure server directly, and no Cloudflare setting (Bot Fight Mode, AI-bot blocking, WAF rules) applies. What this page describes comes from our work for clients whose robots.txt was correct and whose proxy returned challenges. The GEO test states it among its published limits, and the GEO Audit includes the server log reading that alone settles it.
Going further
This page details the CDN paragraph of the first dimension of the AI search optimisation guide. If the proxy lets crawlers through and you are still absent from answers, the GEO Audit reads your logs and covers the six dimensions.
Frequently asked
Does Cloudflare block AI crawlers by default? The documentation we consulted describes an AI-bots blocking setting with three options, block everywhere, block on pages with ads, or allow, and announces its replacement by per-classification policies. It does not say which option applies to your zone: that is in your dashboard, and a request with the crawler's user agent confirms it from outside.