your site might be invisible to chatgpt, and cloudflare did it without asking you.
here's the short version: since July 1, 2025, every new Cloudflare domain has blocked GPTBot, ClaudeBot, and PerplexityBot by default. If your site sits behind Cloudflare, or on a platform that quietly runs Cloudflare underneath it, there's a real chance ChatGPT, Claude, and Perplexity have never actually read a single page of it. Not because your content is weak. Because a firewall said no before your content ever got a chance.
You can do everything right, schema, citations, answer-first pages, and still be invisible to every AI answer engine, because the block happens at the edge, before robots.txt is even read. This is the gap almost nobody checks, and it's the highest-leverage fifteen minutes you can spend this month.
AI crawler access is defined as whether the automated agents that AI systems use to read the web, such as GPTBot, ClaudeBot, and PerplexityBot, can successfully retrieve your pages. It is decided at two layers: robots.txt, which crawlers honor voluntarily, and your firewall, which blocks before robots.txt is ever read.
what actually happened
Cloudflare framed the change as protecting publishers from uncompensated AI training scraping, part of what it called Content Independence Day. Every zone created after July 1, 2025 got a default rule blocking known AI crawler user agents: GPTBot, ClaudeBot, PerplexityBot, and others. Webflow made the same move mandatory in April 2026 when it migrated every Webflow site onto Cloudflare's infrastructure, meaning even site owners who never touched a Cloudflare dashboard inherited the block.
The rule does not distinguish between a crawler that trains a model and a crawler that answers a live question about your brand. It blocks both. If OAI-SearchBot can't reach your pages, ChatGPT Search stops citing you. If PerplexityBot is blocked, Perplexity stops citing you.
It doesn't matter how good the page is if the model was never let in the door.
the part that should worry you
This isn't a hypothetical. In one documented case, a Brooklyn law firm ran an AI visibility check on July 3, 2026, and found every major crawler, GPTBot, PerplexityBot, ClaudeBot, and Common Crawl, coming back with a 403 error. The block had been running since the site moved to Cloudflare in November 2025. Seven months, zero notifications, zero visible traffic drop, because you can't see traffic that never arrives. The firm's own read on it: "the block looked like protection working."
That's the trap. Nothing breaks. Nothing errors out for a human visitor. Your Google rankings might even hold steady, since Googlebot isn't on the AI crawler block list. The only sign is an absence: your name never comes up when someone asks ChatGPT or Perplexity a question you should be the answer to.
why this compounds the problem for solo operators
Perplexity makes it worse in a different way. Cloudflare documented Perplexity running undeclared crawlers that rotate user agents, IPs, and ASNs specifically to get around no-crawl rules, which is part of why Cloudflare delisted Perplexity from its Verified Bots Program in August 2025. As of early 2026, Perplexity is being gradually rehabilitated, but some zones still inherit a network-level reputation block even after you've explicitly allowed PerplexityBot in your own rules. Checking your robots.txt is not enough. You have to check what's actually happening at the firewall.
And Anthropic's crawlers are mid-transition too. ClaudeBot is the training and indexing crawler, but Claude's live web search feature increasingly routes through separate user agents (Claude-User, Claude-SearchBot). Allowing ClaudeBot alone doesn't guarantee Claude can answer a live question about your site. You want all three allowed together.
| crawler | what it's for | what happens if blocked |
|---|---|---|
| GPTBot | training and general crawling | ChatGPT never learns your site exists |
| OAI-SearchBot | ChatGPT's live search citations | ChatGPT can't cite you in real-time answers |
| ClaudeBot | training and indexing | Claude has no baseline knowledge of your site |
| Claude-SearchBot / Claude-User | Claude's live search and browsing | Claude can't read your site when asked about it directly |
| PerplexityBot | Perplexity's search index | Perplexity never surfaces you as a source |
the fix, and what to watch for
If you're on Cloudflare directly: go into your WAF rules and check whether "Block AI Bots" or a Bot Fight Mode rule is active. Explicitly allow GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, and PerplexityBot. This is a rule change, not a full security rollback, you can keep blocking everything else that looks like a scraper.
If you're on Webflow, Netlify with a Cloudflare proxy, or any platform where Cloudflare sits underneath the dashboard you actually use: the setting may not be visible to you at all. That's the case for Webflow specifically, where Cloudflare is managed at the infrastructure level. Ask your host directly whether AI crawlers are allowed through, don't assume the dashboard you can see is the whole picture.
Either way, don't stop at robots.txt. A robots.txt disallow is a request a well-behaved crawler honors. A WAF block is a wall. Given what Cloudflare found about Perplexity, and OpenAI's own documentation warning that ChatGPT-User "may not apply" robots.txt on user-initiated fetches, the firewall layer is the one that actually decides whether you're visible, not the text file.
the fifteen-minute version
Read the next post in this series for the exact step-by-step audit, but the short version: visit your site's robots.txt directly and look for any AI bot paired with a Disallow. Then check your host's firewall or bot-management settings for a rule blocking AI crawlers by category, since that block happens whether or not robots.txt says anything at all. If you find either one, that's the reason you've been getting zero AI-referred traffic no matter how good your content is.
This is also exactly the kind of infrastructure question the how chatgpt decides what to recommend framework assumes is already solved. All six of those moves, entity clarity, answer-first pages, schema, third-party mentions, only work if the model can actually reach your site in the first place. This is the layer underneath that layer.
Related reading in this series: llms.txt won't save you covers the file everyone is selling as the fix for this problem, and what is llms.txt is the plain explainer of what that file actually is.
if you'd rather have someone check it for you
this is a fifteen-minute check when you know what to look for, and an invisible problem when you don't. the free ai visibility scan runs six real checks against your live site, including crawler access, in about sixty seconds. results on the page, no email needed.
run the free scanfrequently asked questions
How do I know if AI crawlers are blocked on my site?
Check two places, not one. Visit yourdomain.com/robots.txt and look for GPTBot, ClaudeBot, or PerplexityBot followed by Disallow. Then check your host's firewall or bot-management settings separately, because an edge-level block happens before robots.txt is read and a clean robots.txt proves nothing.
Does Cloudflare block AI crawlers by default?
Yes, on zones created after July 1 2025. Cloudflare introduced a default rule blocking known AI crawler user agents as part of its Content Independence Day policy. The rule does not distinguish between crawlers that train models and crawlers that answer live questions about your brand.
Will blocking AI crawlers hurt my Google rankings?
No. Googlebot is a separate crawler and is not on the AI-bot block list, so organic rankings can hold steady while your site is completely invisible to ChatGPT, Claude, and Perplexity. That is what makes the problem hard to notice.
Can I allow AI search crawlers but block training crawlers?
Partly. Anthropic documents separate agents, so you can disallow ClaudeBot for training while allowing Claude-SearchBot and Claude-User for live answers. OpenAI's documentation is looser, warning that robots.txt may not apply to user-initiated fetches, so treat that pairing as best effort.
How long does it take to appear in AI answers after unblocking?
Weeks, not days, and it depends on how often the crawlers revisit. Unblocking restores eligibility rather than producing citations directly. You still need pages that answer the questions being asked.
written by elisabeth hitz, certified in anthropic's ai fluency program (framework & foundations, and ai capabilities & limitations), plus claude 101 and claude cowork. sources: cloudflare, content independence day ai crawler policy · cloudflare's august 2025 report on perplexity's undeclared crawlers · documented 2026 case reports on cloudflare's default ai-bot block (the april 2026 webflow infrastructure migration, and a brooklyn law firm's seven-month blocked-crawler incident found july 3, 2026) · openai's public documentation on chatgpt-user and robots.txt.