the 5-minute audit: is gptbot, claudebot, or perplexitybot blocked on your site?
three checks, five minutes, no dev required. this is the exact sequence to find out whether ChatGPT, Claude, or Perplexity can actually reach your site, before you spend another hour writing content they'll never see.
check 1: your robots.txt (60 seconds)
Type your domain followed by /robots.txt directly into a browser (for example, yourdomain.com/robots.txt). Look for any of these user agents followed by Disallow: /
| user agent | belongs to |
|---|---|
| GPTBot | OpenAI / ChatGPT training |
| OAI-SearchBot | ChatGPT live search |
| ClaudeBot | Anthropic / Claude training |
| Claude-SearchBot | Claude live search |
| Claude-User | Claude browsing on a user's behalf |
| PerplexityBot | Perplexity's search index |
| Google-Extended | Gemini / Google AI features |
If you see Disallow: / next to any of these, that crawler has been told not to touch your site. If the file is empty, missing, or only lists generic rules with no AI bots named, robots.txt isn't your problem, move to check 2.
check 2: your firewall or bot-management settings (2 minutes)
This is the one almost nobody checks, and it's the one that actually matters most.
A robots.txt disallow is a polite request. A firewall block is a wall, and it can be active even when your robots.txt looks completely clean.
If you're on Cloudflare: log into your dashboard, go to Security, and look for a "Block AI Bots" rule or a Bot Fight Mode setting. Both can be on by default depending on when your zone was created.
If you're on Webflow: Cloudflare runs underneath your dashboard as of the April 2026 infrastructure migration, and you may not be able to see or change this setting directly. Contact Webflow support and ask specifically whether AI crawlers are allowed through.
If you're on Netlify or another host: ask whether a CDN or proxy layer (Cloudflare or otherwise) sits in front of your site, and whether it has any bot-blocking rules active. Don't assume the answer is no just because you never configured one, some of these ship on by default.
check 3: the direct test (2 minutes)
Search for a phrase that only appears on one of your own pages, something specific enough that a real citation would have to come from you. Then ask ChatGPT, Claude, and Perplexity a direct question your page answers. If none of them ever mention your site, even for a question your content clearly answers better than what they do cite, that's a strong signal something is blocking access rather than a signal your content is weak.
You can also directly ask an AI assistant to fetch a URL from your site, if it says it can't access the page or returns an error, note that. It's not proof of a permanent block on its own, but combined with check 1 and check 2 it tells you where to look.
what to do with what you find
- Robots.txt shows a Disallow on an AI crawler: remove it, or narrow it, if you want live-search visibility while still blocking training. You can allow Claude-SearchBot and Claude-User for live answers while disallowing ClaudeBot for training, if that's the line you want to draw.
- Firewall or bot-management rule is blocking AI crawlers: this is the more serious one, and the one most likely to be silently on. Explicitly allow GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, and PerplexityBot in the rule, don't disable your whole security setup to fix it, just carve out these specific user agents.
- Everything checks out clean and you're still invisible: at that point the issue is content, not access, and that's where the six-move framework in how chatgpt decides what to recommend actually applies.
One more thing worth knowing before you assume a fix has worked: Cloudflare has documented Perplexity running undeclared crawlers that rotate user agents and IPs specifically to route around no-crawl rules, and Perplexity was delisted from Cloudflare's Verified Bots Program over it in August 2025. Rehabilitation is underway in 2026, but don't take one clean check as permanent, run this audit again after any hosting or CDN change.
Related reading in this series: this audit is the access half of the wider ai visibility checklist, and llms.txt won't save you covers the file people reach for instead of running these three checks.
if you want it done for you
this audit takes five minutes once you know where to look, and most solo operators have never looked. the free ai visibility scan checks this properly, along with everything else that decides whether ai assistants can find and recommend you, in about sixty seconds. results on the page, no email needed.
run the free scanfrequently asked questions
How long does an AI crawler audit take?
About five minutes for the three core checks. Reading robots.txt takes a minute, checking your host's bot settings takes two, and testing a few real queries in ChatGPT and Perplexity takes the rest. No developer is required.
Which AI crawlers should I allow?
GPTBot and OAI-SearchBot for OpenAI, ClaudeBot, Claude-SearchBot and Claude-User for Anthropic, PerplexityBot for Perplexity, and Google-Extended for Google's AI features. Allow the search-oriented agents even if you disallow the training ones.
Is robots.txt enough to control AI crawler access?
No. Robots.txt is a request that well-behaved crawlers honor voluntarily. Firewall and WAF rules are enforced blocks that happen before robots.txt is read, so a permissive robots.txt can sit on a site that rejects every AI crawler.
What if my site checks out but AI still does not mention me?
Then the issue is content and authority rather than access. Access is a gate: passing it makes you eligible, not cited. At that point the work moves to answer-first structure, specific sourced claims, and mentions on sites you do not own.
How often should I re-run this check?
After any hosting, CDN, or platform change, and quarterly otherwise. Defaults get reset during migrations and nobody sends a notification.
written by elisabeth hitz, certified in anthropic's ai fluency program (framework & foundations, and ai capabilities & limitations), plus claude 101 and claude cowork. sources: cloudflare content independence day ai crawler policy and verified bots program documentation · cloudflare's august 2025 report on perplexity's undeclared crawlers · documented 2026 webflow/cloudflare infrastructure migration reporting.