Free tool

AI crawler access checker

Can ChatGPT, Claude, Perplexity and Gemini read your site? Free, no signup.

What this tool checks

Checks whether the bots behind ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence and other AI products may read your site — in robots.txt and at the firewall — and whether you publish an llms.txt.

How to use it

  1. Enter a domain or the full URL of a page.
  2. Click “Check now” and wait a few seconds.
  3. Review the results and fix anything marked fail or warn.
In depth

How AI crawlers access your website

Search crawlers vs. training crawlers

AI companies run different bots for different jobs. Search and answer crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to answer questions and cite sources. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect content for future models. Blocking the first group makes you invisible in AI answers; blocking the second is a policy choice.

Two places a bot can be blocked

A crawler can be blocked in robots.txt or by your firewall or CDN (for example a bot-fight setting that returns 403). This checker tests both: it reads your robots rules and requests your homepage with each crawler’s user agent.

FAQ

AI crawler access checker: FAQ

Is the AI crawler access checker free?

Yes — free, with no signup. Fair-use limits apply per hour.

Why would AI crawlers be blocked?

Many sites block them by accident: a copied robots.txt, a security plugin or a CDN bot rule. Blocked crawlers mean AI assistants know less about you and cite you less.

Should I allow every AI bot?

That is a business decision. Search-style bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) help you appear in AI answers; training bots (GPTBot, Google-Extended, CCBot) feed future models. This tool shows the current state so you can decide.

How do I allow ChatGPT to crawl my website?

Make sure robots.txt does not disallow OAI-SearchBot (and GPTBot if you want to allow training), and that your firewall or CDN does not block those user agents.

How do I block AI bots from training on my content?

Add Disallow rules for training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot in robots.txt, while keeping search crawlers allowed if you still want to appear in AI answers.

Why does my site block ClaudeBot even though robots.txt allows it?

Usually a firewall, CDN or security plugin rule. This checker shows a 403 or no response in that case.

Free 14-day trial · no card

See what’s holding your site back

Run a free audit in minutes: every SEO, AI-visibility, accessibility and security issue, ranked by impact, with the fix for your platform.