AI crawler access checker
Can ChatGPT, Claude, Perplexity and Gemini read your site? Free, no signup.
What this tool checks
Checks whether the bots behind ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence and other AI products may read your site — in robots.txt and at the firewall — and whether you publish an llms.txt.
How to use it
- Enter a domain or the full URL of a page.
- Click “Check now” and wait a few seconds.
- Review the results and fix anything marked fail or warn.
How AI crawlers access your website
Search crawlers vs. training crawlers
AI companies run different bots for different jobs. Search and answer crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to answer questions and cite sources. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect content for future models. Blocking the first group makes you invisible in AI answers; blocking the second is a policy choice.
Two places a bot can be blocked
A crawler can be blocked in robots.txt or by your firewall or CDN (for example a bot-fight setting that returns 403). This checker tests both: it reads your robots rules and requests your homepage with each crawler’s user agent.
AI crawler access checker: FAQ
Is the AI crawler access checker free?
Yes — free, with no signup. Fair-use limits apply per hour.
Why would AI crawlers be blocked?
Many sites block them by accident: a copied robots.txt, a security plugin or a CDN bot rule. Blocked crawlers mean AI assistants know less about you and cite you less.
Should I allow every AI bot?
That is a business decision. Search-style bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) help you appear in AI answers; training bots (GPTBot, Google-Extended, CCBot) feed future models. This tool shows the current state so you can decide.
How do I allow ChatGPT to crawl my website?
Make sure robots.txt does not disallow OAI-SearchBot (and GPTBot if you want to allow training), and that your firewall or CDN does not block those user agents.
How do I block AI bots from training on my content?
Add Disallow rules for training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot in robots.txt, while keeping search crawlers allowed if you still want to appear in AI answers.
Why does my site block ClaudeBot even though robots.txt allows it?
Usually a firewall, CDN or security plugin rule. This checker shows a 403 or no response in that case.
More free tools
llms.txt generator
Create an llms.txt file from your site in seconds.
llms.txt validator
Check that your /llms.txt exists and is well-formed.
robots.txt tester
See which bots may crawl a URL on your site.
Meta tags & Google snippet preview
Check titles, descriptions, Open Graph and preview your search result.
Schema markup checker
Find and check the structured data (JSON-LD) on a page.
Schema generator
Generate Organization, LocalBusiness or FAQ JSON-LD.
Security headers grade
Grade HTTPS and security headers (HSTS, CSP, X-Frame-Options…).
Redirect checker
Follow every redirect hop and its status code.
XML sitemap validator
Find your sitemap, count URLs and spot broken entries.
Broken link checker
Find broken links on any page.
Keyword density checker
See which words and phrases a page uses most.
See what’s holding your site back
Run a free audit in minutes: every SEO, AI-visibility, accessibility and security issue, ranked by impact, with the fix for your platform.