How to Allow ClaudeBot and Claude-SearchBot in robots.txt
Your robots.txt is blocking Anthropic's crawlers, either by name or through a blanket rule that also applies to them. As a result, Claude can't read your pages or cite them in its answers. By the end of this guide, ClaudeBot and Claude-SearchBot will be explicitly allowed, your private paths will still be protected, and you'll have confirmed the live file serves the right rules.

Why it matters
Claude answers questions with web search, using pages that Claude-SearchBot has indexed and fetched. txt disallows Anthropic's crawlers, your pages are skipped. Claude may then cite a competitor or a third-party summary of your product instead of your own page.
This also matters for training-time knowledge: ClaudeBot collects public pages that shape what Claude knows about Example Co before any search happens. txt, silently removes you from every Claude answer.
What you need
Step-by-step fix
Fetch the live robots.txt and find Claude rules
Download the file that crawlers actually receive, not the one you think you deployed. Look for these:
- any group naming ClaudeBot, Claude-SearchBot, Claude-User or the old anthropic-ai token
- the User-agent: * group
A crawler obeys only the most specific group that matches its name. If no Claude group exists, the * group applies to Claude in full.
curl -s https://example.com/robots.txt curl -s https://example.com/robots.txt | grep -i -n -A3 -E "claude|anthropic|user-agent: \*"
Remove Disallow rules that block Anthropic
Delete any Disallow: / line under a ClaudeBot, Claude-SearchBot or anthropic-ai group. Also remove broad patterns such as Disallow: /* under those groups. If your User-agent: * group contains Disallow: /, either remove that line or add the explicit Claude groups in the next step.
The explicit groups override * for Anthropic's bots.
Add explicit allow groups for Claude's crawlers
Add a separate group for each Anthropic user agent so the intent is unambiguous. Include Claude-User too: it fetches pages when a person asks Claude to read a specific URL. Keep your sitemap line so crawlers can discover every page.
User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Copy private-path rules into the Claude groups
A bot that matches a named group ignores the * group entirely. Any Disallow lines you rely on, such as /admin/, /cart/, /checkout/ or /account/, must be repeated inside each Claude group. Otherwise those paths become crawlable.
Disallow only what is genuinely private: a Disallow: / here would fail the check again.
User-agent: ClaudeBot Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ User-agent: Claude-SearchBot Allow: / Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/
Publish the file through your platform
Save the change where your platform actually generates robots.txt; see the platform notes for exact menu paths. On a static site or custom server, upload robots.txt to the web root so it is served at /robots.txt with status 200 and Content-Type: text/plain. Then purge any CDN cache for /robots.txt.
# Nginx: make sure robots.txt is served as a static file
location = /robots.txt {
root /var/www/example.com/public;
default_type text/plain;
add_header Cache-Control "public, max-age=3600";
}Turn off CDN or firewall AI-bot blocking
Some CDNs inject Disallow rules into robots.txt or return 403 to AI crawlers, overriding your file. In Cloudflare, check two places:
- AI Crawl Control: make sure ClaudeBot and Claude-SearchBot are set to Allow
- Security → Settings: confirm Block AI bots is off, or scoped so it doesn't cover Anthropic
Also check any WAF custom rules that match Claude in the user agent.
Verify the live result
Fetch robots.txt again and confirm two things:
- the Claude groups are present
- no Disallow line blocks
/for ClaudeBot or Claude-SearchBot, by name or through*
Then request a normal page with a Claude user agent and check for status 200, not 403 or a challenge page. Re-run the audit to confirm the check passes.
curl -s https://example.com/robots.txt | grep -i -A4 claude
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude-SearchBot" https://example.com/Platform-specific notes
WordPress
WordPress serves a virtual robots.txt unless a physical file exists in the site root; a physical file always wins. In Yoast SEO, go to Yoast SEO → Tools → File editor. In Rank Math, go to Rank Math → General Settings → Edit robots.txt. Also check security plugins such as Wordfence and hosting-level bot blockers, which can return 403 to ClaudeBot even when robots.txt allows it.
Shopify
Go to Online Store → Themes → … → Edit code and add a template named robots.txt.liquid if one doesn't exist. Keep Shopify's default output by looping over robots.default_groups (printing each group.user_agent, group.rules and group.sitemap). Then append your ClaudeBot and Claude-SearchBot groups after the loop. Deleting the default loop removes Shopify's checkout and cart protections.
Next.js
With the App Router, create app/robots.ts that exports a function returning MetadataRoute.Robots. Example: rules: [{ userAgent: ['ClaudeBot', 'Claude-SearchBot', 'Claude-User'], allow: '/', disallow: ['/admin/'] }, { userAgent: '*', allow: '/' }], sitemap: 'https://example.com/sitemap.xml'. Delete any public/robots.txt so the two don't conflict.
Wix / Webflow
Wix: go to Dashboard → Marketing & SEO → SEO → Robots.txt Editor (under Tools and settings), add the Claude groups and save. Webflow: go to Site settings → SEO → Indexing → robots.txt, paste the rules, save, then republish the site, because changes only go live on publish.
Cloudflare
Cloudflare can block AI crawlers independently of your robots.txt. In AI Crawl Control, set ClaudeBot and Claude-SearchBot to Allow. Confirm Security → Settings → Block AI bots is off for your zone. Disable the managed robots.txt option if it adds Disallow lines for Anthropic. Finally, purge the cache for /robots.txt.
Check that it worked
✅ How to verify
- Open https://example.com/robots.txt in a private browser window and confirm groups exist for ClaudeBot and Claude-SearchBot with Allow: / and no Disallow: /.
- Run curl with a ClaudeBot or Claude-SearchBot user agent against your homepage and a key product or article page; both should return HTTP 200.
- Search your server or CDN logs for 'ClaudeBot' and 'Claude-SearchBot' over the following days and confirm the requests return 200 status codes.
- Re-run the audit and confirm the ClaudeBot / Claude-SearchBot access check passes.
⚠️ Common mistakes
- Adding a ClaudeBot group but forgetting that it overrides User-agent: *, which leaves /admin/ or /checkout/ open to Claude, or blocks the whole site if Disallow: / was copied in by accident.
- Editing a robots.txt file while the live site serves a different one, for example a WordPress virtual file, a Next.js app/robots.ts, or a Cloudflare-managed version.
- Allowing the crawlers in robots.txt while a WAF, security plugin or 'Block AI bots' toggle still returns 403 to them.
- Only unblocking the deprecated anthropic-ai token, which Anthropic's current crawlers don't use.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?
ClaudeBot collects public web content that may be used to train Anthropic's models. Claude-SearchBot crawls pages to improve the quality of Claude's search results. Claude-User fetches a page when a person asks Claude to look at a specific URL. Each obeys its own robots.txt group, so you can set rules for them separately.
Can I block training but still appear in Claude's answers?
Yes. Disallow ClaudeBot and allow Claude-SearchBot and Claude-User. Your pages can still be found and cited in Claude's search answers but are excluded from training collection. Be aware this audit check expects both ClaudeBot and Claude-SearchBot to be allowed, so a deliberate training opt-out will show as a partial fail you can accept.
How long until Claude picks up my robots.txt change?
Anthropic's crawlers re-read robots.txt periodically, typically within a day or so, though there is no guaranteed schedule. Purge your CDN cache so they get the new file immediately. Appearing in answers then depends on your pages being recrawled and judged relevant.
Will allowing ClaudeBot slow down my site?
For most sites, no. Anthropic says its crawlers respect Crawl-delay, so if you see heavy traffic, add a line such as Crawl-delay: 5 to the Claude groups rather than blocking them. You can also rate-limit by user agent at your CDN.
How to Allow PerplexityBot in robots.txt and Your Firewall
How to Allow GPTBot and OAI-SearchBot in robots.txt
How to Set an Explicit Google-Extended Policy in robots.txt
How to Fix AI Overview Eligibility by Ranking in the Top 10
How to Fix a CDN or WAF Blocking AI Crawlers
How to Fix Competitor Citation Sources in AI Answers