How to Allow ClaudeBot and Claude-SearchBot in robots.txt

Your robots.txt is blocking Anthropic's crawlers, either by name or through a blanket rule that also applies to them. As a result, Claude can't read your pages or cite them in its answers. By the end of this guide, ClaudeBot and Claude-SearchBot will be explicitly allowed, your private paths will still be protected, and you'll have confirmed the live file serves the right rules.

By Updated ⏱ 15 minutes🎯 Beginner● High impactGEO / AI
Illustration: ClaudeBot / Claude-SearchBot access fix guide
GEO / AI · FIX GUIDE ClaudeBot / Claude-SearchBot acce… IMPACTEFFORT TIME15 minutes LEVELBeginner

Why it matters

Claude answers questions with web search, using pages that Claude-SearchBot has indexed and fetched. txt disallows Anthropic's crawlers, your pages are skipped. Claude may then cite a competitor or a third-party summary of your product instead of your own page.

This also matters for training-time knowledge: ClaudeBot collects public pages that shape what Claude knows about Example Co before any search happens. txt, silently removes you from every Claude answer.

✕BeforeUser-agent: ClaudeBot Disallow: / (orUser-agent: * / Disallow: / with noClaude-specific group)✓AfterUser-agent: ClaudeBot Allow: /User-agent: Claude-SearchBot Allow: /

What you need

Access to edit robots.txt (CMS SEO settings, theme files, or web server root)
A terminal with curl, or a browser to open https://example.com/robots.txt
CDN/firewall dashboard access if you use Cloudflare or a similar service
Your sitemap URL

Step-by-step fix

1

Fetch the live robots.txt and find Claude rules

Download the file that crawlers actually receive, not the one you think you deployed. Look for these:

  • any group naming ClaudeBot, Claude-SearchBot, Claude-User or the old anthropic-ai token
  • the User-agent: * group

A crawler obeys only the most specific group that matches its name. If no Claude group exists, the * group applies to Claude in full.

text
curl -s https://example.com/robots.txt
curl -s https://example.com/robots.txt | grep -i -n -A3 -E "claude|anthropic|user-agent: \*"
Pro tip: Also check www vs non-www and http vs https. Each host serves its own robots.txt, and crawlers read the one on the host they are crawling.
2

Remove Disallow rules that block Anthropic

Delete any Disallow: / line under a ClaudeBot, Claude-SearchBot or anthropic-ai group. Also remove broad patterns such as Disallow: /* under those groups. If your User-agent: * group contains Disallow: /, either remove that line or add the explicit Claude groups in the next step.

The explicit groups override * for Anthropic's bots.

Pro tip: The anthropic-ai token is no longer used by Anthropic's crawlers. Removing an old block on it is harmless, but it won't unblock anything on its own.
3

Add explicit allow groups for Claude's crawlers

Add a separate group for each Anthropic user agent so the intent is unambiguous. Include Claude-User too: it fetches pages when a person asks Claude to read a specific URL. Keep your sitemap line so crawlers can discover every page.

text
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Pro tip: Leave Allow lines for GPTBot, OAI-SearchBot and PerplexityBot in place while you're editing, so you don't fix one AI engine and block another.
4

Copy private-path rules into the Claude groups

A bot that matches a named group ignores the * group entirely. Any Disallow lines you rely on, such as /admin/, /cart/, /checkout/ or /account/, must be repeated inside each Claude group. Otherwise those paths become crawlable.

Disallow only what is genuinely private: a Disallow: / here would fail the check again.

text
User-agent: ClaudeBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/

User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Pro tip: Anthropic honors Crawl-delay. If server load is your concern, add `Crawl-delay: 5` to the Claude groups instead of blocking them.
5

Publish the file through your platform

Save the change where your platform actually generates robots.txt; see the platform notes for exact menu paths. On a static site or custom server, upload robots.txt to the web root so it is served at /robots.txt with status 200 and Content-Type: text/plain. Then purge any CDN cache for /robots.txt.

nginx
# Nginx: make sure robots.txt is served as a static file
location = /robots.txt {
    root /var/www/example.com/public;
    default_type text/plain;
    add_header Cache-Control "public, max-age=3600";
}
Pro tip: A robots.txt that returns 5xx can make crawlers treat the whole site as off-limits. Make sure it is never behind a login, a redirect loop or a maintenance page.
6

Turn off CDN or firewall AI-bot blocking

Some CDNs inject Disallow rules into robots.txt or return 403 to AI crawlers, overriding your file. In Cloudflare, check two places:

  • AI Crawl Control: make sure ClaudeBot and Claude-SearchBot are set to Allow
  • Security → Settings: confirm Block AI bots is off, or scoped so it doesn't cover Anthropic

Also check any WAF custom rules that match Claude in the user agent.

Pro tip: If Cloudflare's managed robots.txt is enabled, it can prepend its own Disallow lines for AI crawlers. Re-fetch the live file after changing any of these settings.
7

Verify the live result

Fetch robots.txt again and confirm two things:

  • the Claude groups are present
  • no Disallow line blocks / for ClaudeBot or Claude-SearchBot, by name or through *

Then request a normal page with a Claude user agent and check for status 200, not 403 or a challenge page. Re-run the audit to confirm the check passes.

text
curl -s https://example.com/robots.txt | grep -i -A4 claude
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Claude-SearchBot" https://example.com/
Pro tip: Check your server logs over the next few days for ClaudeBot and Claude-SearchBot hits returning 200. That confirms real crawling, not just a correct file.

Platform-specific notes

WordPress

WordPress serves a virtual robots.txt unless a physical file exists in the site root; a physical file always wins. In Yoast SEO, go to Yoast SEO → Tools → File editor. In Rank Math, go to Rank Math → General Settings → Edit robots.txt. Also check security plugins such as Wordfence and hosting-level bot blockers, which can return 403 to ClaudeBot even when robots.txt allows it.

Shopify

Go to Online Store → Themes → … → Edit code and add a template named robots.txt.liquid if one doesn't exist. Keep Shopify's default output by looping over robots.default_groups (printing each group.user_agent, group.rules and group.sitemap). Then append your ClaudeBot and Claude-SearchBot groups after the loop. Deleting the default loop removes Shopify's checkout and cart protections.

Next.js

With the App Router, create app/robots.ts that exports a function returning MetadataRoute.Robots. Example: rules: [{ userAgent: ['ClaudeBot', 'Claude-SearchBot', 'Claude-User'], allow: '/', disallow: ['/admin/'] }, { userAgent: '*', allow: '/' }], sitemap: 'https://example.com/sitemap.xml'. Delete any public/robots.txt so the two don't conflict.

Wix / Webflow

Wix: go to Dashboard → Marketing & SEO → SEO → Robots.txt Editor (under Tools and settings), add the Claude groups and save. Webflow: go to Site settings → SEO → Indexing → robots.txt, paste the rules, save, then republish the site, because changes only go live on publish.

Cloudflare

Cloudflare can block AI crawlers independently of your robots.txt. In AI Crawl Control, set ClaudeBot and Claude-SearchBot to Allow. Confirm Security → Settings → Block AI bots is off for your zone. Disable the managed robots.txt option if it adds Disallow lines for Anthropic. Finally, purge the cache for /robots.txt.

Check that it worked

✅ How to verify

  • Open https://example.com/robots.txt in a private browser window and confirm groups exist for ClaudeBot and Claude-SearchBot with Allow: / and no Disallow: /.
  • Run curl with a ClaudeBot or Claude-SearchBot user agent against your homepage and a key product or article page; both should return HTTP 200.
  • Search your server or CDN logs for 'ClaudeBot' and 'Claude-SearchBot' over the following days and confirm the requests return 200 status codes.
  • Re-run the audit and confirm the ClaudeBot / Claude-SearchBot access check passes.

⚠️ Common mistakes

  • Adding a ClaudeBot group but forgetting that it overrides User-agent: *, which leaves /admin/ or /checkout/ open to Claude, or blocks the whole site if Disallow: / was copied in by accident.
  • Editing a robots.txt file while the live site serves a different one, for example a WordPress virtual file, a Next.js app/robots.ts, or a Cloudflare-managed version.
  • Allowing the crawlers in robots.txt while a WAF, security plugin or 'Block AI bots' toggle still returns 403 to them.
  • Only unblocking the deprecated anthropic-ai token, which Anthropic's current crawlers don't use.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?

ClaudeBot collects public web content that may be used to train Anthropic's models. Claude-SearchBot crawls pages to improve the quality of Claude's search results. Claude-User fetches a page when a person asks Claude to look at a specific URL. Each obeys its own robots.txt group, so you can set rules for them separately.

Can I block training but still appear in Claude's answers?

Yes. Disallow ClaudeBot and allow Claude-SearchBot and Claude-User. Your pages can still be found and cited in Claude's search answers but are excluded from training collection. Be aware this audit check expects both ClaudeBot and Claude-SearchBot to be allowed, so a deliberate training opt-out will show as a partial fail you can accept.

How long until Claude picks up my robots.txt change?

Anthropic's crawlers re-read robots.txt periodically, typically within a day or so, though there is no guaranteed schedule. Purge your CDN cache so they get the new file immediately. Appearing in answers then depends on your pages being recrawled and judged relevant.

Will allowing ClaudeBot slow down my site?

For most sites, no. Anthropic says its crawlers respect Crawl-delay, so if you see heavy traffic, add a line such as Crawl-delay: 5 to the Claude groups rather than blocking them. You can also rate-limit by user agent at your CDN.