How to Allow GPTBot and OAI-SearchBot in robots.txt
This check fails when your robots.txt stops OpenAI's crawlers, GPTBot and OAI-SearchBot, from fetching your site. The block can come from a named rule or from a catch-all `User-agent: *` group. By the end of this guide you will have found the rule causing the block, added explicit allow groups for both bots and confirmed that a robots.txt parser treats your homepage as crawlable.

Why it matters
OAI-SearchBot builds the index ChatGPT uses to surface and link sources in its search answers. If it is blocked, your pages are unlikely to be cited or linked there, even when they are the best answer. GPTBot collects content that can be used to train OpenAI's models, which affects how well those models know your brand, products and terminology.
Many sites block both by accident through a blanket Disallow: /, a plugin preset or a CDN "block AI bots" toggle. Those sites lose AI referral traffic they never chose to give up.
What you need
Step-by-step fix
Fetch your live robots.txt
Download the file exactly as crawlers see it, not as your CMS settings describe it. Plugins, CDNs and hosting layers can all rewrite it. Check the final URL after redirects: robots.txt must sit at the root of the exact host you want crawled, such as https://www.example.com/robots.txt.
curl -sL https://example.com/robots.txt curl -sIL https://example.com/robots.txt | grep -iE '^(HTTP|location)'
Find every rule that applies to OpenAI's bots
A crawler follows only the most specific group that matches its name. It ignores User-agent: * if a group naming it exists. Look for:
- A
User-agent: GPTBotorUser-agent: OAI-SearchBotline followed byDisallow: / - Grouped lines such as
User-agent: GPTBot+User-agent: CCBotsharing oneDisallow: / - A
User-agent: *group withDisallow: /and no specific OpenAI group - Blocks prepended by a CDN, often marked with a comment such as "BEGIN Cloudflare Managed content"
curl -sL https://example.com/robots.txt | grep -niE -A3 'user-agent: *(gptbot|oai-searchbot|\*)'
Decide your policy for each bot
OAI-SearchBot powers ChatGPT search results and citations. GPTBot gathers content that may be used to train models. A third agent, ChatGPT-User, makes fetches on behalf of a user's request.
This audit passes only when both GPTBot and OAI-SearchBot are allowed. If you deliberately opt out of training, you can still allow OAI-SearchBot and keep GPTBot blocked. Accept that this check will then stay flagged, and record the decision so nobody "fixes" it later.
Add explicit allow groups to robots.txt
Delete any Disallow: / lines aimed at GPTBot or OAI-SearchBot. Then add a dedicated group for each bot. Explicit groups protect you if someone later tightens the * group.
Keep private areas blocked by repeating those Disallow lines inside the OpenAI groups, because those bots do not inherit rules from *.
# Example Co robots.txt User-agent: GPTBot Allow: / Disallow: /account/ Disallow: /checkout/ User-agent: OAI-SearchBot Allow: / Disallow: /account/ Disallow: /checkout/ User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: / Disallow: /account/ Disallow: /checkout/ Sitemap: https://example.com/sitemap.xml
Turn off CDN or plugin AI-bot blocking
txt edits are undone if another layer rewrites the file or rejects the bots. In Cloudflare, open AI Crawl Control (or Security → Settings → Bot traffic on older dashboards). Disable Block AI bots, or set GPTBot and OAI-SearchBot to Allow.
txt) if it adds Disallow lines for them. Check security plugins and WAF custom rules for user-agent blocks too.
Deploy, purge cache and verify
txt. Then confirm with a standards-based parser that both bots may fetch your homepage and a key page. Both values must print True.
Finally, re-run the audit.
curl -sL https://example.com/robots.txt | head -40
python3 -c "import urllib.robotparser as r; p=r.RobotFileParser('https://example.com/robots.txt'); p.read(); [print(ua, p.can_fetch(ua,'https://example.com/')) for ua in ('GPTBot','OAI-SearchBot')]"Platform-specific notes
WordPress
WordPress serves a virtual robots.txt unless a physical file exists in the web root. A physical file always wins, so check for one over SFTP first. Edit the virtual file in Yoast SEO → Tools → File editor, or Rank Math → General Settings → Edit robots.txt. Some security and "AI blocker" plugins inject GPTBot disallows, so deactivate or reconfigure them.
Shopify
Go to Online Store → Themes → … → Edit code → Add a new template → robots. This creates templates/robots.txt.liquid. Keep the default `{% for group in robots.default_groups %}` loop, then append plain-text groups for GPTBot and OAI-SearchBot below it. Shopify's defaults do not block OpenAI bots, so a block usually comes from a custom template or an app.
Wix
Go to Marketing & SEO → SEO → Tools → Robots.txt Editor → View File. Remove any GPTBot or OAI-SearchBot disallows, add the allow groups and save. Wix does not let you upload a robots.txt file directly.
Next.js
With the App Router, return rules from app/robots.ts. For example: `rules: [{ userAgent: ['GPTBot','OAI-SearchBot'], allow: '/' }, { userAgent: '*', allow: '/', disallow: '/admin/' }], sitemap: 'https://example.com/sitemap.xml'`. Delete any static public/robots.txt so the two do not conflict, then redeploy.
Cloudflare
Cloudflare offers AI-crawler blocking, and it can be on by default for newly added domains. It can also prepend managed rules to your robots.txt. In AI Crawl Control, set GPTBot and OAI-SearchBot to Allow and disable the managed robots.txt option. Otherwise your origin file will be overridden at the edge.
Check that it worked
✅ How to verify
- Run `curl -sL https://example.com/robots.txt`. Confirm the GPTBot and OAI-SearchBot groups show `Allow: /` and no `Disallow: /`, and that no CDN-managed block sits above them.
- Run the Python urllib.robotparser one-liner and confirm it prints `True` for both GPTBot and OAI-SearchBot.
- Within a few days, check server or CDN logs for requests whose user agent contains `GPTBot` or `OAI-SearchBot` that return 200 rather than 403.
- Re-run the audit and confirm the "GPTBot / OAI-SearchBot access" check passes.
⚠️ Common mistakes
- Adding `Allow: /` under `User-agent: *` while an older `User-agent: GPTBot` / `Disallow: /` group still exists. The specific group wins.
- Fixing robots.txt at the origin while Cloudflare's managed robots.txt or Block AI bots setting still overrides it at the edge.
- Editing robots.txt on the wrong host (e.g. example.com instead of www.example.com) or leaving a stale cached copy on the CDN.
- Creating dedicated OpenAI groups without copying the private-path `Disallow` lines from `*`, which accidentally exposes /account/ or /checkout/.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot crawls pages so they can appear and be linked in ChatGPT search answers. You can allow or block each one independently in robots.txt.
Can I block AI training but still appear in ChatGPT search?
Yes. Disallow GPTBot and allow OAI-SearchBot. This audit checks both, though, so it will keep flagging GPTBot. Treat that as a documented, intentional choice.
Does allowing GPTBot affect my Google rankings?
No. Google's ranking crawlers are Googlebot and its variants, and they ignore GPTBot and OAI-SearchBot rules. Changing these groups affects only OpenAI's crawlers.
How long until ChatGPT starts citing my pages after unblocking?
OpenAI's crawlers re-fetch robots.txt periodically, typically within about a day. They then need to crawl your pages before those pages can appear in answers, which can take days to weeks. A sitemap linked from robots.txt helps them find your pages.
How to Allow ClaudeBot and Claude-SearchBot in robots.txt
How to Fix AI Overview Eligibility by Ranking in the Top 10
How to Allow PerplexityBot in robots.txt and Your Firewall
How to Fix Competitor Citation Sources in AI Answers
How to Set an Explicit Google-Extended Policy in robots.txt
How to Fix Brand Narrative Consistency for AI Search