How to Allow GPTBot and OAI-SearchBot in robots.txt

This check fails when your robots.txt stops OpenAI's crawlers, GPTBot and OAI-SearchBot, from fetching your site. The block can come from a named rule or from a catch-all `User-agent: *` group. By the end of this guide you will have found the rule causing the block, added explicit allow groups for both bots and confirmed that a robots.txt parser treats your homepage as crawlable.

By Updated ⏱ 15 minutes🎯 Beginner● High impactGEO / AI
Illustration: GPTBot / OAI-SearchBot access fix guide
GEO / AI · FIX GUIDE GPTBot / OAI-SearchBot access IMPACTEFFORT TIME15 minutes LEVELBeginner

Why it matters

OAI-SearchBot builds the index ChatGPT uses to surface and link sources in its search answers. If it is blocked, your pages are unlikely to be cited or linked there, even when they are the best answer. GPTBot collects content that can be used to train OpenAI's models, which affects how well those models know your brand, products and terminology.

Many sites block both by accident through a blanket Disallow: /, a plugin preset or a CDN "block AI bots" toggle. Those sites lose AI referral traffic they never chose to give up.

✕Beforerobots.txt contains `User-agent:GPTBot` / `Disallow: /`, or only`User-agent: *` / `Disallow: /`, so✓Afterrobots.txt has `User-agent: GPTBot`and `User-agent: OAI-SearchBot` groupswith `Allow: /`, and both bots may

What you need

Access to edit robots.txt (CMS, theme files, framework code or web server)
Access to your CDN/WAF dashboard if you use one (e.g. Cloudflare)
A terminal with curl, or a browser
Optionally Python 3 for the verification one-liner

Step-by-step fix

1

Fetch your live robots.txt

Download the file exactly as crawlers see it, not as your CMS settings describe it. Plugins, CDNs and hosting layers can all rewrite it. Check the final URL after redirects: robots.txt must sit at the root of the exact host you want crawled, such as https://www.example.com/robots.txt.

text
curl -sL https://example.com/robots.txt
curl -sIL https://example.com/robots.txt | grep -iE '^(HTTP|location)'
Pro tip: If the file returns a 5xx error or times out, crawlers may treat the whole site as disallowed. Fix server errors before you edit rules.
2

Find every rule that applies to OpenAI's bots

A crawler follows only the most specific group that matches its name. It ignores User-agent: * if a group naming it exists. Look for:

  • A User-agent: GPTBot or User-agent: OAI-SearchBot line followed by Disallow: /
  • Grouped lines such as User-agent: GPTBot + User-agent: CCBot sharing one Disallow: /
  • A User-agent: * group with Disallow: / and no specific OpenAI group
  • Blocks prepended by a CDN, often marked with a comment such as "BEGIN Cloudflare Managed content"
text
curl -sL https://example.com/robots.txt | grep -niE -A3 'user-agent: *(gptbot|oai-searchbot|\*)'
Pro tip: Treat user-agent names as case-insensitive, but spell them exactly: `GPTBot` and `OAI-SearchBot`, not `GPT-Bot` or `OpenAI`.
3

Decide your policy for each bot

OAI-SearchBot powers ChatGPT search results and citations. GPTBot gathers content that may be used to train models. A third agent, ChatGPT-User, makes fetches on behalf of a user's request.

This audit passes only when both GPTBot and OAI-SearchBot are allowed. If you deliberately opt out of training, you can still allow OAI-SearchBot and keep GPTBot blocked. Accept that this check will then stay flagged, and record the decision so nobody "fixes" it later.

Pro tip: Sites that only want search visibility usually allow OAI-SearchBot at minimum, because it determines whether ChatGPT can link to them.
4

Add explicit allow groups to robots.txt

Delete any Disallow: / lines aimed at GPTBot or OAI-SearchBot. Then add a dedicated group for each bot. Explicit groups protect you if someone later tightens the * group.

Keep private areas blocked by repeating those Disallow lines inside the OpenAI groups, because those bots do not inherit rules from *.

text
# Example Co robots.txt
User-agent: GPTBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /
Disallow: /account/
Disallow: /checkout/

Sitemap: https://example.com/sitemap.xml
Pro tip: Leave a blank line between groups. A missing blank line can merge groups in some parsers and apply the wrong rules.
5

Turn off CDN or plugin AI-bot blocking

txt edits are undone if another layer rewrites the file or rejects the bots. In Cloudflare, open AI Crawl Control (or Security → Settings → Bot traffic on older dashboards). Disable Block AI bots, or set GPTBot and OAI-SearchBot to Allow.

txt) if it adds Disallow lines for them. Check security plugins and WAF custom rules for user-agent blocks too.

Pro tip: If you allowlist by IP, OpenAI publishes current ranges at https://openai.com/gptbot.json and https://openai.com/searchbot.json.
6

Deploy, purge cache and verify

txt. Then confirm with a standards-based parser that both bots may fetch your homepage and a key page. Both values must print True.

Finally, re-run the audit.

text
curl -sL https://example.com/robots.txt | head -40
python3 -c "import urllib.robotparser as r; p=r.RobotFileParser('https://example.com/robots.txt'); p.read(); [print(ua, p.can_fetch(ua,'https://example.com/')) for ua in ('GPTBot','OAI-SearchBot')]"
Pro tip: OpenAI's crawlers re-read robots.txt periodically. Expect changes to take effect within about a day.

Platform-specific notes

WordPress

WordPress serves a virtual robots.txt unless a physical file exists in the web root. A physical file always wins, so check for one over SFTP first. Edit the virtual file in Yoast SEO → Tools → File editor, or Rank Math → General Settings → Edit robots.txt. Some security and "AI blocker" plugins inject GPTBot disallows, so deactivate or reconfigure them.

Shopify

Go to Online Store → Themes → … → Edit code → Add a new template → robots. This creates templates/robots.txt.liquid. Keep the default `{% for group in robots.default_groups %}` loop, then append plain-text groups for GPTBot and OAI-SearchBot below it. Shopify's defaults do not block OpenAI bots, so a block usually comes from a custom template or an app.

Wix

Go to Marketing & SEO → SEO → Tools → Robots.txt Editor → View File. Remove any GPTBot or OAI-SearchBot disallows, add the allow groups and save. Wix does not let you upload a robots.txt file directly.

Next.js

With the App Router, return rules from app/robots.ts. For example: `rules: [{ userAgent: ['GPTBot','OAI-SearchBot'], allow: '/' }, { userAgent: '*', allow: '/', disallow: '/admin/' }], sitemap: 'https://example.com/sitemap.xml'`. Delete any static public/robots.txt so the two do not conflict, then redeploy.

Cloudflare

Cloudflare offers AI-crawler blocking, and it can be on by default for newly added domains. It can also prepend managed rules to your robots.txt. In AI Crawl Control, set GPTBot and OAI-SearchBot to Allow and disable the managed robots.txt option. Otherwise your origin file will be overridden at the edge.

Check that it worked

✅ How to verify

  • Run `curl -sL https://example.com/robots.txt`. Confirm the GPTBot and OAI-SearchBot groups show `Allow: /` and no `Disallow: /`, and that no CDN-managed block sits above them.
  • Run the Python urllib.robotparser one-liner and confirm it prints `True` for both GPTBot and OAI-SearchBot.
  • Within a few days, check server or CDN logs for requests whose user agent contains `GPTBot` or `OAI-SearchBot` that return 200 rather than 403.
  • Re-run the audit and confirm the "GPTBot / OAI-SearchBot access" check passes.

⚠️ Common mistakes

  • Adding `Allow: /` under `User-agent: *` while an older `User-agent: GPTBot` / `Disallow: /` group still exists. The specific group wins.
  • Fixing robots.txt at the origin while Cloudflare's managed robots.txt or Block AI bots setting still overrides it at the edge.
  • Editing robots.txt on the wrong host (e.g. example.com instead of www.example.com) or leaving a stale cached copy on the CDN.
  • Creating dedicated OpenAI groups without copying the private-path `Disallow` lines from `*`, which accidentally exposes /account/ or /checkout/.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

What is the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot crawls pages so they can appear and be linked in ChatGPT search answers. You can allow or block each one independently in robots.txt.

Can I block AI training but still appear in ChatGPT search?

Yes. Disallow GPTBot and allow OAI-SearchBot. This audit checks both, though, so it will keep flagging GPTBot. Treat that as a documented, intentional choice.

Does allowing GPTBot affect my Google rankings?

No. Google's ranking crawlers are Googlebot and its variants, and they ignore GPTBot and OAI-SearchBot rules. Changing these groups affects only OpenAI's crawlers.

How long until ChatGPT starts citing my pages after unblocking?

OpenAI's crawlers re-fetch robots.txt periodically, typically within about a day. They then need to crawl your pages before those pages can appear in answers, which can take days to weeks. A sitemap linked from robots.txt helps them find your pages.