How to Declare an AI Crawler Policy in robots.txt and llms.txt

Your site has no stated position on AI crawlers. Neither robots.txt nor an llms.txt file says whether bots like GPTBot, ClaudeBot, PerplexityBot or Google-Extended may use your content. By the end of this guide you will have chosen a policy, published it as named robots.txt rules and an llms.txt file, and confirmed both files are live at your domain root.

By Updated ⏱ 15 minutes🎯 Beginner● Low impactGEO / AI
Illustration: AI usage policy declared fix guide
GEO / AI · FIX GUIDE AI usage policy declared IMPACTEFFORT TIME15 minutes LEVELBeginner

Why it matters

txt never names an AI user agent, you have no stated policy. AI bots fall back to your generic User-agent: * rules, and a site-wide block written for a different reason can quietly shut you out of ChatGPT search, Claude or Perplexity answers. Naming each bot does two things.

It lets you allow AI search and citation bots, which bring referral traffic and brand mentions, while still opting out of model training if you want to. txt file also gives AI systems a short, curated map of your most important pages and your usage terms. The direct ranking effect is small, but it removes doubt for both AI crawlers and anyone reviewing your content licensing.

✕Beforehttps://example.com/robots.txtcontains only "User-agent: * /Disallow: /admin/" and✓Afterrobots.txt has named groups forGPTBot, OAI-SearchBot, ClaudeBot,Claude-SearchBot, PerplexityBot and

What you need

Access to edit robots.txt (CMS, SEO plugin, theme code or web server)
Ability to upload a plain-text file to your site root, or a plugin that generates llms.txt
A decision from the business on whether AI training and AI search use are allowed
The URL of your XML sitemap

Step-by-step fix

1

Decide your AI usage policy

Choose one of three positions before you touch any files:

  • Allow all: AI search and model training may both use your content.
  • Search yes, training no: you get cited in AI answers but opt out of training.
  • Block all: no AI use at all.

The main AI vendors run separate bots for each job:

  • Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.
  • Search and citation: OAI-SearchBot, Claude-SearchBot, PerplexityBot.

Agree on the position with whoever owns legal or content licensing.

Pro tip: Most publishers who want AI visibility pick "search yes, training no". Blocking search bots removes you from AI answers, and you lose the citations and links that come with them.
2

Check your current robots.txt

txt in a browser and note what is already there. Look for a blanket Disallow: / under User-agent: *. Also look for rules a CDN or plugin may have injected.

Keep any existing rules you still need, such as blocks on /cart/ or /admin/. You will add the AI rules above them.

text
curl -s https://example.com/robots.txt
Pro tip: A bot follows only the most specific User-agent group that matches it. If you add a GPTBot group, GPTBot will ignore your * rules entirely, so repeat any paths you want it to stay out of.
3

Add named AI crawler rules to robots.txt

Add one group per AI user agent, matching the policy you chose. The example below uses "search yes, training no". For allow all, change every Disallow: / to Allow: /.

For block all, use Disallow: / in every AI group. Keep the Sitemap line at the end.

text
# AI usage policy for example.com
# AI search/citation crawlers: allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Model training crawlers: not allowed
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
Pro tip: Google-Extended does not affect Google Search rankings or AI Overviews eligibility. It only controls whether your content is used for Gemini training and grounding.
4

Publish an llms.txt file at the site root

txt. Start with an H1 site name and a one-line summary in a blockquote. Then add short sections linking your key pages.

Include a Usage policy section that states your AI terms in plain language. txt.

text
# Example Co

> Example Co sells refurbished office furniture and publishes buying guides for small businesses.

## Key pages
- [Product catalog](https://example.com/shop): current stock and prices
- [Buying guides](https://example.com/guides): how to choose desks and chairs
- [About](https://example.com/about): company background and contact

## Usage policy
- AI search and answer engines may quote and summarize our pages with a link to the source.
- Our content may not be used to train AI models without written permission.
- Full terms: https://example.com/terms#ai
Pro tip: Serve the file with Content-Type text/plain or text/markdown and status 200. Do not redirect it to an HTML page.
5

Mirror the policy in your terms of use

Add a short "AI and automated access" clause to your Terms of Use page and give it an anchor such as #ai. txt links to that anchor. The clause should say what is permitted, what is not, and who to contact for licensing.

txt is a voluntary standard, so the written terms are your enforceable record.

html
<h2 id="ai">AI and automated access</h2>
<p>Example Co permits AI search services to index, quote and link to our public pages. Use of our content to train machine-learning models is not permitted without a written licence. Contact licensing@example.com.</p>
6

Verify both files are live and correct

Fetch both URLs with curl and confirm each returns HTTP 200:

  • robots.txt should list each AI user agent by name.
  • llms.txt should return plain text, not your 404 page or an HTML template.

Then open the robots.txt report in Google Search Console (Settings → robots.txt) to confirm Google has fetched the new version.

text
curl -sI https://example.com/llms.txt | head -n 5
curl -s https://example.com/robots.txt | grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot|Google-Extended"
Pro tip: Re-run the audit after any CDN cache purge. A stale cached robots.txt is the most common reason the check still fails.

Platform-specific notes

WordPress

WordPress serves a virtual robots.txt that you can edit through your SEO plugin: - **Yoast SEO**: Yoast SEO → Tools → File editor. - **Rank Math**: Rank Math → General Settings → Edit robots.txt. If a physical robots.txt file exists in the web root, it overrides both plugins, so edit that file instead. Recent Yoast and Rank Math versions can generate llms.txt from their settings. Otherwise, upload llms.txt to the same folder as wp-config.php via SFTP or your host's file manager.

Shopify

In the admin, go to Online Store → Themes → … → Edit code → Templates → Add a new template → robots. This creates robots.txt.liquid. Keep the default Liquid loop that outputs Shopify's rules, and add your AI user-agent groups as plain text above or below it. Shopify does not let you upload files to the domain root. To publish llms.txt, upload it under Content → Files, then add a URL redirect from /llms.txt to that file in Online Store → Navigation → URL Redirects, or use an app that serves it.

Next.js

With the App Router, either create app/robots.ts or place a static public/robots.txt (do not use both): - In app/robots.ts, export a function that returns MetadataRoute.Robots, with a rules array holding one object per AI user agent and its allow or disallow value. - Put llms.txt in the public/ folder and it will be served at /llms.txt. Check that middleware or i18n rewrites do not redirect either file to a localized path.

Wix

Go to Dashboard → Marketing & SEO → SEO → Robots.txt Editor, then add the AI user-agent groups and save. Wix does not let you place arbitrary files at the root. The robots.txt rules alone are enough to pass this check, so treat llms.txt as optional on Wix.

Cloudflare

Cloudflare's AI Crawl Control and managed robots.txt features can add Disallow rules and Content-Signal lines for AI bots. They can also block AI crawlers at the edge regardless of what your robots.txt says. Review Security → Bots and the AI Crawl Control settings so they match your written policy. Purge the cache for /robots.txt and /llms.txt after every change.

Check that it worked

✅ How to verify

  • Open https://example.com/robots.txt and confirm each AI user agent you chose (e.g. GPTBot, ClaudeBot, PerplexityBot, Google-Extended) appears in its own User-agent group.
  • Run curl -sI https://example.com/llms.txt and confirm HTTP 200 with a text/plain or text/markdown content type.
  • In Google Search Console, open Settings → robots.txt and confirm the latest fetch shows your new rules with no errors.
  • Re-run the site audit and confirm the AI usage policy check now reports that a policy is declared.

⚠️ Common mistakes

  • Relying only on User-agent: * and assuming AI bots will "get it". Without a named group there is no explicit AI policy, and a blanket Disallow blocks AI search citations as well.
  • Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot while only meaning to opt out of training, which removes the site from AI answers and the links they include.
  • Adding an AI group with Allow: / and forgetting that the bot now ignores the * group, which exposes paths like /cart/ or /admin/ you previously disallowed.
  • Publishing llms.txt as an HTML page, behind a redirect to the homepage, or with a 404 status that only looks like content in the browser.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

Does blocking GPTBot remove my site from ChatGPT search?

No. GPTBot collects data for model training. ChatGPT search results use OAI-SearchBot, and links that users open from inside ChatGPT are fetched by ChatGPT-User. You can disallow GPTBot and still allow OAI-SearchBot so your pages can be cited.

Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended is a control token, not a separate crawler. It only governs use of your content for Gemini model training and grounding. Googlebot crawling, Search rankings and AI Overviews eligibility are unaffected.

Is llms.txt an official standard?

No. It is a community proposal that AI tools and some SEO plugins have adopted, not an IETF or W3C standard. It costs little to publish and documents your policy in plain language, but robots.txt rules remain the main signal crawlers act on.

Do AI crawlers actually obey robots.txt?

The major vendors, including OpenAI, Anthropic, Perplexity, Google and Apple, publish their user agents and say they honor robots.txt. Compliance is still voluntary. To enforce blocks, pair robots.txt with firewall or CDN bot rules and clear terms of use.