How to Declare an AI Crawler Policy in robots.txt and llms.txt
Your site has no stated position on AI crawlers. Neither robots.txt nor an llms.txt file says whether bots like GPTBot, ClaudeBot, PerplexityBot or Google-Extended may use your content. By the end of this guide you will have chosen a policy, published it as named robots.txt rules and an llms.txt file, and confirmed both files are live at your domain root.

Why it matters
txt never names an AI user agent, you have no stated policy. AI bots fall back to your generic User-agent: * rules, and a site-wide block written for a different reason can quietly shut you out of ChatGPT search, Claude or Perplexity answers. Naming each bot does two things.
It lets you allow AI search and citation bots, which bring referral traffic and brand mentions, while still opting out of model training if you want to. txt file also gives AI systems a short, curated map of your most important pages and your usage terms. The direct ranking effect is small, but it removes doubt for both AI crawlers and anyone reviewing your content licensing.
What you need
Step-by-step fix
Decide your AI usage policy
Choose one of three positions before you touch any files:
- Allow all: AI search and model training may both use your content.
- Search yes, training no: you get cited in AI answers but opt out of training.
- Block all: no AI use at all.
The main AI vendors run separate bots for each job:
- Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.
- Search and citation: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
Agree on the position with whoever owns legal or content licensing.
Check your current robots.txt
txt in a browser and note what is already there. Look for a blanket Disallow: / under User-agent: *. Also look for rules a CDN or plugin may have injected.
Keep any existing rules you still need, such as blocks on /cart/ or /admin/. You will add the AI rules above them.
curl -s https://example.com/robots.txt
Add named AI crawler rules to robots.txt
Add one group per AI user agent, matching the policy you chose. The example below uses "search yes, training no". For allow all, change every Disallow: / to Allow: /.
For block all, use Disallow: / in every AI group. Keep the Sitemap line at the end.
# AI usage policy for example.com # AI search/citation crawlers: allowed User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # Model training crawlers: not allowed User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Publish an llms.txt file at the site root
txt. Start with an H1 site name and a one-line summary in a blockquote. Then add short sections linking your key pages.
Include a Usage policy section that states your AI terms in plain language. txt.
# Example Co > Example Co sells refurbished office furniture and publishes buying guides for small businesses. ## Key pages - [Product catalog](https://example.com/shop): current stock and prices - [Buying guides](https://example.com/guides): how to choose desks and chairs - [About](https://example.com/about): company background and contact ## Usage policy - AI search and answer engines may quote and summarize our pages with a link to the source. - Our content may not be used to train AI models without written permission. - Full terms: https://example.com/terms#ai
Mirror the policy in your terms of use
Add a short "AI and automated access" clause to your Terms of Use page and give it an anchor such as #ai. txt links to that anchor. The clause should say what is permitted, what is not, and who to contact for licensing.
txt is a voluntary standard, so the written terms are your enforceable record.
<h2 id="ai">AI and automated access</h2> <p>Example Co permits AI search services to index, quote and link to our public pages. Use of our content to train machine-learning models is not permitted without a written licence. Contact licensing@example.com.</p>
Verify both files are live and correct
Fetch both URLs with curl and confirm each returns HTTP 200:
- robots.txt should list each AI user agent by name.
- llms.txt should return plain text, not your 404 page or an HTML template.
Then open the robots.txt report in Google Search Console (Settings → robots.txt) to confirm Google has fetched the new version.
curl -sI https://example.com/llms.txt | head -n 5 curl -s https://example.com/robots.txt | grep -iE "GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|PerplexityBot|Google-Extended"
Platform-specific notes
WordPress
WordPress serves a virtual robots.txt that you can edit through your SEO plugin: - **Yoast SEO**: Yoast SEO → Tools → File editor. - **Rank Math**: Rank Math → General Settings → Edit robots.txt. If a physical robots.txt file exists in the web root, it overrides both plugins, so edit that file instead. Recent Yoast and Rank Math versions can generate llms.txt from their settings. Otherwise, upload llms.txt to the same folder as wp-config.php via SFTP or your host's file manager.
Shopify
In the admin, go to Online Store → Themes → … → Edit code → Templates → Add a new template → robots. This creates robots.txt.liquid. Keep the default Liquid loop that outputs Shopify's rules, and add your AI user-agent groups as plain text above or below it. Shopify does not let you upload files to the domain root. To publish llms.txt, upload it under Content → Files, then add a URL redirect from /llms.txt to that file in Online Store → Navigation → URL Redirects, or use an app that serves it.
Next.js
With the App Router, either create app/robots.ts or place a static public/robots.txt (do not use both): - In app/robots.ts, export a function that returns MetadataRoute.Robots, with a rules array holding one object per AI user agent and its allow or disallow value. - Put llms.txt in the public/ folder and it will be served at /llms.txt. Check that middleware or i18n rewrites do not redirect either file to a localized path.
Wix
Go to Dashboard → Marketing & SEO → SEO → Robots.txt Editor, then add the AI user-agent groups and save. Wix does not let you place arbitrary files at the root. The robots.txt rules alone are enough to pass this check, so treat llms.txt as optional on Wix.
Cloudflare
Cloudflare's AI Crawl Control and managed robots.txt features can add Disallow rules and Content-Signal lines for AI bots. They can also block AI crawlers at the edge regardless of what your robots.txt says. Review Security → Bots and the AI Crawl Control settings so they match your written policy. Purge the cache for /robots.txt and /llms.txt after every change.
Check that it worked
✅ How to verify
- Open https://example.com/robots.txt and confirm each AI user agent you chose (e.g. GPTBot, ClaudeBot, PerplexityBot, Google-Extended) appears in its own User-agent group.
- Run curl -sI https://example.com/llms.txt and confirm HTTP 200 with a text/plain or text/markdown content type.
- In Google Search Console, open Settings → robots.txt and confirm the latest fetch shows your new rules with no errors.
- Re-run the site audit and confirm the AI usage policy check now reports that a policy is declared.
⚠️ Common mistakes
- Relying only on User-agent: * and assuming AI bots will "get it". Without a named group there is no explicit AI policy, and a blanket Disallow blocks AI search citations as well.
- Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot while only meaning to opt out of training, which removes the site from AI answers and the links they include.
- Adding an AI group with Allow: / and forgetting that the bot now ignores the * group, which exposes paths like /cart/ or /admin/ you previously disallowed.
- Publishing llms.txt as an HTML page, behind a redirect to the homepage, or with a 404 status that only looks like content in the browser.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
Does blocking GPTBot remove my site from ChatGPT search?
No. GPTBot collects data for model training. ChatGPT search results use OAI-SearchBot, and links that users open from inside ChatGPT are fetched by ChatGPT-User. You can disallow GPTBot and still allow OAI-SearchBot so your pages can be cited.
Will blocking Google-Extended hurt my Google rankings?
No. Google-Extended is a control token, not a separate crawler. It only governs use of your content for Gemini model training and grounding. Googlebot crawling, Search rankings and AI Overviews eligibility are unaffected.
Is llms.txt an official standard?
No. It is a community proposal that AI tools and some SEO plugins have adopted, not an IETF or W3C standard. It costs little to publish and documents your policy in plain language, but robots.txt rules remain the main signal crawlers act on.
Do AI crawlers actually obey robots.txt?
The major vendors, including OpenAI, Anthropic, Perplexity, Google and Apple, publish their user agents and say they honor robots.txt. Compliance is still voluntary. To enforce blocks, pair robots.txt with firewall or CDN bot rules and clear terms of use.
How to Fix Missing or Vague Meta Descriptions for AI Answers
How to Fix Local Entity Data for AI Assistants
How to Add Customer Q&A Sections to Product Pages
How to Fix Missing Outbound Citations in Your Content
How to Get a Google Knowledge Graph Entity for Your Brand
How to Fix Passage-Level Content Chunking for AI Answers