How to Set an Explicit Google-Extended Policy in robots.txt
Your robots.txt has no rule for Google-Extended, so Google treats your content as available for Gemini model training and grounding by default. In this guide you will decide whether to allow or block that use, add one explicit rule to robots.txt on your platform, and confirm Google can read it.

Why it matters
txt control token that tells Google whether content crawled by Googlebot may be used to train future Gemini models and to ground answers in Gemini apps and the Vertex AI Gemini API. Without a rule, you get the default, which is allow. An explicit line documents a deliberate business decision.
It also gives legal, content and SEO teams one place to check. Neither choice changes your Google Search rankings, indexing, or eligibility for AI Overviews. Those are governed by Googlebot and snippet controls like nosnippet.
What you need
Step-by-step fix
Decide your Gemini policy
Agree with whoever owns the content on one of three options:
- Allow: Gemini may train on your content and use it for grounding. This suits brands that want visibility in Gemini answers.
- Disallow: you opt out of Gemini training and grounding. This suits publishers protecting licensed or paywalled content.
- Partial: you block specific folders such as /members/ or /research/.
Whichever you pick, Googlebot keeps crawling and indexing your pages for Search.
Check your current robots.txt
txt in a browser or run the command below. Note any existing groups, Sitemap lines and plugin-generated rules so you don't overwrite them. Also look for a misspelled token such as 'Google-Extend' or 'GoogleExtended'.
Neither is recognised.
curl -s https://example.com/robots.txt
Write the Google-Extended rule
Add a separate group with User-agent: Google-Extended. Use Allow: / to permit use, Disallow: / to opt out, or Disallow lines for specific paths. Keep your existing User-agent: * group.
Google-Extended follows its own group, and the wildcard group still controls Googlebot for Search. If you also want explicit rules for other AI bots (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot), add them now.
# Example Co AI policy, reviewed 2026-01 User-agent: Google-Extended Allow: / # Alternative opt-out (use instead of the group above): # User-agent: Google-Extended # Disallow: / # Partial opt-out example: # User-agent: Google-Extended # Disallow: /members/ # Disallow: /research/ User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
Publish the file on your platform
Save the rule where your platform generates robots.txt:
- Static site or server: the robots.txt file in the web root.
- WordPress: Yoast or Rank Math.
- Shopify: templates/robots.txt.liquid.
- Next.js: app/robots.ts.
The file must be served at the root of the host as plain text with a 200 status. See the platform notes for exact menu paths.
// app/robots.ts (Next.js App Router)
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{ userAgent: 'Google-Extended', allow: '/' },
{ userAgent: ['GPTBot', 'OAI-SearchBot', 'ClaudeBot', 'PerplexityBot'], allow: '/' },
{ userAgent: '*', allow: '/' },
],
sitemap: 'https://example.com/sitemap.xml',
}
}Clear caches and purge the CDN
Purge robots.txt in your CDN or caching plugin so visitors and crawlers get the new version. Google may cache robots.txt for up to 24 hours, so allow about a day before Google applies the change.
curl -sI https://example.com/robots.txt
Verify the live rule
Fetch the live file and confirm the Google-Extended group appears exactly once with the directive you chose. Then open Google Search Console → Settings → robots.txt. Confirm the latest fetch succeeded, and use Request a recrawl if the fetched version is old.
curl -s https://example.com/robots.txt | grep -i -A2 'google-extended'
Platform-specific notes
WordPress
In Yoast SEO, go to Yoast SEO → Tools → File editor → Edit robots.txt. In Rank Math, go to Rank Math → General Settings → Edit robots.txt. If a physical robots.txt file exists in the site root, it overrides these virtual editors. Edit that file over SFTP instead, or delete it.
Shopify
Go to Online Store → Themes → … → Edit code and add the template robots.txt.liquid. Keep the default Liquid loop that outputs robots.default_groups, then add a plain-text block below it. For example: 'User-agent: Google-Extended' on one line, then 'Disallow: /' or 'Allow: /' on the next. Don't remove Shopify's default checkout and cart rules.
Next.js
With the App Router, define rules in app/robots.ts, which generates /robots.txt at build or request time. If public/robots.txt also exists, remove one of them to avoid conflicts. With the Pages Router, edit public/robots.txt directly.
Wix / Webflow
On Wix, go to Dashboard → Marketing & SEO → SEO → Tools and settings → Robots.txt Editor, and add the group under the default rules. On Webflow, go to Site settings → SEO → Indexing → robots.txt, paste the rules, save, and publish the site.
Cloudflare
Cloudflare's managed robots.txt option (under AI Crawl Control / Bots) can prepend AI-crawler rules, including a Google-Extended Disallow, to your existing file. If you enable it, make sure it doesn't contradict a Google-Extended Allow you wrote yourself. Purge the cache after any change.
Check that it worked
✅ How to verify
- Open https://example.com/robots.txt and confirm a 'User-agent: Google-Extended' group with an Allow or Disallow line.
- Run curl -s https://example.com/robots.txt | grep -i google-extended and confirm it returns a match on every hostname you serve.
- In Google Search Console → Settings → robots.txt, confirm the latest fetched version includes the rule and shows no parse errors.
- Re-run the audit and confirm the check reports that robots.txt sets an explicit Google-Extended policy.
⚠️ Common mistakes
- Blocking Googlebot to keep content out of Gemini. This removes the site from Google Search and is the wrong control. Use Google-Extended instead.
- Expecting Google-Extended to remove content from AI Overviews or AI Mode. Those are Search features controlled by Googlebot and snippet directives such as nosnippet.
- Editing a plugin's virtual robots.txt while a physical robots.txt file is actually being served, so the change never goes live.
- Misspelling the token as 'GoogleExtended' or 'Google-Extended-Bot', or adding the rule only on www while the non-www host serves a different file.
Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.
Check my site freeFAQ
Does blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. Googlebot continues to crawl and index your pages normally.
Is Google-Extended a separate crawler I'll see in my logs?
No. It is only a robots.txt product token. Crawling is done by Googlebot's existing user agents, so you won't see a Google-Extended user agent in server logs.
Does Disallow for Google-Extended remove my content from Gemini immediately?
It stops future use for Gemini training and grounding once Google re-reads your robots.txt, usually within about 24 hours. It does not retroactively remove data from models that have already been trained.
Should I allow or block Google-Extended?
Allow it if you want your brand and content to appear in Gemini answers. Block it, fully or by folder, if you license, sell or gate content and don't want it used for AI training. Either choice passes this check, as long as it is explicit.
How to Fix a CDN or WAF Blocking AI Crawlers
How to Allow PerplexityBot in robots.txt and Your Firewall
How to Create and Publish an llms.txt File for AI Tools
How to Allow ClaudeBot and Claude-SearchBot in robots.txt
How to Fix Content That Isn't Readable Without JavaScript
How to Allow GPTBot and OAI-SearchBot in robots.txt