How to Set an Explicit Google-Extended Policy in robots.txt

Your robots.txt has no rule for Google-Extended, so Google treats your content as available for Gemini model training and grounding by default. In this guide you will decide whether to allow or block that use, add one explicit rule to robots.txt on your platform, and confirm Google can read it.

By Updated ⏱ 15 minutes🎯 Beginner● Medium impactGEO / AI
Illustration: Google-Extended policy fix guide
GEO / AI · FIX GUIDE Google-Extended policy IMPACTEFFORT TIME15 minutes LEVELBeginner

Why it matters

txt control token that tells Google whether content crawled by Googlebot may be used to train future Gemini models and to ground answers in Gemini apps and the Vertex AI Gemini API. Without a rule, you get the default, which is allow. An explicit line documents a deliberate business decision.

It also gives legal, content and SEO teams one place to check. Neither choice changes your Google Search rankings, indexing, or eligibility for AI Overviews. Those are governed by Googlebot and snippet controls like nosnippet.

✕Beforehttps://example.com/robots.txt listsonly 'User-agent: *' and a Sitemapline, with no Google-Extended group.✓Afterhttps://example.com/robots.txtcontains 'User-agent: Google-Extended'followed by 'Allow: /' or 'Disallow:

What you need

Access to edit robots.txt (CMS, theme, code repository or server)
A decision from the content owner on whether Gemini may use the site's content
A browser or terminal to fetch https://example.com/robots.txt
Google Search Console access (optional, for the robots.txt report)

Step-by-step fix

1

Decide your Gemini policy

Agree with whoever owns the content on one of three options:

  • Allow: Gemini may train on your content and use it for grounding. This suits brands that want visibility in Gemini answers.
  • Disallow: you opt out of Gemini training and grounding. This suits publishers protecting licensed or paywalled content.
  • Partial: you block specific folders such as /members/ or /research/.

Whichever you pick, Googlebot keeps crawling and indexing your pages for Search.

Pro tip: Write the decision and date in a comment line at the top of robots.txt so future editors know the rule is intentional.
2

Check your current robots.txt

txt in a browser or run the command below. Note any existing groups, Sitemap lines and plugin-generated rules so you don't overwrite them. Also look for a misspelled token such as 'Google-Extend' or 'GoogleExtended'.

Neither is recognised.

text
curl -s https://example.com/robots.txt
Pro tip: Fetch the www and non-www hosts separately. Each hostname, including subdomains like shop.example.com, has its own robots.txt.
3

Write the Google-Extended rule

Add a separate group with User-agent: Google-Extended. Use Allow: / to permit use, Disallow: / to opt out, or Disallow lines for specific paths. Keep your existing User-agent: * group.

Google-Extended follows its own group, and the wildcard group still controls Googlebot for Search. If you also want explicit rules for other AI bots (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot), add them now.

text
# Example Co AI policy, reviewed 2026-01
User-agent: Google-Extended
Allow: /

# Alternative opt-out (use instead of the group above):
# User-agent: Google-Extended
# Disallow: /

# Partial opt-out example:
# User-agent: Google-Extended
# Disallow: /members/
# Disallow: /research/

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Pro tip: Leave a blank line between groups. A missing blank line can merge rules into the wrong group in some parsers.
4

Publish the file on your platform

Save the rule where your platform generates robots.txt:

  • Static site or server: the robots.txt file in the web root.
  • WordPress: Yoast or Rank Math.
  • Shopify: templates/robots.txt.liquid.
  • Next.js: app/robots.ts.

The file must be served at the root of the host as plain text with a 200 status. See the platform notes for exact menu paths.

javascript
// app/robots.ts (Next.js App Router)
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: 'Google-Extended', allow: '/' },
      { userAgent: ['GPTBot', 'OAI-SearchBot', 'ClaudeBot', 'PerplexityBot'], allow: '/' },
      { userAgent: '*', allow: '/' },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}
Pro tip: To opt out in Next.js, change the first rule to { userAgent: 'Google-Extended', disallow: '/' }.
5

Clear caches and purge the CDN

Purge robots.txt in your CDN or caching plugin so visitors and crawlers get the new version. Google may cache robots.txt for up to 24 hours, so allow about a day before Google applies the change.

text
curl -sI https://example.com/robots.txt
Pro tip: Check that the response shows 'content-type: text/plain' and status 200. A 404 or HTML error page means crawlers treat the site as having no rules.
6

Verify the live rule

Fetch the live file and confirm the Google-Extended group appears exactly once with the directive you chose. Then open Google Search Console → Settings → robots.txt. Confirm the latest fetch succeeded, and use Request a recrawl if the fetched version is old.

text
curl -s https://example.com/robots.txt | grep -i -A2 'google-extended'
Pro tip: Re-run your site audit. This check passes once any explicit Google-Extended rule is present, whether Allow or Disallow.

Platform-specific notes

WordPress

In Yoast SEO, go to Yoast SEO → Tools → File editor → Edit robots.txt. In Rank Math, go to Rank Math → General Settings → Edit robots.txt. If a physical robots.txt file exists in the site root, it overrides these virtual editors. Edit that file over SFTP instead, or delete it.

Shopify

Go to Online Store → Themes → … → Edit code and add the template robots.txt.liquid. Keep the default Liquid loop that outputs robots.default_groups, then add a plain-text block below it. For example: 'User-agent: Google-Extended' on one line, then 'Disallow: /' or 'Allow: /' on the next. Don't remove Shopify's default checkout and cart rules.

Next.js

With the App Router, define rules in app/robots.ts, which generates /robots.txt at build or request time. If public/robots.txt also exists, remove one of them to avoid conflicts. With the Pages Router, edit public/robots.txt directly.

Wix / Webflow

On Wix, go to Dashboard → Marketing & SEO → SEO → Tools and settings → Robots.txt Editor, and add the group under the default rules. On Webflow, go to Site settings → SEO → Indexing → robots.txt, paste the rules, save, and publish the site.

Cloudflare

Cloudflare's managed robots.txt option (under AI Crawl Control / Bots) can prepend AI-crawler rules, including a Google-Extended Disallow, to your existing file. If you enable it, make sure it doesn't contradict a Google-Extended Allow you wrote yourself. Purge the cache after any change.

Check that it worked

✅ How to verify

  • Open https://example.com/robots.txt and confirm a 'User-agent: Google-Extended' group with an Allow or Disallow line.
  • Run curl -s https://example.com/robots.txt | grep -i google-extended and confirm it returns a match on every hostname you serve.
  • In Google Search Console → Settings → robots.txt, confirm the latest fetched version includes the rule and shows no parse errors.
  • Re-run the audit and confirm the check reports that robots.txt sets an explicit Google-Extended policy.

⚠️ Common mistakes

  • Blocking Googlebot to keep content out of Gemini. This removes the site from Google Search and is the wrong control. Use Google-Extended instead.
  • Expecting Google-Extended to remove content from AI Overviews or AI Mode. Those are Search features controlled by Googlebot and snippet directives such as nosnippet.
  • Editing a plugin's virtual robots.txt while a physical robots.txt file is actually being served, so the change never goes live.
  • Misspelling the token as 'GoogleExtended' or 'Google-Extended-Bot', or adding the rule only on www while the non-www host serves a different file.

Is this still an issue on your site? Run a free audit — ZORO checks this and 294 other SEO, AI-visibility, accessibility and security signals, with fixes for your platform.

Check my site free

FAQ

Does blocking Google-Extended hurt my Google rankings?

No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. Googlebot continues to crawl and index your pages normally.

Is Google-Extended a separate crawler I'll see in my logs?

No. It is only a robots.txt product token. Crawling is done by Googlebot's existing user agents, so you won't see a Google-Extended user agent in server logs.

Does Disallow for Google-Extended remove my content from Gemini immediately?

It stops future use for Gemini training and grounding once Google re-reads your robots.txt, usually within about 24 hours. It does not retroactively remove data from models that have already been trained.

Should I allow or block Google-Extended?

Allow it if you want your brand and content to appear in Gemini answers. Block it, fully or by folder, if you license, sell or gate content and don't want it used for AI training. Either choice passes this check, as long as it is explicit.