Google-Extended is a standalone robots.txt control token for Gemini model training and grounding. Blocking it does not block Googlebot, remove pages from Google Search, or act as a Google Search ranking signal.
User-agent: Googlebot Allow: / User-agent: Google-Extended Disallow: /
Google-Extended is a standalone robots.txt product token. Publishers can use it to manage whether content Google crawls may be used for training future Gemini models and for grounding Gemini experiences with content from the Google Search index.
It is not a separate crawler with its own HTTP user-agent string. Google explains that crawling is performed with existing Google user agents, while Google-Extended acts as the control token inside robots.txt. That is why you should not expect to find a distinct Google-Extended bot string in server logs.
The example below keeps Googlebot allowed while disallowing Google-Extended. Google states that a Google-Extended preference does not affect inclusion in Google Search and is not used as a Google Search ranking signal.
This makes Google-Extended useful when your policy is: keep normal search visibility, but opt out of the specified Gemini training and grounding uses. For a broader multi-agent configuration, see the allow search, block AI training template.
User-agent: Googlebot controls Google Search crawling. User-agent: Google-Extended controls the separate Gemini-related uses described by Google. Replacing Google-Extended with Googlebot would be a materially different decision and could prevent Google from crawling affected pages for Search.
A broad User-agent: * rule can also override your intended selective policy. Run the final file through the checker and inspect the live response at /robots.txt.
Use Disallow: / to apply the preference to the whole host, or specify selected directories when only part of the site should be excluded. Remember that robots.txt is scoped per host, protocol, and port, so subdomains require their own files.
No. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not used as a Search ranking signal.
No. Google says it has no separate HTTP user-agent string; existing Google user agents perform the crawling and Google-Extended acts as a robots.txt control token.
Google documents it as a control for content use in training future Gemini models and for grounding in Gemini Apps and the Vertex AI API for Gemini.
Yes. Use path-specific Disallow directives under the Google-Extended group instead of blocking the entire host.
Yes. Robots.txt applies to a specific host, protocol, and port, so each relevant subdomain needs its own robots.txt file.
Block Google-Extended robots.txt TemplateAdd a Google-Extended Disallow group to restrict eligible content from Gemini model training and grounding uses. Google-Extended is a control token rather than a separate HTTP crawler, and Google states that blocking it does not affect inclusion or ranking in Google Search.
Google-Extended vs Googlebot: Search and Gemini ControlGooglebot is the crawler used for Google Search and related search features. Google-Extended is a standalone robots.txt control token, not a separate HTTP crawler identity; it governs whether Google-crawled content may be used for Gemini model training and grounding. You can allow Googlebot while disallowing Google-Extended without opting out of Google Search.
robots.txt for AI Training: Build a Selective Opt-OutTo restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
AI Robots.txt GeneratorChoose a policy mode, enter your website and sitemap, add any path or crawler overrides, then generate and download a robots.txt file. Publish it at the root of the correct host and verify the live rules with the checker before relying on them.
