Googlebot is the crawler used for Google Search and related search features. Google-Extended is a standalone robots.txt control token, not a separate HTTP crawler identity; it governs whether Google-crawled content may be used for Gemini model training and grounding. You can allow Googlebot while disallowing Google-Extended without opting out of Google Search.
User-agent: Googlebot Allow: / User-agent: Google-Extended Disallow: /
Googlebot and Google-Extended appear in robots.txt, but they do not describe two equivalent crawlers. Googlebot is the familiar web crawler used to discover and refresh content for Google Search. Google-Extended is a product control token that tells Google whether content it has crawled may be used for specified Gemini training and grounding purposes.
This separation lets publishers keep ordinary search visibility while opting out of the extended AI uses covered by the token.
Rules addressed to Googlebot affect crawling for Google Search, including Search features such as Discover and several dependent surfaces. Blocking Googlebot can prevent Google from fetching page content and resources. It is therefore a major SEO decision, not a narrow AI preference.
Robots.txt manages crawling, not guaranteed removal from the index. A blocked URL may still be discovered through links and displayed without a normal snippet. Use an appropriate noindex directive on a crawlable page, authentication, or removal tooling when exclusion from search results is the actual goal.
Google-Extended does not have a separate HTTP user-agent string. Google explains that crawling is performed with existing Google user agents and the token works in a control capacity. The preference applies to use of crawled content for training future Gemini models and for grounding in specified Gemini and Vertex AI experiences.
Google also states that Google-Extended does not affect inclusion in Google Search and is not a Google Search ranking signal. You should not expect to find a distinct “Google-Extended” crawler in access logs.
The common selective policy is straightforward:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /The first group leaves Google Search crawling available. The second records your preference for the uses covered by Google-Extended. Keep the groups separate and avoid a wildcard rule such as User-agent: * Disallow: /, which would request that all compliant crawlers stay out.
Because Google-Extended is not a separate request identity, a log entry from Googlebot does not tell you whether the fetched content will later be considered under the Google-Extended preference. Evaluate the published robots.txt configuration rather than trying to block a nonexistent HTTP user-agent string at the firewall.
When validating genuine Google crawler traffic, use Google’s documented reverse-DNS or published-IP verification process. A client can copy the text “Googlebot” into its header.
For a multi-provider version of this policy, use Allow AI Search but Block AI Training.
No. Google says Google-Extended has no separate HTTP user-agent string. Existing Google user agents perform the crawling, while Google-Extended acts as a robots.txt control token.
No. Google states that Google-Extended does not affect inclusion in Google Search and is not used as a Search ranking signal. Keep Googlebot allowed if search visibility matters.
It controls whether Google-crawled content may be used for training future Gemini models and for grounding in specified Gemini Apps and Vertex AI experiences.
You can write those groups, but blocking Googlebot prevents normal Search crawling and may leave Google with little or no newly fetched content to which an extended-use preference could apply. The usual selective policy is the reverse.
Not as a separate crawler category. Validate genuine Google crawler traffic using Google’s published verification methods, but evaluate Google-Extended through the robots.txt token rather than a dedicated request identity.
AI Crawler ListThis directory groups crawler tokens by purpose so you can decide what to allow or block without treating every automated request as the same. Open a crawler page for its documented role, exact robots.txt token, policy trade-offs, and verification guidance.
Google-Extended robots.txt Control for GeminiGoogle-Extended is a standalone robots.txt control token for Gemini model training and grounding. Blocking it does not block Googlebot, remove pages from Google Search, or act as a Google Search ranking signal.
How to Allow AI Search but Block AI TrainingUse separate User-agent groups for each purpose: allow AI search crawlers such as OAI-SearchBot and Claude-SearchBot, while disallowing training-oriented crawlers such as GPTBot and ClaudeBot. These controls are independent, so do not block an entire provider when your goal is only to opt out of training.
robots.txt for AI Training: Build a Selective Opt-OutTo restrict AI training access, identify each provider’s exact training or dataset token and disallow it in a dedicated robots.txt group. Keep search crawlers in separate allowed groups, treat product-control tokens such as Google-Extended according to their documentation, and use authentication or server-side controls when access must be enforced.
robots.txt for AI Search Visibility and CitationsTo support AI search visibility, allow each provider’s documented search crawler—such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot—while keeping Googlebot and Bingbot accessible. Robots.txt only permits crawling; it does not guarantee indexing, citation, ranking, or access through a firewall.