Block Google-Extended Without Blocking Googlebot

Copy the Google-Extended rule

Use this crawler-specific group when you want Google Search to continue crawling the site but do not want eligible content used through the products controlled by Google-Extended:

User-agent: Google-Extended
Disallow: /

The rule can also target selected paths instead of the whole host. For example, replace / with /members/ or another public section whose use you want to restrict.

Google-Extended is not Googlebot

Google-Extended is a standalone robots.txt product token. It does not have its own separate HTTP User-Agent string; Google crawls with existing Google agents and applies this token as a control over whether eligible content may be used for training future Gemini models and for certain grounding uses.

Google states that a Google-Extended preference does not affect a site’s inclusion in Google Search and is not used as a Google Search ranking signal. Do not block Googlebot when your intention is only to opt out of Google-Extended uses.

Keep Search open explicitly only when needed

A standalone Google-Extended group does not require an additional Googlebot group in a simple file. If your existing robots.txt has a wildcard block, however, add or review the more specific Googlebot rules so the final configuration still matches your search policy:

User-agent: Google-Extended
Disallow: /

User-agent: Googlebot
Allow: /

Always evaluate the entire file. Repeated groups for the same token may be combined by the crawler, and a broad rule elsewhere can create an outcome that is not obvious from one copied snippet.

Publish the rule correctly

1. Merge the Google-Extended group into the robots.txt file for the exact host you want to control.
2. Serve it from the host root, such as https://example.com/robots.txt.
3. Confirm that the live response is plain text and returns HTTP 200.
4. Test Google-Extended and Googlebot separately with the robots.txt checker.

Because Google-Extended is a control token, do not expect to find a distinct Google-Extended crawler string in server logs. If you need to verify whether a request genuinely comes from Google infrastructure, use Google’s published crawler verification methods rather than trusting a self-declared header.

Know the limit of the template

This rule communicates a use preference to Google. It does not make a public page private, remove it from Google Search, or prevent every third party from copying it. Use noindex for supported search-indexing control, and use authentication or authorization when access itself must be restricted.