Opting Out of AI Training in robots.txt: What Google’s Shared Control Means for an Online Store

On 30 October 2026 Google announced that it plans to support a shared AI training control in robots.txt: one line a website can use to tell every participating AI developer whether its pages may be used to train models, instead of one instruction for each company. Google aims to support it, with other providers, in 2027.

For an online store the question is practical, not philosophical. Product pages, buying guides, comparisons and product photography are what AI models learn from and what AI assistants quote when a shopper asks what to buy. This is what Google has said, what its existing control already does, what opting out changes and what it does not, and how a store should decide. It sits with the rest of the commerce work behind a store that sells.

The new control is a shared “policy token” in robots.txt, based on work from the AIPREF working group at the Internet Engineering Task Force. A website uses it to set a broad preference across participating AI providers in a single step. Google’s crawler documentation names the line it intends to read, user-agent: __usage_ai-training, and gives the timing as early 2027.

Google says that when a website uses the token to opt out of training, and does not mention Google-Extended explicitly, it will read that as if the site had disallowed Google-Extended. Company-specific controls carry on working beside it, so a store can opt out of everything with the shared line, or opt out broadly and still make a different choice for one company. Google adds that the proposal is consistent with the EU’s Code of Practice for providers of general-purpose AI models under the EU AI Act, which recognises robots.txt as a primary way for websites to state machine-readable preferences.

Google-Extended has been Google’s AI training control since 2023. It is a product token, not a crawler: Google crawls with its usual user agents, and the token only appears in robots.txt, where it decides whether what Google crawls may be used for two things. One is training future Gemini models, the ones behind Gemini Apps and the Vertex AI API for Gemini. The other is grounding, the content Google draws on to make answers more accurate, in Gemini Apps and in Grounding with Google Search on Vertex AI.

The line that matters most to a store is in Google’s own documentation: Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” Blocking it does not move a product page in the search results. Google’s announcement adds the other side: websites that specifically allow Google-Extended are eligible to appear in responses, and to receive clicks, through grounding in Gemini Apps.

Google is one company. A store's robots.txt is read by many others, and until the shared token arrives each has its own name to address. Most separate the purposes in ways that matter to a store.

Training and search are often separate. Several AI companies run one crawler that gathers pages for training models and another that fetches pages to answer a user's question or build a search result. OpenAI, for example, documents GPTBot for training and OAI-SearchBot for its search features. A store can refuse the first and allow the second, keeping its pages out of training while staying eligible to be found and linked.

Some controls are tokens, like Google-Extended. Apple, for instance, uses Applebot-Extended as a separate token that governs whether content its crawler gathers may be used for training, leaving its ordinary crawling alone.

Some crawlers fetch on a user's behalf. When a person asks an assistant to read a specific page, the fetch may come from an agent that acts for that user rather than a crawler gathering pages in bulk, and companies treat those requests differently.

Read each company's own page. The names change, new ones appear, and what each one is used for is stated by the company that runs it. A list copied from a forum goes out of date within months.

Check the logs. The store's server or hosting logs show which crawlers actually visit and how often. A crawler that never visits does not need a line; one that visits heavily deserves a decision.

It does not change Google Search: Google says so for Google-Extended, and the shared token is read as a disallow of Google-Extended. It does change two things with Google. The store’s pages are no longer used to train future Gemini models, and they are no longer eligible to be drawn on, and clicked, in Gemini Apps’ grounded answers, because one token covers both. Google-Extended does not let a site refuse training while staying available for grounding.

That is the detail a store should not miss when the shared token arrives. Opting out of training with it, and saying nothing about Google-Extended, takes the store out of Gemini’s grounded answers as well. A store that wants to keep its pages out of other companies’ training but stay in Gemini’s answers would add an explicit line allowing Google-Extended, which Google says it honours ahead of the shared signal. For other AI developers, the effect depends on how each reads the token; Google says only that they can handle it similarly.

On our reading, the decision is about what each kind of page is for. A product page carries facts that dozens of retailers publish: dimensions, materials, compatibility. Keeping it from training protects very little, and keeping it available is what lets an assistant name the store when a shopper asks where to buy. A buying guide, a comparison or an original sizing chart is different: it is the store’s own work, it is exactly what an assistant would paraphrase, and it is where a store may reasonably want either credit or a refusal.

robots.txt works by path, so the choice does not have to be all or nothing. Google’s documentation shows an allow and a disallow for the same token in one group, and a store can do the same, allowing its product and category pages and disallowing a guides section, or the reverse. What a store should not do is decide by accident. Some hosting, security and content delivery services add AI-related lines to robots.txt for the sites they serve, and a store may already be refusing, or allowing, more than it chose to.

robots.txt is short and unforgiving. A few common mistakes make a file say something different from what its owner decided.

A crawler follows only its own group. Lines are grouped under user-agent names. A crawler that finds a group naming it follows that group and ignores the general group for all crawlers. A store that disallows its checkout for every crawler, then adds a group for one AI crawler, has to repeat the checkout line in that group too, or the AI crawler is no longer told about it.

The most specific path wins. Where an allow and a disallow both match a page, Google follows the longer, more specific rule. Allowing a section while disallowing one folder inside it works; the reverse needs care.

Paths match from the start. A rule for a path matches every address that begins with it. A disallow for the guides folder written without care can also catch a product page whose address happens to begin with the same letters.

The file has to be where crawlers look. At the root of the store's own address, as plain text. A robots.txt on a staging site, or one served with an error, protects nothing on the live store.

Test after every change. Search Console's robots.txt report shows the file Google last fetched and any problems it found. Fetch the live file directly as well, because a plugin or a provider may have rewritten it since it was last edited.

Keep the file under change control with the written decisions beside it, so anybody editing it later can see why each line is there.

Take a store selling kitchen equipment, with product and category pages, a guides section of original buying advice, and its own product photography. A reasonable choice, on our reading, looks like this. Product and category pages stay open to Google-Extended, because they are what an assistant needs to say where a pan can be bought, and Google says allowing Google-Extended is what makes a site eligible for clicks from Gemini’s grounded answers. The guides section is a decision the owner makes once: open, for the chance of being cited, or closed, because the advice is the business’s own and competitors’ tools should not learn it for free. The photography follows the product pages it sits on.

Written down like that, the robots.txt is a few lines grouped by token and path, and when the shared token arrives in 2027 the store adds one line for it and one explicit Google-Extended group for the choice it made for Google. Without the note, somebody repeats the whole argument next year, or a plugin decides it for them.

robots.txt is a public statement of a site's wishes. The companies that publish their crawlers' names have said they will respect it, which is why it works for them. It has limits a store should know before relying on it.

It is a request, not a lock. A crawler that chooses to ignore it can still read every public page. Stopping a crawler that misbehaves takes blocking at the server or the network, which is a different job from robots.txt.

It does not reach back in time. A disallow added today stops future collection by crawlers that honour it. It does not remove a store's pages from data gathered before the line existed.

It does not hide a page. Every visitor, human or automated, can read robots.txt and every page it names. A file that disallows a private folder announces that the folder exists. Pages that must stay private need a sign in, not a disallow.

It does not keep a page out of Google Search. For search, a page Google cannot crawl can still appear in results if other pages link to it. Keeping a page out of search takes a noindex on a page Google is allowed to crawl, which is a separate decision from any AI token.

It cannot cover copies elsewhere. Product descriptions syndicated to marketplaces, feeds sent to comparison sites, and content republished by others sit under those sites' robots.txt, not the store's.

Used for what it is, a clear and public statement that the major crawlers honour, it is the right tool for the training decision. Treated as protection, it promises more than it can give.

First, read the live robots.txt, at the store’s own address, and list every line that names an AI company or a token such as Google-Extended, including any a plugin, a theme or a provider added. Second, decide per section of the site, as above, and write it down, because the decision will need repeating for the shared token. Third, change nothing for the shared token yet: Google’s documentation says early 2027, and a line Google does not read today does nothing today.

Then measure, because the cost of either choice shows up in traffic rather than in rankings. Search Console’s reporting on how a store appears in Google’s AI features is covered in what Search Console’s generative AI report shows a store, and visits from crawlers rather than shoppers in traffic but no sales: bot sessions or a store problem. The wider question of how Google finds and shows a store is set out in Search Console for eCommerce.

We make this decision with clients as part of eCommerce SEO: the robots.txt reviewed line by line, a choice per section of the site, and a note in the calendar for when the shared token goes live. If you are not sure what your store currently allows, ask us to read it with you.