Website owners will soon be able to tell Google, OpenAI, Microsoft and Anthropic, with one rule, not to use their content to train AI models. OpenAI, Google and Microsoft said on Friday, October 30, that they will honor one shared signal in robots.txt, the file a site uses to give instructions to the programs that crawl it. The proposal comes from Anthropic's Kevin Kelley, who submitted it as an Internet-Draft to the IETF, the body that sets the internet's technical standards. Anthropic says it has already started building support.
Training an AI model means having it read huge numbers of pages so it learns from them how to write and answer. Until now, a blogger or small online store that wanted to stay out of it needed a separate rule for each company, naming agents such as GPTBot, Google-Extended or ClaudeBot.
What robots.txt is and what to add
robots.txt is a plain text file at the root of a website, for example yoursite.com/robots.txt. Crawlers, the programs that copy pages for search engines or AI companies, check it before going in. Each block opens with a "User-Agent" line naming who the rule is for, followed by "Disallow" lines for what is off limits.
The new rule names a purpose instead of a bot. According to the Bing blog, opting out a whole site takes two lines: "User-Agent: __usage_ai-training" and, below it, "Disallow: /". The slash covers everything. To protect one section, use its path, such as "Disallow: /blog/". A site that adds nothing has stated no preference, and each company keeps applying its usual policy.
When each company will honor it
Microsoft is the only company with a date. It will honor the signal as it crawls starting February 1, 2027. Google plans to support it "in 2027," and a site that uses it to opt out will also count as blocking Google-Extended, even without naming it. OpenAI aims to implement it "in the coming months." Anthropic has not set a date, and Kelley wrote on the IETF mailing list that bringing it to all of the company's services will take time.
What the rule does not do
The signal is voluntary. Only the companies that signed on will follow it, and a crawler that already ignores robots.txt will keep doing so. The file has never been a lock, as site owners learned when shared Claude chats turned up in Google Search despite being blocked in robots.txt.
Nor does it keep bots off your pages. The draft limits the signal to training, which it defines as changing what a model has learned. Under the draft, what counts is what robots.txt said when each page was fetched, so adding the line today does not claw back anything already collected.
The signal also leaves out an AI reading your page on the spot to build an answer, which Microsoft calls grounding. Microsoft handles that with a separate preference in each page's code, and warns that opting out with it cuts a site's presence in Copilot, its AI assistant, and its other AI answers, though the page stays in regular Bing results.
What site owners can do now
You can add the two shared lines now. They won't take effect until each company starts reading them, but you won't have to remember to come back to the file. For now, the controls that actually work are company-specific: a "User-Agent: GPTBot" block with "Disallow: /" for OpenAI, and matching blocks for Google-Extended and ClaudeBot to cover Google and Anthropic. Help pages from OpenAI and Anthropic note that blocking their training bots does not pull your site from ChatGPT or Claude search, which relies on different crawlers, OAI-SearchBot and Claude-SearchBot.
All of this covers what you publish on your site. What you type into ChatGPT, Gemini or Claude is controlled in each account's settings, which we walk through in how to stop chatbots from training on your conversations.
The conversation starts here
Sign in with a supporter account to comment. Sign in
Nobody has commented yet. Want to go first?