Google just spelled out exactly what robots.txt can and can’t do about AI, and the answer disappoints anyone hoping a text file could keep their content out of AI training.
TL;DR: In a new Google Search Central video series, Search Relations staffers Gary Illyes and Cherry Prommawin confirmed that robots.txt only controls whether a crawler is allowed to access a page, not what happens to the content after it’s been crawled. The file follows RFC 9309, supports exactly four directives, and silently ignores anything it doesn’t recognize, including hand-written rules trying to say “crawl this for search but not for AI training.” Google also recommends percent-encoding non-English characters and emoji in URL paths so other crawlers don’t misread your rules.
A lot of site owners treat robots.txt like a permissions panel where you can tick boxes for different use cases. Google’s new explainer makes clear that’s never been how it works. “Robots.txt is an access preference protocol,” according to the video. “It doesn’t dictate what a system does with the data after it has already crawled it.”
Disallow means don’t enter the room, nothing more
The framing Google used is useful precisely because it’s so blunt: “Disallow bluntly means do not enter the room.” That’s the entire scope of the file. It’s a lock on a door, not a contract about what happens once someone’s inside.
If you want to stop a specific AI crawler from indexing your site at all, the video says you have to target that crawler’s specific user agent token explicitly. There’s no shortcut, no wildcard clause that says “block all AI, allow search,” because robots.txt was never built with that distinction in mind. It predates the current fight over AI training data by decades.
Site owners who’ve tried getting creative with robots.txt, writing rules in plain English or inventing their own custom directives hoping a smarter crawler will understand the intent, are wasting their time according to Google’s own parser behavior. Only four rules get recognized: user agent, allow, disallow, and sitemap. Anything else, including a cleverly worded prose instruction, gets thrown out silently. The file doesn’t error, doesn’t warn you, and doesn’t apply the rule. It just moves on as if the line was never there.
The encoding detail nobody thinks about until it breaks
The second episode of the series, called The Non-Canonical Cabinet, also covered a technical wrinkle that trips up international sites more than most people realize. Google’s own parser accepts raw UTF-8 characters in URL paths without complaint. But Illyes and Prommawin recommend using the percent-encoded version of those characters anyway, because not every crawler reading your robots.txt file handles raw non-English characters or emoji the same way Google’s does.
Put a path with an emoji or a non-Latin character into your robots.txt file unencoded, and Google might read it fine while a smaller or less forgiving crawler chokes on it. The fix is simple: run non-ASCII paths through percent-encoding before they go into the file, rather than assuming every bot parses Unicode the way Google’s does.
User-agent tokens themselves are stricter still. Those can’t contain emoji or non-English characters at all, according to the video, full stop.
This confusion has been building for years
None of this is entirely new ground. Google introduced the Google-Extended user agent token specifically so sites could opt certain content out of being used for Gemini training while still allowing normal Search crawling, which is proof the underlying demand for finer control is real. What this new video series does is close the gap between that narrow, purpose-built exception and the much broader misconception that robots.txt as a whole can express that kind of nuance on a per-use-case basis. It can’t, and apparently a lot of site owners needed to hear that directly from Google rather than infer it from scattered documentation updates.
The Non-Canonical Cabinet series itself is a notable format choice. Google picking a conversational video series, rather than another dense documentation page, to explain something this fundamental suggests the company has accepted that written docs alone weren’t getting the message across. The first episode, published a week earlier, tackled a related mess: how Google’s crawler handles unusual characters in domain names and URLs, a problem that overlaps heavily with the percent-encoding advice in this second episode.
What this actually means if you’re trying to keep AI out
None of this gives site owners a real lever over AI training the way a lot of people assumed robots.txt gave them. If you want a specific AI company’s crawler blocked from your site entirely, you can do that with the right user agent directive. What you can’t do is tell a crawler “come here for search visibility but skip the training pipeline,” because that distinction lives entirely outside what robots.txt was designed to express. Anyone who wants that kind of control needs to look at license terms, licensing agreements, or newer proposals like content signals, not a decades-old access file.
Related: This clarification lands right as a new Semrush study shows AI Overviews now appearing on 88% of tracked commercial search results, which makes the question of what AI crawlers can and can’t use that much more pressing. It also follows Google’s own guidance after September’s spam update on how crawl behavior and site availability interact.
The bottom line
Robots.txt was never an AI opt-out switch, and Google just said so in plain language instead of letting the confusion run another year. If you’re trying to manage how AI systems use your content, target specific crawler user agents with real disallow rules, and stop writing custom instructions into the file hoping a smarter bot will understand what you meant. It won’t. It’ll just skip the line and keep crawling.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
