Decoding Robots.txt: Google Clarifies AI Training Limits, the Limits of RFC 9309, and the Future of Content Usage Rules

As artificial intelligence continues to reshape the digital landscape, publishers and webmasters are grappling with a complex challenge: how to welcome web traffic and search visibility while keeping their intellectual property out of greedy generative AI training pipelines.

In a recent installment of Google Search Relations’ technical video series, The Non-Canonical Cabinet, Search Advocates Gary Illyes and Cherry Prommawin tackled this exact friction point. Their core message was both sobering and instructive: under current internet standards, standard robots.txt files govern access and fetching, not post-crawl data usage.

This comprehensive deep-dive explores Google’s recent guidance, the legal and technical limitations of RFC 9309, the proprietary role of Google-Extended, ongoing efforts to draft an international standard for AI preferences, and best practices for managing internationalized web paths.

1. Main Facts: What robots.txt Can and Cannot Do for AI

The fundamental misunderstanding among many web administrators is assuming that robots.txt acts as an all-encompassing digital copyright shield. According to Google’s Gary Illyes, this is a dangerous misconception.

The Access Protocol vs. Post-Crawl Usage

Defined strictly by internet standard RFC 9309, robots.txt operates as an access preference protocol. Its exclusive function is to dictate which URLs a web crawler is permitted to request and fetch from a server.

Once a bot successfully crawls and downloads a webpage, robots.txt loses jurisdiction. The protocol does not legally or technically dictate what a scraper or AI system does with the data downstream.

• The Myth of Custom Directives: In the second episode of The Non-Canonical Cabinet, the hosts demonstrated a fictional rule (e.g., No-AI-Chicken-Training) written into a robots.txt file. Standard-compliant parsers completely ignore invented directives. According to Google’s official specifications, only four core elements are universally supported: User-Agent, Allow, Disallow, and Sitemap. Any unrecognized or custom line is simply bypassed.
• Targeting Specific Bots: To successfully block a specific AI scraper from accessing your site, webmasters must explicitly target that crawler’s specific user-agent token within the robots.txt file using standard Disallow directives.

The Google-Extended Exception

The boundary lines become more nuanced when dealing with Google’s own ecosystem. Google utilizes a specialized token known as Google-Extended.

Unlike traditional user-agents, Google-Extended does not have its own distinct crawling bot. Instead, Google executes crawls under its existing, standard user-agents, applying the Google-Extended token in a control capacity.

• What it Controls: Webmasters can use Google-Extended to dictate whether content crawled from their sites can be utilized to train future Gemini models, as well as to power grounding features in Gemini Apps and Vertex AI.
• What it Doesn’t Control: Critically, blocking Google-Extended has zero impact on a website’s inclusion, visibility, or ranking within standard Google Search results.

Crucially, the governance of Google-Extended stems directly from Google’s proprietary product documentation rather than the multi-vendor RFC 9309 standard.

2. Chronology: The Evolution of the AI Control Debate

To understand how the industry arrived at the current state of robots.txt confusion, it helps to examine the timeline of web standards, proprietary implementations, and legislative proposals.

• October 2022 (RFC 9309 Publication): The Internet Engineering Task Force (IETF) formally published RFC 9309, standardizing the robots.txt exclusion protocol for the modern web. Section 2.2.4 explicitly notes that crawlers retain the autonomy to decide how they process fetched records outside the scope of the protocol.
• August 2024 (The IETF Content-Usage Draft): Gary Illyes, alongside Mozilla’s Martin Thomson, co-authored an Internet-Draft submitted to the IETF. The proposal aims to update RFC 9309 by introducing a standardized Content-Usage rule and a matching HTTP header to explicitly separate crawling permissions from training permissions.
• September 2024 (Vocabulary Draft Expansion): A complementary vocabulary draft emerged, introducing explicit labels such as train-ai=n to give webmasters granular control over artificial intelligence utilization. However, the IETF noted that this specific vocabulary draft reflects no formal consensus within the Working Group at this time.
• Late 2024 / Early 2025 (The Non-Canonical Cabinet Launch): Google Search Relations launched its video series targeting edge-case technical SEO questions. Episode 2 brought the conversation regarding AI training versus crawling permissions to the mainstream webmaster community.
• August 2026 (Projected IETF Timeline): According to current IETF Datatracker schedules, the Working Group anticipates submitting the Content-Usage draft for formal review by the Internet Engineering Steering Group (IESG) by August 2026.
• February 2027 (Draft Expiration Deadline): Unless formally renewed, updated, or accelerated into a Request for Comments prior to this date, the current iteration of the IETF Content-Usage draft is scheduled to expire on February 20, 2027.

3. Supporting Data and Technical Nuances

Navigating technical specifications requires meticulous attention to syntax and parsing behavior. Beyond high-level strategy, the recent Google video and associated documentation highlighted critical engineering mechanics that every web developer should master.

Percent-Encoding for Internationalized Paths

Cherry Prommawin offered crucial advice for site owners operating multilingual websites or URLs featuring non-English characters and emojis: always use percent-encoded forms in your robots.txt rules.

• The Google Behavior: Google’s internal parser is forgiving; it automatically normalizes raw characters and emojis by encoding them internally.
• The Interoperability Risk: Other crawlers, scrapers, or legacy parsers across the open web lack this advanced normalization. If a webmaster leaves raw characters or emojis in a robots.txt path, alternative bots may misinterpret the rule, resulting in broken blocking rules or unintended exposure.
• The Standard Way: The safe, universally compliant approach is to translate non-ASCII characters into their percent-encoded equivalents (e.g., UTF-8 hex values preceded by %). While RFC 9309 grammar technically permits UTF-8 characters in paths, percent-encoding guarantees uniform interpretation across diverse scraper architectures.

Breakdown of Current AI Control Methods

4. Official Responses and Industry Reactions

The intersection of generative AI and web publishing has triggered intense dialogue across legal, engineering, and SEO communities.

When asked whether a single robots.txt configuration could simultaneously allow a bot to index a page for search discovery while forbidding it from ingesting those same words for generative modeling, Google’s official stance is clear: No universal standard currently exists to achieve this cleanly.

The Gap Between Access and Intent

Publishers have expressed frustration that tech giants utilize their content for AI training under the guise of web discovery. Industry advocates argue that forcing webmasters to block standard crawlers—thereby sacrificing organic search traffic—just to prevent AI ingestion is an unviable ultimatum.

By co-authoring the IETF Internet-Draft with Mozilla’s Martin Thomson, Google’s Gary Illyes has signaled institutional recognition of this flaw. The proposed Content-Usage rule attempts to bridge this gap by establishing a cooperative framework. Under the draft’s vision, a webmaster could write a rule indicating: “Yes, you may fetch this URL for indexing, but train-ai=n dictates you may not use it to train your foundational models.”

The Limits of Voluntary Compliance

Critics of the draft point out a glaring vulnerability: robots.txt has historically been a voluntary protocol honored by ethical actors. Rogue AI scrapers, bad-faith data brokers, and underground crawling operations routinely ignore standard robots.txt exclusions entirely. Therefore, while standardized usage rules will help regulate compliant enterprise AI developers (such as OpenAI, Anthropic, and Google), they will not automatically deter malicious actors. For hard enforcement, publishers must rely on server-side firewalls, IP rate-limiting, and behavioral bot-detection software.

5. Practical Implications for Webmasters and SEOs

As search engines evolve and AI models demand an endless supply of fresh training data, digital marketers and technical SEO professionals must adapt their infrastructure strategies. Here is how you should act on Google’s recent guidance:

1. Audit Your robots.txt Immediately

Check your robots.txt files for obsolete, homemade, or fictional directives. If you are using custom tags like No-AI-Training, understand that they are dead weight; compliant bots ignore them entirely.

• To block specific AI crawlers, explicitly identify their user-agent strings and apply clear Disallow rules.
• Familiarize yourself with proprietary tokens like Google-Extended if you wish to opt out of Gemini training without harming your traditional Google Search rankings.

2. Enforce Percent-Encoding on International Paths

If your website features URLs in non-Latin scripts (Cyrillic, Arabic, Chinese, Japanese, etc.) or utilizes emojis in directory structures, audit your robots.txt paths immediately. Convert raw UTF-8 strings into their percent-encoded variants to ensure third-party crawlers interpret your access rules accurately.

3. Monitor the IETF Standard

Keep a close eye on the IETF Content-Usage draft over the next year and a half. As the proposal progresses through the Working Group toward its projected 2026 IESG review, webmasters should prepare to adopt new HTTP headers and robots.txt syntax designed specifically to decouple web crawling from AI training rights.

4. Diversify Your Defense Strategy

Recognize that robots.txt is only one layer of your digital security and content governance stack. Relying solely on text files will not stop determined scrapers. Combine explicit user-agent blocking with robust server-side analytics, Cloudflare or similar edge-security firewalls, and clear terms-of-service agreements to protect your proprietary intellectual property.

The ongoing conversation surrounding AI scraping and robots.txt highlights a pivotal transitional era for the web. As technical standards slowly catch up to commercial realities, webmasters must remain vigilant, adaptable, and proactive in defending their digital assets.