ai.txt is a proposed file name for a text file in which website operators state how AI systems may use their content. Several proposals use the name. It is not an established standard, as of October 6, 2026.
At a glance
- Publisher
- Individual author Kayla Cardillo (independent), no IETF body
- Connects
- Websites and AI systems (usage preferences, licensing)
- Status
- Internet-Draft, not a standard; expires 2026-12-14
- Version
- draft-car-ai-txt-wellknown-00 (2026-06-12)
- Specification
- IETF draft for
/.well-known/ai.txt
Last checked:
What is the status of ai.txt?
The name goes back to Spawning, which announced ai.txt on May 30, 2023. In Spawning’s version, the file sits in the root directory of a site and states whether text, images, audio, video and code may be used for commercial AI training. Spawning’s generator blocks all content types by default. According to Spawning, the file is read when media is downloaded, while robots.txt applies at crawl time. Spawning says its API passes these permissions on to AI researchers and partners, including Hugging Face and Stability AI. Whether those partners act on them is not independently documented (assessment).
In 2024, Guardian News & Media submitted a paper to the IAB AI-CONTROL workshop that proposes an AI.txt protocol modeled on robots.txt.
The most formal document found is an IETF Internet-Draft titled “AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy”, submitted by an independent author, K. Cardillo, on June 12, 2026. The draft proposes machine-readable files, ai.txt in a block-based key-value format inspired by robots.txt and ai.json as a structured alternative. They are meant to declare training permissions, licensing terms and per-agent rules.
The draft says it is a work in progress. Internet-Drafts are valid for a maximum of six months, and the document expires on December 14, 2026. It is not an adopted standard, and the IETF notes that drafts should only be cited as work in progress. The checked specifications RFC 9309, llmstxt.org and the W3C TDMRep report do not mention ai.txt.
| File | Standard or proposal? | Location | Purpose | Who reads it (documented) |
|---|---|---|---|---|
| robots.txt | IETF standard: RFC 9309 (September 2022) | / | tells crawlers which URLs they may fetch; per the RFC not a form of access authorization | Crawlers; Google's common crawlers always respect it for automatic crawls |
| llms.txt | Proposal (Jeremy Howard, 2024; version 2 from August 2026) | / or in a subpath | Markdown overview linking to Markdown versions of pages, for AI agents | per the proposal mainly coding agents; Google Search ignores it, says Google; Lighthouse only checks the fetch |
| llms-full.txt | no specification; not part of the llms.txt proposal, a convention of individual vendors (e.g. Mintlify) | / (Mintlify also serves /.) | entire documentation as Markdown in one file | no reader documented; Google Search doesn't use such files, says Google |
| ai.txt (this term) | Individual IETF Internet-Draft (June 2026, expires 2026-12-14), not a standard; predecessor by Spawning (2023) | /. (draft), / (Spawning) | AI usage preferences (training, scraping, indexing, caching), licensing, attribution | no reader documented in primary sources; Google Search doesn't use 'AI text files', says Google |
TDMRep (tdmrep.) | W3C Community Group Final Report (2024-05-10), not a W3C standard | /., HTTP header tdm-reservation or HTML meta tag | machine-readable reservation of text and data mining rights (Art. 4 DSM Directive) | parties performing text and data mining (per the specification) |
| security.txt | RFC 9116 (April 2022, informational, not standards track) | /. | contact for reporting security vulnerabilities; required fields Contact and Expires | Security researchers |
| ads.txt | IAB Tech Lab specification, version 1.1 (August 2022) | / on the root domain, subdomains as needed | declares the companies authorized to sell the site's ad inventory | advertising systems and buyers (ad system crawlers) |
| humans.txt | Initiative (humanstxt.org), no formal specification | / next to robots.txt, optionally linked via rel="author" | names the people who contributed to the website | people who want to know who is behind the website |
Only robots.txt, security.txt and ads.txt are based on a finalised specification (IETF or IAB Tech Lab). Google states that Google Search doesn't use new machine-readable files or 'AI text files' and ignores llms.txt files, which neither help nor harm Search visibility. TDMRep is a technical way to express the reservation under Art. 4 of Directive (EU) 2019/790. Not legal advice. Sources: RFC 9309: Robots Exclusion Protocol; Google: Overview of Google crawlers and fetchers (last updated 2026-06-12); The /llms.txt file, v2 (llmstxt.org); llmstxt.org: Changes (v2, August 2026); Google Search Central: Optimizing for generative AI features (last updated 2026-07-10); Chrome for Developers: Lighthouse llms.txt audit (2026-05-05); Mintlify: llms.txt and llms-full.txt; IETF Internet-Draft draft-car-ai-txt-wellknown-00 (2026-06-12); W3C TDMRep Community Group: TDM Reservation Protocol, Final Report 2024-05-10; RFC 9116: A File Format to Aid in Security Vulnerability Disclosure; IAB Tech Lab: ads.txt; IAB Tech Lab: ads.txt Specification Version 1.1 (GitHub); humanstxt.org. Last checked: .
Which documented options exist instead?
- robots.txt: The Robots Exclusion Protocol has been published as RFC 9309 since September 2022. It tells crawlers per user agent which areas to fetch, and the RFC states that these rules are not a form of access authorization.
- llms.txt: A Markdown file with a list of links, proposed by Jeremy Howard on September 3, 2024. It helps AI systems read content and does not regulate usage rights.
- TDMRep: The TDM Reservation Protocol of a W3C Community Group (final report of May 10, 2024) declares a text and data mining reservation in machine-readable form.
If you come across a file called ai.txt, find out which specification it refers to and whether any crawler reads it.
Example
A site owner wants to exclude AI training on the site. A documented way is a robots.txt entry for the relevant bot, for instance User-agent: Google-Extended with Disallow: /. A rights reservation can be added with TDMRep.
FAQ
Do search engines or AI providers read ai.txt? For search engines, the sources checked show no evidence of it. Spawning says it passes ai.txt permissions on to partners through its API. Google also states that no new AI text files are needed for its AI features.
Sources
- AI.TXT: A Declaration File for AI Usage Preferences, Licensing, and Policy - IETF Internet-Draft (K. Cardillo, 12 June 2026)ietf.org
- RFC 9309: Robots Exclusion Protocol (September 2022)rfc-editor.org
- The /llms.txt file - llmstxt.orgllmstxt.org
- TDM Reservation Protocol (TDMRep), Final Report 10 May 2024 - W3Cw3.org
- llms.txt vs robots.txt vs ai.txt - DEV Community (secondary source)dev.to
- ai.txt: A new way for websites to set permissions for AI – Spawning (May 30, 2023)spawning.substack.com
- Spawning's ai.txt Generator – Spawningsite.spawning.ai
- Guardian News & Media: Draft Paper on an AI.txt Protocol – IAB Workshop AI-CONTROL (2024)datatracker.ietf.org
- draft-car-ai-txt-wellknown – IETF Datatrackerdatatracker.ietf.org
Last updated: