Training data

What does training data consist of?

IBM distinguishes labeled data, where each input comes with the correct output, from unlabeled data, in which the model looks for patterns itself. Before training, data is collected, cleaned and converted into a machine-readable format. It is then split into a training set that fits the model, a validation set used to tune it, and a test set that measures the result. IBM and Stanford HAI both point out that quality and diversity of the data affect how well a model performs.

How does web content end up in training data?

Providers collect pages with their own crawlers. OpenAI separates two of them. According to its documentation, GPTBot is used for training generative AI models, and a Disallow rule signals that a site’s content should not be used for that. OAI-SearchBot surfaces sites in ChatGPT search and is a separate bot. Google uses the Google-Extended token in robots.txt to control use for Gemini training and states that it does not affect inclusion in Google Search.

The TDM Reservation Protocol (TDMRep) from a W3C Community Group adds a way to declare a text and data mining reservation in machine-readable form, through a /.well-known/tdmrep.json file, an HTTP header or a meta tag.

Example

A publisher wants its articles excluded from model training but still available in ChatGPT search. The robots.txt reads:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

The publisher also sends the header tdm-reservation: 1 to declare the reservation.

FAQ

Can I stop my content from being used as training data? You can restrict training-related crawlers in robots.txt and document a reservation with TDMRep. Whether a vendor follows these signals depends on the vendor’s own statements. A block does not undo training that already happened.

Does blocking training crawlers affect my Google rankings? For Google-Extended, Google says it has no effect on inclusion in Google Search.

Last updated: October 7, 2026