Token

How long is a token?

Microsoft describes tokens as words, character sets or combinations of words and punctuation, produced by a tokenizer. OpenAI gives a rule of thumb of about 4 characters or 0.75 words per token for English text. Common words are often a single token, and rare or long words are split into several. Languages differ, and the exact split depends on the model’s tokenizer, so the same text can have a different token count in different models. Anthropic, for example, says 1 million tokens correspond to about 555,000 words with its current tokenizer, while earlier models fit about 750,000 words.

After splitting, each token gets a numeric ID from the model’s vocabulary. The model then works with these IDs, for example by turning them into embeddings. It generates its answer one token at a time.

Why are tokens important?

Limits are given in tokens. Per OpenAI, the prompt and the generated output together must not exceed the model’s maximum context length. This limit is the context window. Many providers also bill by tokens, and Microsoft notes that cost can differ between input and output tokens.

To prepare prompts or page content for an AI application, you can count tokens with the tokenizer of the target model.

Example

The English sentence “I heard a dog bark loudly at a cat” has nine words. With word-based tokenization it is nine tokens, and with character-based tokenization it is 34, as Microsoft’s example shows. Real models mostly use subword tokenization, so the count falls in between and depends on the model.

FAQ

How many words is one token? About 0.75 words for English text as a rough guide. It varies by language and model.

Why do LLMs use tokens instead of words? Subword tokens let a model handle rare words, typos and many languages with a vocabulary of manageable size.

Last updated: October 7, 2026