TF-IDF

How is TF-IDF calculated?

The score is the product of two values.

Term frequency (TF) measures how often term t appears in document d. Common variants are the raw count, the count divided by the document length, and a logarithmic version such as log(1 + count) that dampens repetition.

Inverse document frequency (IDF) measures how rare the term is across the whole collection:

idf(t) = log(N / n_t)
tf-idf(t, d) = tf(t, d) × idf(t)

N is the number of documents, n_t the number of documents that contain t. A word that appears in every document gets an IDF of zero. Libraries often use smoothed versions. scikit-learn, for example, adds 1 to numerator and denominator and adds 1 to the result by default, so scores from different tools are not directly comparable.

Where does it come from?

The idea behind IDF was published by Karen Spärck Jones in 1972 in the Journal of Documentation. Her argument was that a term found in only a few documents says more about those documents than a term found everywhere. Combined with term frequency, the method became a standard building block of text search and later of machine learning on text.

In German-speaking SEO, a variant called WDF*IDF is common (an assessment based on search data, not on a primary source). WDF (within-document frequency) is a logarithmic form of term frequency.

What does TF-IDF mean for SEO?

Content tools compare a page with the pages that rank for a keyword. They list terms that are frequent across the competitors and missing or rare on your own page. Used this way, TF-IDF is a research aid for spotting subtopics a text doesn’t cover yet.

The method has clear limits. It counts words and ignores meaning, word order, and synonyms. Many tools analyze the entire HTML page, including navigation and footer text. Google has not confirmed using TF-IDF as a ranking method, and its guidance on helpful content says there is no preferred word count. Adding terms only to hit a target score can turn into keyword stuffing, which violates Google’s spam policies.

Example

In a made-up example, a collection has 1,000 product guides. In a 500-word guide on descaling coffee machines, the word “descaling” appears 10 times (TF = 10/500 = 0.02). Only 10 guides in the collection use the word, so IDF = log10(1,000/10) = 2 and TF-IDF = 0.04. The word “coffee” appears just as often in the guide, but 900 guides contain it. Its IDF is log10(1,000/900) ≈ 0.05, and its TF-IDF ends up around 0.001. “Descaling” describes the guide far better than “coffee,” and the scores reflect that.

FAQ

What are the limitations of TF-IDF? It treats text as a bag of words. It cannot tell that “car” and “automobile” mean the same thing, and very rare terms such as typos can get high scores.

Is TF-IDF the same as keyword density? No. Keyword density is the share of one word among all words on a page. TF-IDF also considers how common the word is across a set of other documents.

How do you calculate IDF? Divide the total number of documents by the number of documents that contain the term and take the logarithm of the result.

Last updated: October 8, 2026