# Transformer

Category: [AI Search & GEO](https://seo-geo-glossary.com/en/glossary/category/ai-search-geo/) › [AI & LLM basics](https://seo-geo-glossary.com/en/glossary/category/ai-search-geo/ai-llm-basics/)

A **transformer** is a neural network architecture that processes text and other sequences with an attention mechanism, called self-attention. It was introduced in 2017 in the paper "Attention Is All You Need" and underlies modern language models. The word also stands for electrical equipment and a film series, which this entry does not cover.

## How does a transformer work?

Ashish Vaswani and co-authors posted the paper on June 12, 2017. Their model relies entirely on attention mechanisms and drops recurrent and convolutional networks. The authors reported better translation results with less training time, for example 28.4 BLEU on English-to-German translation and 41.8 on English-to-French.

In practice, text is first split into tokens and converted into vectors (embeddings). Positional encoding adds information about word order, since the model processes all positions in parallel. Self-attention then works out, for every token, how strongly it relates to each other token in the input. IBM describes this using alignment scores that are turned into attention weights between 0 and 1. Because the calculations run in parallel, transformers train faster than older sequential networks such as RNNs, as AWS and NVIDIA also point out.

## What variants exist?

IBM distinguishes encoder-only models such as BERT, which suit understanding tasks like classification, from decoder-only models such as GPT (Generative Pre-trained Transformer), which generate text one token at a time. Vision transformers apply the same idea to images.

## What does it mean for SEO and AI search?

Language models in AI answers read page content as tokens inside a limited context window. As an assessment, this is one technical reason why systems usually load selected passages instead of whole sites.

## Example

In the sentence "The bank by the river was freshly painted", self-attention can give the word "river" strong weight when interpreting "bank". The model then reads it as a place to sit, not a financial institution. The example is simplified.

## FAQ

**What is a transformer model?**
A neural network architecture that uses self-attention to model relationships between all parts of an input sequence.

**Is ChatGPT a transformer model?**
GPT stands for Generative Pre-trained Transformer, so the architecture is part of the name.

## Related terms

- [Large language model (LLM)](https://seo-geo-glossary.com/en/glossary/large-language-model/index.md)
- [BERT](https://seo-geo-glossary.com/en/glossary/bert/index.md)
- [NLP (natural language processing)](https://seo-geo-glossary.com/en/glossary/nlp/index.md)
- [Token](https://seo-geo-glossary.com/en/glossary/token/index.md)
- [Context window](https://seo-geo-glossary.com/en/glossary/context-window/index.md)
- [Vector search](https://seo-geo-glossary.com/en/glossary/vector-search/index.md)

## Sources

- [Attention Is All You Need - arXiv:1706.03762 (Vaswani et al., 12 June 2017)](https://arxiv.org/abs/1706.03762)
- [What is a transformer model? - IBM Think](https://www.ibm.com/think/topics/transformer-model)
- [What is a transformer model? - NVIDIA Blog](https://blogs.nvidia.com/blog/what-is-a-transformer-model/)
- [What are transformers in AI? - AWS](https://aws.amazon.com/what-is/transformers-in-artificial-intelligence/)

---

Last updated: October 7, 2026
URL: https://seo-geo-glossary.com/en/glossary/transformer/