# What is a transformer?

Transformers are neural networks built from a stack of layers that each combine an attention block with a feed-forward block. The transformer is the architecture behind large language models.

## How it works

Text is split into tokens, and each token becomes a vector. The vectors pass through a stack of layers that share one structure. Each layer has two parts:

- **Attention.** Every token looks at the other tokens and takes in information from the ones that matter to it. This is how the model uses context.
- **A feed-forward block**, also called an MLP (multilayer perceptron). It transforms each token's vector on its own.

Residual connections and normalization around both parts keep a deep stack trainable. Attention has no sense of order by itself, so position information is supplied separately.

Most large language models are decoder-only transformers: each token attends only to the tokens before it, and the last layer's output becomes a probability for every possible next token. Text is generated one token at a time.

## Why it matters

Attention processes a whole sequence in parallel, which made training on very large datasets practical on GPUs. The vocabulary of fine-tuning follows from the architecture:

- [Context length](/glossary/context-length) is how many tokens the model can attend over at once.
- [LoRA target modules](/glossary/lora-target-modules) are the attention and MLP projections.
- A [mixture-of-experts](/glossary/mixture-of-experts) model splits the MLP block into experts.

The architecture comes from the 2017 paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762).

