What is a transformer?

Transformers are neural networks built from a stack of layers that each combine an attention block with a feed-forward block. The transformer is the architecture behind large language models.

How it works

Text is split into tokens, and each token becomes a vector. The vectors pass through a stack of layers that share one structure. Each layer has two parts:

  • Attention. Every token looks at the other tokens and takes in information from the ones that matter to it. This is how the model uses context.
  • A feed-forward block, also called an MLP (multilayer perceptron). It transforms each token's vector on its own.

Residual connections and normalization around both parts keep a deep stack trainable. Attention has no sense of order by itself, so position information is supplied separately.

Most large language models are decoder-only transformers: each token attends only to the tokens before it, and the last layer's output becomes a probability for every possible next token. Text is generated one token at a time.

Why it matters

Attention processes a whole sequence in parallel, which made training on very large datasets practical on GPUs. The vocabulary of fine-tuning follows from the architecture:

The architecture comes from the 2017 paper Attention Is All You Need.

Train your own models

Post-train any Hugging Face model with Halo. Up to 4× faster than TRL