Transformers are neural networks built from a stack of layers that each combine an attention block with a feed-forward block. The transformer is the architecture behind large language models.
How it works
Text is split into tokens, and each token becomes a vector. The vectors pass through a stack of layers that share one structure. Each layer has two parts:
- Attention. Every token looks at the other tokens and takes in information from the ones that matter to it. This is how the model uses context.
- A feed-forward block, also called an MLP (multilayer perceptron). It transforms each token's vector on its own.
Residual connections and normalization around both parts keep a deep stack trainable. Attention has no sense of order by itself, so position information is supplied separately.
Most large language models are decoder-only transformers: each token attends only to the tokens before it, and the last layer's output becomes a probability for every possible next token. Text is generated one token at a time.
Why it matters
Attention processes a whole sequence in parallel, which made training on very large datasets practical on GPUs. The vocabulary of fine-tuning follows from the architecture:
- Context length is how many tokens the model can attend over at once.
- LoRA target modules are the attention and MLP projections.
- A mixture-of-experts model splits the MLP block into experts.
The architecture comes from the 2017 paper Attention Is All You Need.
