Transformer Model
The neural network architecture, based on self-attention, that underpins virtually all modern large language models and many vision and audio models.
What Is Transformer Model?
The Transformer is a neural network architecture introduced in the 2017 paper "Attention Is All You Need." It replaced earlier recurrent (RNN) and convolutional approaches to sequence modelling with a mechanism called self-attention, which lets every token in a sequence directly attend to every other token. This removed the sequential bottleneck of RNNs, enabling massive parallel training on modern GPUs and unlocking the scale that produced today's large language models.
At the heart of the Transformer is multi-head self-attention: for each token, the model computes how much it should focus on every other token, allowing it to capture long-range dependencies and context. Stacked attention layers, combined with feed-forward networks, positional encodings, and residual connections, form the deep network. Variants include decoder-only models (GPT, Llama, Claude) for text generation, encoder-only models (BERT) for understanding, and encoder-decoder models (T5) for translation-style tasks.
Why It Matters
The Transformer is the foundational architecture of the modern AI era. Nearly every breakthrough since 2018 — GPT, BERT, Claude, Gemini, Stable Diffusion's text encoder, and multimodal models — is built on it. Understanding the Transformer explains why scaling data and parameters produces emergent capabilities, why context windows are computationally expensive, and why GPUs and attention optimisations matter so much for AI performance.