Transformer Architecture Explained: The Foundation of Every Modern LLM

The Paper That Changed Everything

In 2017, Google researchers published "Attention Is All You Need" — a paper that discarded recurrent and convolutional networks entirely in favour of a novel architecture built purely on self-attention. The transformer they described became the foundation for every major language model in existence today.

Self-Attention: The Core Idea

Self-attention allows every token in a sequence to "look at" every other token and decide how much to attend to it. This is what lets "bank" in "I went to the river bank" attend strongly to "river" rather than "money".

The mechanism works via three learned matrices:

• Query (Q): What am I looking for?
• Key (K): What do I contain?
• Value (V): What should I output if attended to?

Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V

Multi-Head Attention

A single attention head captures one type of relationship. Multi-head attention runs several attention operations in parallel, each learning different relationship types — syntax, semantics, coreference — then concatenates the results.

Positional Encoding

Unlike RNNs, transformers have no built-in notion of order. Positional encodings (sinusoidal or learned) are added to token embeddings to inject position information. Modern models use RoPE (Rotary Position Embedding) which generalises better to long sequences.

The Full Architecture

A transformer consists of stacked encoder and/or decoder blocks. Decoder-only models (GPT, LLaMA, Claude) are most common for language generation. Each block contains:

1. Multi-head self-attention
2. Feed-forward network (two linear layers with GELU activation)
3. Layer normalisation (pre-norm in modern models)
4. Residual connections

Scaling transformers — more layers, larger embeddings, more data — consistently improves performance. This "scaling law" discovery is what kicked off the LLM race.