The transformer is the neural network architecture introduced in the 2017 paper "Attention Is All You Need." It replaced earlier recurrent architectures and became the universal foundation of modern language AI — GPT, Claude, Gemini, Llama, and virtually every state-of-the-art model today.
Before transformers, language models read text sequentially — word by word, left to right. This made it hard to connect context across long distances. Transformers eliminated this constraint: every position in the input can attend to every other position simultaneously, regardless of how far apart they are.
Self-Attention — lets every token see every other token at once and determine which context is most relevant
Positional Encoding — adds word order information so the model knows where each token sits in the sequence
Stacked Layers — many transformer blocks stacked on top of each other, each refining the representation from the one below
Self-attention lets the model determine, for each token, how much to attend to every other token when building that token's representation.
When processing the word "bank" in "I deposited money at the bank," self-attention helps the model figure out that "deposited" and "money" are highly relevant context — more than "the" or "I" — for understanding which sense of "bank" is intended. The model learns these relationships during training.
The mechanism uses three learned vectors for each token:
Query — "What am I looking for?" — represents what this token needs from the context
Key — "What do I represent?" — represents what this token offers to other tokens looking for context
Value — "What information do I carry?" — the actual content that gets aggregated into the output
Attention scores are computed by comparing each token's Query against all other tokens' Keys. High score = attend a lot. Low score = mostly ignore. The output is a weighted average of Value vectors — each token's final representation is enriched by the context it attended to.
In practice, transformers use multi-head attention — multiple attention mechanisms running in parallel, each attending to the input from a different perspective.
One head might track grammatical relationships (subject → verb agreement).
Another might track semantic similarity (synonyms and related concepts).
Another might track co-reference (which pronouns refer to which nouns).
Language is simultaneously structured by grammar, semantics, and discourse — one attention pattern cannot capture all of these at once.
Multiple heads let the model build richer, multidimensional representations by attending to the input through many lenses simultaneously.
Attention computation has no inherent sense of order. Without position information, "The dog bit the man" and "The man bit the dog" look identical to the model — same tokens, same attention patterns. Positional encoding adds each token's position in the sequence to its representation before the first attention layer, giving the model order awareness.
A full transformer model is built by stacking many transformer blocks. Each block contains: self-attention → feed-forward network → layer normalization.
Lower layers tend to capture low-level patterns: grammar, punctuation, common phrases. Higher layers tend to capture abstract semantics: meaning, reasoning steps, factual relationships. Think of each layer as an editorial pass — reading the draft from the layer below and producing a more refined, higher-level understanding. GPT-3 has 96 layers; modern frontier models often have hundreds.
Reads tokens left to right with causal masking — each token can only attend to previous tokens, never future ones.
Optimized for text generation. The standard architecture for chat assistants and code generation.
Reads the full input bidirectionally — each token attends to all others in both directions simultaneously.
Optimized for understanding and classification. Not suited for generation tasks.
Ask the AI if you need help understanding the transformer architecture, how self-attention works, what Queries, Keys, and Values are, why positional encoding is needed, or the difference between encoder-only and decoder-only models.