Stage 4: Embeddings Give IDs Meaning
Each ID indexes into a giant lookup table, pulling out a learned vector of d numbers. These vectors are where meaning lives. Words used in similar contexts end up with similar vectors, and this table is trained along with everything else.
Notice the shape change: a list of n integers becomes an n × d matrix. Every stage from here on preserves that shape — the transformer transforms these vectors repeatedly without ever changing the grid size.
Stage 5: Position Must Be Injected
Self-attention has no inherent sense of order — it sees a set of vectors, so "dog bites man" and "man bites dog" would be identical. Position is therefore added directly into the vectors, classically with sinusoids of geometrically increasing wavelength:
PE(pos, 2i) = sin(pos / 100002i/d) PE(pos, 2i+1) = cos(pos / 100002i/d)
The values in this lab are the genuine formula. Look along a row and you will see fast oscillation in the early dimensions and near-constant values in the later ones — that mix gives every position a unique fingerprint the model can decode.
Stage 6: Attention Mixes the Tokens
Now every token looks at every other token. Each position produces a query and a key; their dot product, scaled by √d, scores how relevant one token is to another, and a softmax over each row turns those scores into weights that sum to 1:
Attention(Q, K, V) = softmax(QKᵀ / √d) V
The heatmap in this stage is a real computation from the vectors above it. Each row shows where one token sent its attention. This is the step where a token stops being an isolated word and becomes a contextual representation.
Stages 7–8: Depth, Then a Single Prediction
One attention block is repeated dozens of times, each with a feed-forward network, residual connections and normalisation. Early layers capture syntax; deeper layers capture semantics and task structure.
At the very end only the last position's vector is used for the next token. It is multiplied by the vocabulary matrix to produce one logit per possible token — and we are back to a distribution over words. The chosen token is appended to the input and the entire pipeline runs again, from stage 1, for every single word generated.
The full path, once
Text goes in, text comes out, and five stages sit between them. Knowing which stage a problem lives in resolves most confusion about model behaviour.
- Tokenise — the string becomes integer ids via a subword vocabulary.
- Embed — each id indexes a learned matrix, giving a vector per token.
- Add position — attention is order-blind, so positional information is injected.
- N transformer blocks — attention moves information between positions, feed-forward layers process each position.
- Project and sample — one logit per vocabulary entry, softmax, pick a token.
"The capital of France is" → [464, 3139, 286, 4881, 318] → 5×4096 → blocks → " Paris"
Then the chosen token is appended to the input and the whole thing runs again. That loop is generation.
Inside one block
Every block is the same two sublayers, each wrapped in a residual connection and a normalisation:
x → LayerNorm → Attention → (+x) → LayerNorm → FFN → (+x)
Attention lets each token gather information from others. Each token produces a query, compares it against every token's key, and takes a weighted mixture of their values. In a decoder, future positions are masked out so generation is possible.
The feed-forward network processes each position independently, expanding to roughly four times the model dimension and back. This is where most of the parameters live — typically two thirds of the model — which is worth knowing because attention gets all the attention.
Residual connections give gradients a clean path back through dozens of blocks. Without them, deep stacks do not train.
The tensor shape through all of this is constant: (batch, sequence, model_dim). That constancy is what makes blocks stackable, and why the same architecture works at 12 layers and at 80.
Two phases, very different costs
Generation splits into two workloads, and the split explains most latency and pricing behaviour.
Prefill processes the prompt. Every token is known, so all positions are computed in parallel in one pass. Compute-bound, efficient, cheap per token.
Decode produces the output. Each token depends on the last, so it is strictly sequential, and every step must read the entire model's weights plus the KV cache from memory. Bandwidth-bound, and expensive per token.
| | Prefill | Decode |
|---|
| Parallel | Yes | No |
| Bottleneck | Compute | Memory bandwidth |
| Cost per token | Low | High |
| User-visible metric | Time to first token | Tokens per second |
That is why input tokens are priced below output tokens, why a long prompt delays the first word rather than slowing the stream, and why generating 500 tokens takes longer than reading 5,000.
The KV cache is what keeps decode linear: previous tokens' keys and values are stored rather than recomputed. It costs memory — several gigabytes for a long context — and it is the dominant memory consumer in serving.
The shapes, end to end
The stages are described above. Following the SHAPES through them is what makes the pipeline concrete -- where the parameters actually are, why prefill and decode cost such different amounts, and what the model does not have that you might assume it does.
Try it yourself
- Walk from stage 1 to 8 with Next and keep an eye on the shape tag. Watch n integers become n × d and stay there until the final projection.
- At stage 5, compare the two rows of "the". Identical embeddings, different positional vectors — which is the only reason the model can tell the two occurrences apart.
- At stage 6, read the attention rows. Repeated words attend to each other strongly, because attention is driven by vector similarity.
- Change the sentence to "man bites dog", then "dog bites man". Same tokens, different attention pattern — order genuinely changes the computation once positions are added.
- Raise the model dimension. Every downstream matrix widens, which is the crudest possible illustration of why parameter counts explode with d.
What to remember
Text in, integers, vectors, mixed vectors, logits, text out. The embeddings and attention weights in this lab are deterministic stand-ins for learned parameters, but the operations — sinusoidal positions, scaled dot-product attention, softmax — are the real ones. Understanding this pipeline makes context windows, tokenizer quirks and attention costs stop feeling arbitrary.
What the model does and does not have
No memory between calls. A chat interface re-sends the whole conversation every time. The model reads the history; it does not remember it. That is why long conversations get slower and more expensive, and why "remember what I said earlier" only works within the window.
No access to anything outside the prompt. No files, no internet, no database — unless a tool call provides it. Retrieval exists precisely to put external information into the context.
No view of characters. The model sees tokens. "Strawberry" may be two or three tokens, none of which is a letter, which is why letter-counting and spelling questions are unreliable.
No calculator. Numbers tokenise inconsistently, so arithmetic is pattern-matched rather than computed. Give it a tool.
No calibrated confidence. It produces a distribution over tokens, not a belief about truth. A confident tone is a property of the text distribution.
Those five limitations account for a large share of the surprising failures people report, and all five are structural rather than fixable by prompting.
Where the parameters are
For a 7B-parameter model with 32 layers and a model dimension of 4096:
| Component | Approximate share |
|---|
| Feed-forward layers | ~65% |
| Attention projections | ~30% |
| Embedding and output | ~5% |
| LayerNorm | Negligible |
The embedding and output matrices are frequently tied — the same weights used in both directions — which saves a substantial number of parameters (a 128,000-token vocabulary times 4,096 dimensions is 524 million) and typically helps slightly.
Memory at inference is roughly the parameter count times the bytes per parameter, plus the KV cache. That is the arithmetic behind every quantisation decision: 7 billion parameters is 14GB at 16-bit and 3.5GB at 4-bit.
Practical implications
Count tokens with the real tokeniser when budgeting a prompt. Word-count estimates are unreliable, especially for code and non-Latin scripts.
Put important material at the ends of a long prompt. Models attend less reliably to the middle of a long context.
Shorter prompts are cheaper and often better. Irrelevant context degrades answers as well as costing money.
Temperature 0 for anything that must be determinate — extraction, classification, code. Higher only where variety is wanted.
Stream the output if a user is waiting. Time to first token is what feels like latency, and it is dominated by prefill.
Questions people ask
Does the model understand what it reads? It models statistical structure in text extremely well. Whether that constitutes understanding is a genuine open question rather than a settled one.
Why is the same question answered differently each time? Sampling, unless temperature is 0 — and even then, batching and hardware non-determinism produce small variations.
Can it read a file I mention? Only if something puts the contents into the context. Naming a path does nothing.
Why does it fail at simple arithmetic? Tokenisation fragments numbers, and the model pattern-matches rather than calculating. Use a tool.
What is the context window shared between? Prompt, system instructions, retrieved documents, conversation history and generated output — one budget for all of them.
Why are output tokens more expensive? Prefill is parallel; decode is sequential and bandwidth-bound.
Recap in one screen
- Tokenise, embed, add position, pass through N identical blocks, project to the vocabulary, sample — then repeat.
- Attention moves information between positions; the feed-forward layers hold most of the parameters.
- Prefill is parallel and cheap; decode is sequential and expensive, which explains the pricing and latency asymmetry.
- The KV cache makes decode linear at a large memory cost.
- No memory between calls, no view of characters, no calculator, no calibrated confidence.