How an LLM works

The shape of a transformer

Before we dig into how each part works, here's the overall shape: input on the left, transformer in the middle, one output token on the right. Inside the transformer, the same sub-block repeats - ATTENTION followed by MLP, dozens of times. Step through this orientation, then we'll drill into each part below.

The aha: there's no special "next-word predictor" sub-network. The model produces an answer by stacking simple operations dozens of times. The intelligence is in the weights, not the structure. Now let's look at each part.

What's a token?

The model doesn't see characters or words - it sees tokens. A token is roughly "a chunk of text the tokenizer decided to keep together." Common words are usually one token. Long or unusual words split into several. Whitespace is part of the token, which is why " cat" and "cat" are different tokens to the model.

Type below and see how the prompt would be split. (This is a simplified mock - real tokenizers use BPE/SentencePiece and have learned vocabularies of ~50k-200k tokens.)

Why this matters: token count drives cost and context limits. "Antidisestablishmentarianism" is one English word but ~7 tokens. A model with a 128k context window means 128k of tokens, not characters or words.

How does it know what "she" means?

Inside each transformer block, attention answers a single question for every token: "which other tokens should I look at to understand this one?" In the sentence "Sarah told Jane that she was leaving", your brain decides who she refers to by glancing back at the prior names. Attention does the same thing, with numbers.

A model has many heads, each computing a different relationship in parallel. Hover any token below and switch heads to see how the same word can attend to different things depending on what relationship is being captured.

Why this matters: attention is what gives transformers their long-range reasoning. The model doesn't need to "remember" - every token can directly look at any other token in one step. That's a wildly different architecture from RNNs, which had to pass information sequentially.

Why is generation slower than prefill?

Now that we've seen the architecture and one of its key operations, let's watch a full forward pass run twice - once for the prompt (prefill, all at once) and once per output token (generate, sequential). The KV cache is what bridges them: prefill fills it; generation reads from it and grows it by one cell per token.

Why this matters: when a chatbot "feels fast" up front but then types out the answer, you're seeing this asymmetry. Prefill is one big GPU-friendly matmul. Generation is hundreds of small matmuls in a sequence. Most of LLM inference engineering - speculative decoding, continuous batching, paged attention - exists to soften that gap.

What you just saw

The whole loop in pseudocode

Two phases. Prefill processes the prompt all at once. Generation produces output one token at a time, reusing the cached K/V from prefill so each new token is cheap.

# 1. PREFILL - runs once, all tokens at once
tokens     = tokenize(prompt)              # "The Beatles were a famous" → 5 tokens
embeddings = embed(tokens)                 # 5 × hidden_dim

x = embeddings
for layer in transformer_layers:           # dozens of layers
    x = x + attention(x)                   # residual stream
    x = x + mlp(x)

kv_cache = collect_kv(transformer_layers)  # one K/V tensor per layer per token

# 2. GENERATE - runs N times, one token at a time
output = []
while not done:
    next_token = predict_next(kv_cache)    # softmax over vocab
    output.append(next_token)              # → "band"
    kv_cache.append(K, V for next_token)   # cache grows by 1
    done = is_stop_token(next_token) or len(output) > MAX

return detokenize(output)

The closing aha: attention is parallel - every token's attention against every other token is one big matmul. MLP is independent per-token - also a big matmul. That's why GPUs love transformers: the whole architecture is just "do many matmuls". Every optimization in the LLM stack - KV caching, speculative decoding, batching - is squeezing more performance out of those matmuls.