How to work through this subject
An order that works, not a checklist. Nothing here is tracked — the steps sequence this subject’s own topics and add none.
- Tokenize several strings and look at the actual token boundaries, including for numbers and rare words.
- Read what the vocabulary is and why a fixed one forces subword splitting.
- Follow a token through its embedding and the addition of positional information.
- Work through attention as query, key and value on a three-token example, by hand.
- Assemble the pieces into one transformer block and say what each sublayer contributes.
- Read what causal masking prevents, and what encoder and decoder mean in this setting.
- Describe pretraining as next-token prediction, then follow inference and sampling out the other end.
- Change temperature and observe how the output distribution behaves, including at zero.
- Finish by reading what fine-tuning, LoRA and quantization each change, and what none of them change.
What to watch for
- Tokenization explains a surprising number of model failures. Look at actual token boundaries before blaming the model.
- Attention is a weighted lookup: query against keys, retrieve values. Everything else is plumbing around that.
- Positional information exists because attention alone is order-blind. Remove it and the sentence becomes a bag of words.
- The model predicts a distribution over the next token, every time. Sampling is a separate decision applied to that distribution.
- The context window is a hard limit, not a soft preference. Decide what gets dropped before the model decides for you.
- Fine-tuning changes weights; prompting changes inputs. LoRA changes a small number of weights; quantization changes their precision. None of them add knowledge the data never contained.
Curriculum topics
- language modeling
- Learning to predict text, which turns out to be enough to learn a great deal about language.
- tokens
- The chunks text is split into before a model sees it. A token is often part of a word, not a whole one.
- tokenization
- The process of converting text into tokens. Inspecting real token boundaries explains many model failures.
- vocabulary
- The fixed set of tokens a model knows, fixed at training time.
- embeddings
- Learned dense vectors representing discrete items, placing similar items near each other.
- positional information
- How the model is told the order of tokens, since attention alone has no sense of position.
- attention
- A weighted lookup: each token gathers information from other tokens according to how relevant they are.
- query/key/value intuition
- The query asks, keys advertise what each token offers, values carry the content retrieved.
- self-attention
- Attention where tokens attend to other tokens in the same sequence.
- transformer block
- The repeating unit: attention followed by a feed-forward layer, with normalisation and residual connections.
- encoder/decoder concepts
- Encoders read a whole input at once; decoders generate one token at a time. Models use one or both.
- causal language model
- A model that may only attend to earlier tokens, so it can generate text left to right.
- pretraining
- The large-scale initial training on generic text, before any task-specific adaptation.
- next-token prediction
- The pretraining objective: given what came before, predict what comes next.
- context window
- The maximum number of tokens the model can consider at once. Exceeding it means something must be dropped.
- inference
- Running a trained model to produce output, as opposed to training it.
- sampling
- How the next token is chosen from the predicted distribution.
- temperature intuition
- A knob on randomness in sampling. Zero is deterministic; higher values produce more varied output.
- fine-tuning
- Continuing training on your own data to adapt a pretrained model to a specific task.
- LoRA concept
- Fine-tuning by training a small number of added parameters instead of all of them, cutting cost dramatically.
- quantization concept
- Storing weights at lower numeric precision to reduce memory and speed up inference, trading a little quality.
Resources for this subject
- PRIMARYHugging Face LLM Course Hugging Face
The teaching path for this stage: tokenization, attention, transformer internals and the library that exposes them. Chapters 1 to 3 cover what the gate asks you to explain.Open
Need another explanation?4 more references
- REFERENCEMLCC — LLM module Google
A short, conceptual account of language models and how they are trained. Useful as a second explanation of tokens, context and fine-tuning when the primary course moves quickly.Open - REFERENCENeural Networks: Zero to Hero Andrej Karpathy
Builds autograd and then a transformer from scratch. The single best antidote to explaining transformers as 'magic', which Stage 10's gate forbids.Open - REFERENCEThe Illustrated Transformer Jay Alammar
The clearest visual explanation of query/key/value and multi-head attention. Read alongside the HF course chapter on attention.Open - OPTIONALAttention Is All You Need Vaswani et al., 2017
The original paper. The roadmap targets paper-reading capability, and this is the right first paper to read properly.Open
A reviewer should be able to answer these
Answer from memory, out loud, before moving on. These are not tracked and do not affect the gate — they tell you whether you actually understand this subject.
- What is a token, and why is it not the same as a word?
- What do query, key and value mean in attention, and what is being computed?
- What does 'causal' mean in a causal language model, and what does it prevent?
- What does the model actually predict at each step during pretraining?
- What does the context window limit, and what happens when you exceed it?
- What does temperature change about sampling, and when would you set it to zero?
- What is fine-tuning changing that prompting cannot?
- Why is positional information needed at all, given that attention sees every token?
- What does quantization trade away, and when is that trade acceptable?