How to work through this subject
An order that works, not a checklist. Nothing here is tracked — the steps sequence this subject’s own topics and add none.
- Work through a single neuron by hand: weights, bias, sum, activation.
- Stack a few of them into a layer and a couple of layers into a network, on paper.
- Trace one forward pass numerically on a tiny example.
- Compute the loss for that example and say what number you are trying to make smaller.
- Follow the error backwards through one layer using the chain rule from Stage 7.
- Describe what the optimizer does with the gradient once it has it.
- Finish by distinguishing batch, epoch and learning rate out loud, without conflating any two.
What to watch for
- Backpropagation is the chain rule applied repeatedly. If Stage 7 felt shaky, revisit gradients before continuing.
- Batch, epoch and learning rate are three different knobs. Confusing them makes training logs meaningless.
- Without a non-linear activation, stacking layers gains you nothing — the whole network collapses to one linear step.
- The loss must be differentiable because the entire training procedure is 'move in the direction that reduces it'.
- Weights are what the network learns; the architecture is what you chose. Keep the two straight when something goes wrong.
Curriculum topics
- perceptron intuition
- The simplest unit: weigh the inputs, add a bias, apply a threshold. Everything else builds on this.
- neuron
- One unit computing a weighted sum of its inputs followed by an activation.
- weight/bias
- The learned parameters — weights scale each input, the bias shifts the result.
- activation
- A non-linear function applied to a neuron's output. Without it, stacking layers collapses to a single linear one.
- layer
- A group of neurons applied together, transforming one representation into the next.
- network
- Layers stacked so the output of one becomes the input of the next.
- forward pass
- Running inputs through the network to produce a prediction.
- loss
- A single number measuring how wrong the prediction was — the thing training minimises.
- backpropagation
- Working the loss backwards through the network with the chain rule to get each parameter's gradient.
- optimizer
- The rule that turns gradients into parameter updates, such as SGD or Adam.
- batch
- The group of examples processed together before one parameter update.
- epoch
- One full pass over the entire training set.
- learning rate
- How far each optimizer step moves the weights. The single most consequential training hyperparameter.
Resources for this subject
- PRIMARYPyTorch — Learn the Basics PyTorch
The official beginner path from tensors to a full training loop. Type the training loop rather than copying it — reproducing it from memory is this stage's gate.Open
Need another explanation?2 more references
- REFERENCENeural Networks: Zero to Hero Andrej Karpathy
Builds autograd and then a transformer from scratch. The single best antidote to explaining transformers as 'magic', which Stage 10's gate forbids.Open - OPTIONALDive into Deep Learning d2l.ai
Interactive textbook with runnable PyTorch. Use for depth on a specific architecture, not as a parallel course.Open
A reviewer should be able to answer these
Answer from memory, out loud, before moving on. These are not tracked and do not affect the gate — they tell you whether you actually understand this subject.
- What does a forward pass compute, and what does a backward pass compute?
- What role does the activation function play, and what happens without one?
- What is the difference between a batch and an epoch?
- How does the optimizer use the gradient to change the weights?
- Why does the loss function have to be differentiable?
- What do the weight and bias of a single neuron each control?
- What is the learning rate scaling, and what would too large a value do to training?