LESSON 03 / THE CHAIN RULE

Backpropagation

Inspect the forward pass, loss, gradients and parameter update in a small neural network.

A network small enough to follow

Values flow forward →

Two inputs → two tanh neurons → one linear output. The bright connections mark the active calculation. Biases enter as additions at each neuron.

Prediction—

Target0.8000

Loss—

Change the example

Edits pause playback and recalculate the current step. The inputs and target are fixed during a weight update.

Input x₁0.60Input x₂-0.40Target y0.80Learning rate η0.20Inspect a parameterValue of w110.4000

STEP 1 / FORWARD PASS

Weighted sums

A weight scales an input; a bias shifts the sum. Each hidden neuron adds its own weighted inputs. Nothing has learned yet—we are calculating with the current weights.

\begin{aligned}z_1&=w_{11}x_1+w_{12}x_2+b_1\\&=(0.4000)(0.6000)+(-0.2000)(-0.4000)+(0.1000)=0.4200\\z_2&=w_{21}x_1+w_{22}x_2+b_2\\&=(-0.3000)(0.6000)+(0.5000)(-0.4000)+(-0.1000)=-0.4800\end{aligned}

Input 1 contribution to z₁+0.2400

Input 2 contribution to z₁+0.0800

Bias shifts z₁+0.1000

This lesson fits one example with squared-error loss. It isolates backpropagation from the games’ replay buffer, reward targets, and Huber loss. Matching one example does not mean the network will generalize. Values are rounded for display; calculations use full precision.

Why multiply derivatives—and when do we add them?

A derivative measures local sensitivity. If a small change in a weight changes an activation by a factor of two, and that activation changes the loss by a factor of three, the overall factor is six. The chain rule multiplies sensitivities along a path.

\frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}\frac{\partial h}{\partial w}

If a variable reaches the loss through multiple paths, add the contribution from every path. For example, x₁ feeds both hidden neurons in this network:

\frac{\partial L}{\partial x_1}=\frac{\partial L}{\partial z_1}w_{11}+\frac{\partial L}{\partial z_2}w_{21}

We do not update x₁: it is a supplied training input. This equation explains the general rule. Shared trainable parameters also accumulate gradients from all the paths that use them.

Backpropagation calculates these derivatives efficiently by reusing intermediate values. Gradient descent is the separate operation that uses them to change the parameters (Goodfellow et al., 2016, §6.5). A useful further explanation is Stanford’s notes on backward gradient flow (CS231n notes, n.d.).

Read the actual forward pass and backward pass

The lesson computes every displayed value with this code. Every gradient is calculated before any parameter is updated.

export const INITIAL_PARAMETERS = {
  w11: 0.4, w12: -0.2, w21: -0.3, w22: 0.5,
  b1: 0.1, b2: -0.1, v1: 0.7, v2: -0.4, c: 0.05,
};
export type Parameters = typeof INITIAL_PARAMETERS;
export type ParameterKey = keyof Parameters;
export const PARAMETER_KEYS = Object.keys(INITIAL_PARAMETERS) as ParameterKey[];
export const SYMBOLS: Record<ParameterKey, string> = {
  w11: 'w_{11}', w12: 'w_{12}', w21: 'w_{21}', w22: 'w_{22}',
  b1: 'b_1', b2: 'b_2', v1: 'v_1', v2: 'v_2', c: 'c',
};

export function forward(p: Parameters, x1: number, x2: number, target: number) {
  const z1 = p.w11 * x1 + p.w12 * x2 + p.b1;
  const z2 = p.w21 * x1 + p.w22 * x2 + p.b2;
  const h1 = Math.tanh(z1);
  const h2 = Math.tanh(z2);
  const prediction = p.v1 * h1 + p.v2 * h2 + p.c;
  const error = prediction - target;
  return { z1, z2, h1, h2, prediction, error, loss: 0.5 * error ** 2 };
}

export function backpropagate(p: Parameters, x1: number, x2: number, target: number) {
  const f = forward(p, x1, x2, target);
  // Chain rule: upstream gradient × local derivative.
  const dh1 = f.error * p.v1;
  const dh2 = f.error * p.v2;
  const dz1 = dh1 * (1 - f.h1 ** 2);
  const dz2 = dh2 * (1 - f.h2 ** 2);
  const gradients: Parameters = {
    v1: f.error * f.h1, v2: f.error * f.h2, c: f.error,
    w11: dz1 * x1, w12: dz1 * x2, b1: dz1,
    w21: dz2 * x1, w22: dz2 * x2, b2: dz2,
  };
  return { ...f, dh1, dh2, dz1, dz2, gradients };
}

export function applyGradients(p: Parameters, gradients: Parameters, rate: number): Parameters {
  // All gradients come from the same forward pass, before any weight changes.
  return Object.fromEntries(PARAMETER_KEYS.map(key => [key, p[key] - rate * gradients[key]])) as Parameters;
}

History of the method

Backpropagation has no single inventor. The chain rule is seventeenth-century calculus. Applying it backwards through a calculation, reusing intermediate values, turned up independently in several fields—in the 1960s and 1970s mostly in control problems (Goodfellow et al., 2016, §6.6).

Seppo Linnainmaa described reverse accumulation of derivatives in his 1970 master’s thesis in Helsinki, published in 1976 (Linnainmaa, 1976). He used it to estimate how rounding errors propagate through arithmetic, not to train networks. Andreas Griewank’s history of the idea also finds it in chemical-engineering process models, circuit optimization, compiler work and weather-data assimilation—and he rediscovered it himself in 1987 (Griewank, 2012).

Paul Werbos proposed applying these techniques to neural networks. After further independent rediscoveries, David Rumelhart, Geoffrey Hinton and Ronald Williams showed in 1986 that hidden units trained this way come to represent important features of the task (Rumelhart et al., 1986). That work, and the Parallel Distributed Processing book chapter that followed, did much to popularize the method (Goodfellow et al., 2016, §6.6).

Outside neural networks the general technique is called reverse-mode automatic differentiation, and the two communities have at times independently discovered each other’s results (Baydin et al., 2018). This lesson writes out its nine gradients by hand, which is the version worth seeing once. Gradient descent has a separate and older history, summarized in the network depth lesson.

References

Atılım Güneş Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul & Jeffrey Mark Siskind (2018). Automatic Differentiation in Machine Learning: a Survey. Journal of Machine Learning Research 18(153), 1–43.Open access · Read: Journal record and abstract · Checked 2026-09-13
Stanford CS231n course staff (n.d.). Backpropagation, Intuitions. Stanford CS231n: Deep Learning for Computer Vision, course notes.Open access · Read: Existing lesson link; notes on gates, local gradients and upstream gradients · Checked 2026-09-13
Ian Goodfellow, Yoshua Bengio & Aaron Courville (2016). Deep Learning. MIT Press, 1st ed.Open access · Read: Authors’ online edition: chapter 6 (deep feedforward networks), including §6.3 hidden units, §6.4.1 universal approximation and depth, §6.5 back-propagation and §6.6 historical notes; chapter 8 §8.2.5 long-term dependencies and §8.4 parameter initialization; chapter 9 §9.2 motivation for convolution · Checked 2026-09-13
Andreas Griewank (2012). Who invented the reverse mode of differentiation? Documenta Mathematica, Extra Volume ISMP: Optimization Stories, 389–400.Open access · open copy · Read: Full chapter read; a historical review, cited for chronology · Checked 2026-09-13
Seppo Linnainmaa (1976). Taylor expansion of the accumulated rounding error. BIT Numerical Mathematics 16(2), 146–160.Publisher record; full text may require access · Read: Crossref bibliographic record; content as described by Griewank (2012) · Checked 2026-09-13
David E. Rumelhart, Geoffrey E. Hinton & Ronald J. Williams (1986). Learning representations by back-propagating errors. Nature 323(6088), 533–536.Publisher record; full text may require access · Read: Publisher abstract and bibliographic record · Checked 2026-09-13