TL;DR
- Backpropagation applies the chain rule in reverse, accumulating every downstream contribution so hidden weights can learn without hidden-unit targets.
- The 1986 paper demonstrated learned internal representations; it was not the first invention of reverse differentiation or neural-network backpropagation.
- Compute the full gradient from one unchanged parameter state, then let the optimizer apply the update.
- Finite differences check the supplied network's derivatives; the original symmetry and family-tree experiments provide separate evidence of representation learning.
- Correct derivatives do not resolve saturation, difficult optimization, or the broader challenges of scaling a network.
How hidden weights receive a learning signal
An output unit can compare its prediction with a target. A hidden unit has no supplied target telling it what representation to produce. The central question is how the output error should change an internal weight when that weight affects the loss through several later computations. Backpropagation answers it by applying the chain rule in reverse and reusing derivatives of shared intermediate results.
Rumelhart, Hinton, and Williams's 1986 Nature paper makes that answer tangible through multilayer networks that learn internal representations. It is a landmark demonstration and popularization, not the first invention of reverse differentiation or neural-network backpropagation. The paper itself notes independent rediscoveries by David Parker and Yann LeCun. Its historical significance should not erase the work preceding and surrounding it. Original paper, pages 533–535
This review uses the author-hosted scan of Learning representations by back-propagating errors, Nature 323, pages 533–536. The scan includes neighboring articles at its boundaries; the excerpts below select only this paper. The interactive network is authored with supplied parameters, not a trained reproduction. Every displayed value is a deterministic calculation, and the canonical reference checks all twelve analytic parameter derivatives against finite differences.
Start with the forward computation
The paper first considers layered feed-forward networks. Each non-input unit adds weighted outputs from earlier units and applies a sigmoid. A bias can be represented as an extra incoming connection from a unit fixed at one. These operations determine the network output before any derivative is calculated. Paper, Eqs. 1–2
Our notation uses z for a unit's weighted preactivation and a for its activation. The original paper calls the weighted preactivation x, so its x should not be mistaken for the input vector in the code. A sigmoid maps z to a value between zero and one and has derivative a(1−a). Caching the activation therefore supplies the local derivative without recomputing an exponential during the reverse pass.
Store a forward trace before differentiating it
A supplied 2→2→2 network uses sigmoid hidden and output units. Its parameters are authored, not learned.
Inputs supplied to the network. Loss = 0.221711.
The small network has two input values, two sigmoid hidden units, and two sigmoid output units. The first weight matrix is 2×2, followed by two hidden biases; the second is 2×2, followed by two output biases. Flattening those arrays gives twelve supplied parameters. Matrix rows index destination units and columns index source units. Stating that convention prevents a transposed implementation from appearing correct merely because both matrices happen to be square.
The loss is half the sum of squared differences between the two outputs and targets for one input case. The half cancels the factor of two when differentiating the square. This is not a mean over output coordinates, so there is no extra division by two in the gradient. Summing losses across examples and averaging them are both legitimate objectives, but their gradients differ by the chosen normalization. A derivative check must use the exact same objective as the forward program.
The output layer provides the initial derivative
For squared error, the derivative with respect to an output activation is its prediction minus its target. To obtain the derivative with respect to the output's preactivation, multiply by the sigmoid derivative. This gives an output delta equal to (a−target)·a·(1−a). “Delta” is simply a convenient name for a particular derivative; it is not the raw output error alone. Paper, Eqs. 4–5
This starting expression depends on both the chosen loss and activation. Replacing sigmoid-plus-squared-error with a softmax cross-entropy objective changes the local derivative. Memorizing one delta formula without its forward assumptions is a common way to introduce a mathematically consistent-looking but incorrect backward pass.
For a weighted sum z = Σw·a + b, a particular weight has local derivative equal to its incoming activation. Its loss derivative is therefore delta times that activation. The bias derivative is delta, since the derivative of adding b is one. These local rules explain both parameter gradients; they do not require assigning desired targets to hidden units. Paper, Eq. 6
Hidden units receive a sum over every consumer
A hidden activation can affect more than one later unit. The chain rule adds all of those routes. For each downstream connection, multiply the downstream preactivation derivative by the connection's weight; sum the results to obtain the derivative with respect to the hidden activation. Only then multiply by the hidden sigmoid derivative to cross that local nonlinearity. Paper, Eq. 7
Add downstream paths before crossing a hidden sigmoid
A hidden unit feeds both outputs. Each output supplies a weighted derivative contribution, which must be summed.
Path from output 1
-0.067490
Path from output 2
-0.048701
Sum ∂L/∂h
-0.116191
After multiplying by h(1−h): hidden preactivation derivative = -0.027625.
The two-output network makes the sum unavoidable. Each hidden unit affects both predictions, so each output sends a contribution. Opposite-signed paths can partly cancel. That cancellation does not mean that either path can be ignored, and a small sum does not imply that each underlying influence is small. The reverse pass preserves these contributions according to the actual graph.
This is also where the algorithm gains its efficiency. Once the derivative of a shared intermediate is accumulated, it can be reused for all parameters upstream of that intermediate. Repeating the entire downstream derivation independently for every weight would duplicate work. Backpropagation organizes the chain rule according to the dependency graph; it is not a numerical method that perturbs every parameter during normal training.
The graph must be traversed in a valid reverse order. A unit's derivative is ready after the needed downstream contributions have been computed. For a simple layered network, that means output layer first, then successive hidden layers. More general differentiable graphs require the same dependency discipline, with accumulation at nodes that have several consumers. The arithmetic illustrates reverse-mode differentiation without claiming that every operation in a modern model is a sigmoid or a matrix multiplication.
Compute derivatives before changing the weights
All paths in a backward pass refer to the forward computation whose loss is being differentiated. Updating an output weight and then using its new value to propagate a hidden derivative mixes two parameter states. The result no longer differentiates the original scalar loss. The supplied reference computes the complete gradient before taking a separate gradient-descent step.
The paper describes both changing weights after each input-output case and accumulating derivatives across the training cases before changing weights; its reported research uses the latter arrangement. It also presents a momentum update after the simplest gradient-descent rule. These are choices about using the derivatives. They are distinct from the reverse computation that produces them. Paper, Eqs. 8–9 and accompanying discussion
That separation remains useful when reading modern training code. Autodiff can compute gradients, while SGD, momentum, or Adam decides how to turn gradients and optimizer history into parameter updates. Backpropagation is not itself a promise that one particular update schedule reaches a global optimum. A correct gradient can point along a difficult, poorly conditioned, or locally unhelpful optimization landscape.
Check a derivative without trusting its derivation
Central finite differences evaluate the loss once with a parameter increased by h and once with it decreased by h, then divide their difference by 2h. This produces an independent numerical approximation to that parameter's derivative. The test reruns the full forward computation on both sides; it does not reuse an unperturbed hidden state after changing an upstream weight.
Ask the loss whether the derivative is right
Central differences perturb one parameter and rerun the full forward computation. Compare that independent numerical slope with backpropagation.
Analytic derivative
-2.7625427e-2
Central difference
-2.7625427e-2
Absolute error
< 1e−9
With all 12 parameters updated by −0.1∇L: loss 0.221711 → 0.216659.
The reference checks all twelve parameters, including both bias vectors, at multiple input and target settings. A zero input additionally requires zero derivatives for its corresponding first-layer weights. Setting the targets equal to the model's own outputs makes the squared-error gradient zero. These are different invariants, so they catch mistakes that a single successful finite-difference example might miss.
The perturbation size is a tradeoff. Large h measures the slope across a visibly curved interval rather than the local derivative. Extremely small h causes subtraction of nearly equal floating-point losses and amplifies numerical error. A tiny nonzero discrepancy is therefore expected; the check uses a tolerance rather than requiring exact bit equality. At a nondifferentiable operation, such as a ReLU exactly at zero, the issue would be different. This sigmoid network deliberately uses smooth operations so the numerical comparison tests ordinary derivatives.
Original equations beside canonical Python
1. Cache weighted sums and sigmoid outputs
Original paper
Equations 1–2 define weighted sums and logistic units; Equation 3 supplies squared output error.
Math and tensor shapes
Our authored network has two inputs, two hidden units and two outputs. The paper calls a unit’s total input x; this reference uses z to distinguish preactivations from network inputs. Parameter rows index destination then source. Biases are explicit, equivalent to the paper’s extra fixed-one input.
Python implementation
def sigmoid(z):
"""Numerically stable logistic activation, including saturated boundaries."""
e = math.exp(-z if z >= 0 else z)
return 1 / (1 + e) if z >= 0 else e / (1 + e)
def forward(p, x=(1, -.5), target=(1, 0)):
"""Two hidden and two output units; half sum-of-squares over outputs."""
if len(p) != 12 or len(x) != 2 or len(target) != 2:
raise ValueError("Expected 12 parameters, 2 inputs and 2 targets")
if not all(math.isfinite(v) for v in [*p, *x, *target]):
raise ValueError("Finite inputs required")
z1 = [p[4+i] + p[2*i]*x[0] + p[2*i+1]*x[1] for i in range(2)]
hidden = [sigmoid(z) for z in z1]
z2 = [p[10+i] + p[6+2*i]*hidden[0] + p[7+2*i]*hidden[1] for i in range(2)]
output = [sigmoid(z) for z in z2]
loss = .5 * sum((y-t)**2 for y, t in zip(output, target))
return dict(z1=z1, hidden=hidden, z2=z2, output=output, loss=loss)2. Start the reverse pass at the loss
Original paper
Equations 4–6 combine the squared-error derivative with the sigmoid derivative and the local weighted sum.
Math and tensor shapes
The factor a(1−a) belongs to the sigmoid-plus-squared-error combination used here. Changing to a different output activation or loss changes this starting derivative. The one-case loss is half the sum over outputs, not a mean over outputs or examples.
Python implementation
def output_deltas(output, target):
"""dL/dz = (y-target)*y*(1-y), for sigmoid plus half squared error."""
return [(y-t)*y*(1-y) for y, t in zip(output, target)]3. Sum all consumers, then differentiate each local operation
Original paper
Equation 7 accumulates all downstream contributions. Equation 8 uses the resulting derivatives for gradient descent.
Math and tensor shapes
The reverse helper returns all twelve parameter derivatives in the documented flattened order. Both outputs contribute to each hidden signal. It uses the same forward weights throughout differentiation, then a separate step updates parameters. Finite differences independently recompute both perturbed forward passes.
Python implementation
def gradient(p, x=(1, -.5), target=(1, 0)):
"""Sum EVERY downstream contribution before applying a hidden derivative."""
f = forward(p, x, target)
delta2 = output_deltas(f['output'], target)
signal = [sum(d*p[6+2*i+j] for i, d in enumerate(delta2)) for j in range(2)]
delta1 = [h*(1-h)*s for h, s in zip(f['hidden'], signal)]
values = ([d*v for d in delta1 for v in x] + delta1
+ [d*h for d in delta2 for h in f['hidden']] + delta2)
return dict(values=values, outputDelta=delta2,
hiddenSignal=signal, hiddenDelta=delta1)Original excerpts: David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams; Nature 323, 533–536 (1986), author-hosted scan. Selections exclude neighboring articles in the scan. Original colors and PDF/crop hashes are retained.
Run backprop.py with Python 3.10 or later. The README documents parameter order, objective normalization, and omissions; requirements contains no external packages. The browser model and Python program are checked for numerical agreement. This establishes the small reference's forward and reverse arithmetic, not the learned representations or training counts of the original paper.
What the original experiments show
Figure 1 presents a network that detects mirror symmetry in six-bit input patterns using two hidden units and one output. Its caption reports training across all 64 input vectors and 1,425 sweeps. The learned hidden units combine opposite sides of the input with opposing signs, forming an internal representation useful for the symmetry decision. This is an example of learning features from an output objective, rather than supplying hidden-unit targets. The reported sweep count describes that experiment, not a general training-time bound or a hardware measurement. Paper, Figure 1
The family-tree task is a second, different demonstration. Two isomorphic families provide triples of a person, a relation, and another person. The network must predict the related person from the first two fields. Figure 3 uses 24 person-identity inputs and 12 relation inputs, passes through intermediate layers, and predicts among 24 people. The text reports training on 100 of 104 possible triples and predicting the four remaining cases correctly in the described run. Paper, Figures 2–3 and family-tree discussion
Figure 4 visualizes learned weights for internal person representations. The authors interpret patterns that distinguish family membership, generation, and other relational regularities rather than simply copying one-hot identities through the network. This supports their central claim that error-driven learning can discover useful distributed internal codes. It is evidence from a small structured task and a particular trained network; it is not a modern large-scale evaluation, an estimate of language understanding, or proof that every hidden feature has a uniquely recoverable semantic meaning. Paper, Figure 4
What remains difficult after the chain rule is correct
Sigmoid derivatives approach zero when activations saturate near zero or one. Multiplying many such local derivatives can attenuate signals to early layers, even when every formula is implemented correctly. Initialization, depth, architecture, normalization, and the loss all influence whether the gradients that arrive are useful. The paper's examples establish representation learning in its tested networks without solving all of those later scaling problems.
The authors also explicitly distinguish the learning procedure from a plausible biological mechanism in the paper's closing discussion. A mathematical route to useful internal representations does not, by itself, establish that brains compute the same reverse signals. Keeping that limitation visible is part of reading the original contribution accurately. Paper, closing paragraph
For practical work, I would retain a small gradient check alongside the first implementation of any custom differentiable mechanism. I would verify shape conventions, loss reduction, accumulation over multiple paths, and update timing before attributing unstable training to optimization. The gradient-flow concept follows how derivatives behave through depth; Xavier initialization addresses the scale of forward and backward signals at the start of training. Backpropagation supplies the derivative machinery that makes those questions measurable.
