TL;DR
- Dropout randomly masks activations during training, creating many subnetworks that share one set of weights. The original paper's p means probability of keeping a unit, rather than dropping it.
- Original dropout scales outgoing weights at inference; inverted dropout scales surviving activations during training. Both track expected linear contributions, but use different numerical training conventions.
- Weight scaling is not an exact arithmetic average of nonlinear networks. The small mask examples calculate expectations and local gradients; they do not measure generalization.
- The published MNIST results depend on architecture, activation, max-norm constraints and training procedure. Before comparing validation behavior, check rate semantics, mask sharing, scaling and the inference path.
Why train with missing activations
A hidden feature can become useful only when a particular group of other features is present. On the training set, that cooperation may look excellent; on new examples, it may encode an accidental pattern. Dropout changes the training problem by repeatedly removing a random subset of activations. A feature must help under many possible collections of companions, rather than depend on the same entire set every time.
The 2014 JMLR paper develops this idea as both a regularizer and a practical approximation to combining many shared-weight networks. Its economy comes from reusing one parameter set: each mask selects a different active computation, while all masks update the weights they share. The paper connects the mechanism to experiments in vision, speech, text, and computational biology. Original paper, §§1–5
This review uses the published JMLR 15(56), pages 1929–1958 version. All interactive values are authored, deterministic calculations. They do not measure generalization accuracy, learn hidden features, or reproduce a benchmark. The experiments discussed later are explicitly attributed to the original study.
Keep probability is not drop probability
The paper samples a binary variable r for each activation: r is one with probability p, and zero otherwise. Thus p is the probability of retaining a unit. A modern API that accepts the probability of dropping a value would use 1−p for the same mask distribution. Confusing those meanings reverses the strength of regularization. At p=1, the original mechanism keeps every unit; smaller p removes more units on average. Paper, §4
Masking an activation also removes its contribution through outgoing connections for that particular forward pass. It does not permanently delete the weight array. On a subsequent training presentation, the unit may return with its existing weights. During backpropagation, the mask gates the corresponding path: if an activation was multiplied by zero, the local derivative through that multiplication is zero as well. Other uses of a shared parameter can still contribute gradients.
A mask changes the active computation, not the stored weights
Toggle three hidden activations. The original dropout convention leaves retained values unscaled during training.
Bias = 0.25. Stored weights stay [2, −1, 1.5]; masked activations are [1, 0, -1].
Independent Bernoulli sampling does not keep exactly p times the units in every realization. A layer with three units and p=0.5 can retain zero, one, two, or three units. What equals 1.5 is the expected count. A tool that always chooses a fixed number of survivors would implement a different mask distribution, even if its average count matched. The figure exposes explicit masks so their graph effects are inspectable rather than hiding the mechanism in a random resampling animation.
With n independently masked units there are 2ⁿ possible binary patterns. They share weights, are generally correlated through those weights, and are not 2ⁿ separately trained models. It is also misleading to imply that each subnetwork receives an equal number of training examples or that training visits every possible mask. The number of possible graphs grows exponentially, while the training budget remains finite.
Why the original paper scales weights at test time
For a fixed activation h, multiplying by a Bernoulli mask gives expectation p·h. A downstream linear sum therefore has an expected contribution w·p·h from that input. The original convention retains activations without rescaling during training, then multiplies outgoing weights by p when using the full network at test time. This reproduces the expected linear contribution without drawing a mask. Paper, Figure 2
The scaling must use the retention probability of the source unit supplying a contribution. When different layers use different probabilities, one global multiplier is insufficient. The bias is not an ordinary masked activation in the paper's schematic and should not acquire an extra p merely because nearby weights do. Getting the convention right requires following the graph, not mechanically scaling every stored number.
Move the scaling from inference to training
Compare the original paper convention with inverted dropout. Here p always means retention probability.
| Quantity | h₁=1 | h₂=2 | h₃=−1 |
|---|---|---|---|
| Expected training value | 0.500 | 1.000 | -0.500 |
| Training variance | 0.250 | 1.000 | 0.250 |
| Deterministic test value | 0.500 | 1.000 | -0.500 |
Many implementations instead use inverted dropout: retain an activation with probability p but divide surviving values by p during training. Its expected training activation is then h, so inference can use the unscaled full network. This moves where the scale adjustment occurs. It does not mean that r·h and r·h/p produce identical numerical training trajectories for a fixed parameterization. They are different magnitudes unless the surrounding parameterization and procedure account for the convention.
The variance also changes with the convention. Original unscaled masking has variance p(1−p)h². Inverted masking has variance (1−p)h²/p. Lower retention therefore makes individual surviving values larger in the inverted form and increases noise around its fixed mean. The p=0 boundary is undefined for inverted scaling because it divides by p. The reference permits p=0 only when demonstrating the original unscaled mechanism, where every activation is zero.
Exact expectation matching ends at nonlinearities
Linearity allows expectation to move through a weighted sum. A nonlinear activation generally prevents it from moving through the entire network: evaluating f at the expected preactivation is not the same as averaging f over random preactivations. This is why replacing all masks by their means is not a general exact arithmetic ensemble of a deep network.
For three units, we can enumerate every mask and evaluate that distinction directly. At a retention probability other than 0.5, masks with different survivor counts have different probabilities. The expected output must weight each one by p raised to its survivor count times (1−p) raised to its dropped count. Simply taking the unweighted mean of eight predictions would be wrong for most p values.
Enumerate all eight networks: where averaging stops being exact
Every mask has its Bernoulli probability. Compare the arithmetic mean of outputs with evaluating the mean preactivation.
r=[0,0,0]
P=0.064
y=0.562
r=[1,0,0]
P=0.096
y=0.905
r=[0,1,0]
P=0.096
y=0.148
r=[1,1,0]
P=0.144
y=0.562
r=[0,0,1]
P=0.096
y=0.223
r=[1,0,1]
P=0.144
y=0.679
r=[0,1,1]
P=0.144
y=0.037
r=[1,1,1]
P=0.216
y=0.223
E[f(z)] = 0.390651 · f(E[z]) = 0.342990 · gap = 0.047661
Choose a linear readout and the two values agree to floating-point tolerance. Choose a sigmoid and they can differ. This is an authored arithmetic-expectation example, not a reproduction of a result from a learned network. The introduction motivates an approximate geometric combination of shared-weight models, while Section 7.5 empirically compares Monte Carlo prediction averaging against weight scaling. Those are different averaging constructions; our small sigmoid example tests only arithmetic expectation and pointwise equality. Paper, introduction and §7.5
In the paper's MNIST Monte Carlo comparison, averaging about 50 sampled networks per test case matches the weight-scaled network's classification error; larger samples perform slightly better but remain within one standard deviation of it. This supports weight scaling as a useful empirical approximation in that experiment. It does not assert identical predictions for every input or make the approximation an algebraic identity. Paper, Figure 11 and §7.5
Likewise, calling dropout “Bayesian model averaging” without qualification overstates the mechanism. The paper motivates efficient model combination, but the sampled subnetworks share parameters and do not receive independently learned posterior weights. The regularizer's practical appeal does not require treating each sampled mask as an exact posterior draw from a fully specified Bayesian model.
Read the source beside the implementation
1. Multiply activations by a Bernoulli mask
Original paper
Section 4 masks each layer’s activations before the next weighted sum. The paper’s p is retention probability.
Math and tensor shapes
A supplied mask makes the calculation reproducible. It changes the current forward graph without deleting stored parameters. The same mask gates the corresponding backward path. The original convention does not divide retained training activations by p.
Python implementation
def apply_mask(x, mask, keep=0.5, inverted=False):
"""Independent Bernoulli masks are supplied explicitly, not sampled here."""
validate(x, keep)
if len(mask) != len(x) or any(v not in (0, 1) for v in mask):
raise ValueError("Binary mask of matching shape required")
if inverted and keep == 0:
raise ValueError("Inverted dropout requires positive retention")
return [v * r / (keep if inverted else 1) for v, r in zip(x, mask)]2. Match the expected linear contribution at test time
Original paper
Figure 2 multiplies outgoing weights by p at inference. Inverted dropout moves the reciprocal scaling to training.
Math and tensor shapes
The mean and variance here are exact for a fixed activation and an independent Bernoulli mask. With inverted dropout, variance becomes (1−p)h²/p and inference is identity. The p=0 boundary is allowed only for the original unscaled convention; inverted scaling would divide by zero.
Python implementation
def moments(x, keep, inverted=False):
"""Exact coordinate means/variances; original p is KEEP probability."""
validate(x, keep)
if inverted and keep == 0:
raise ValueError("Inverted dropout requires positive retention")
scale = 1 / keep if inverted else 1
return dict(mean=[keep * v * scale for v in x],
variance=[keep * (1 - keep) * v * v * scale * scale for v in x],
test=[v * (1 if inverted else keep) for v in x])3. Enumerate a tiny shared-weight mask family
Original paper
Figure 3 connects masking to the sampled subnetwork. The exhaustive three-unit sum is an authored calculation of this mechanism.
Math and tensor shapes
There are 2ⁿ binary masks but their probabilities are equal only at p=0.5. The code compares the arithmetic expected output with f(E[z]). They agree for a linear readout and can differ for a sigmoid. This tests arithmetic expectation, distinct from the paper’s geometric model-combination motivation and does not recreate training or benchmark accuracy.
Python implementation
def enumerate_masks(x, weights, keep, bias=0.25, nonlinear=True):
"""Enumerate an authored tiny network; nonlinear expectation is not exact scaling."""
validate(x, keep)
if len(x) > 12 or len(weights) != len(x) or not all(math.isfinite(v) for v in [bias, *weights]):
raise ValueError("Matching finite weights and at most 12 units required")
def activate(z):
if not nonlinear:
return z
e = math.exp(-z if z >= 0 else z)
return 1 / (1 + e) if z >= 0 else e / (1 + e)
rows = []
for bits in range(2 ** len(x)):
mask = [(bits >> i) & 1 for i in range(len(x))]
count = sum(mask)
probability = keep ** count * (1 - keep) ** (len(x) - count)
z = bias + sum(v * w for v, w in zip(apply_mask(x, mask, keep), weights))
rows.append(dict(mask=mask, probability=probability, z=z, output=activate(z)))
expected_logit = bias + sum(keep * v * w for v, w in zip(x, weights))
return dict(rows=rows, expectedLogit=expected_logit,
expectedOutput=sum(r['probability'] * r['output'] for r in rows),
scaledOutput=activate(expected_logit))Original excerpts: Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov; JMLR 15(56), 2014 publication PDF. Rasterized PDF selections preserve original colors; source and crop hashes are recorded.
The Python reference uses explicit masks and exact enumeration so that every number is reproducible. Its README and requirements document the standard-library-only setup and omissions. A fixed-mask finite-difference test checks that a dropped input has zero local gradient and that an inverted retained input contributes its expected reciprocal-p derivative. Probability-mass and expectation checks test different invariants from the forward values themselves.
This code omits learned layers, an optimizer, training data, and max-norm constraints. Those omissions are intentional: the example explains stochastic masking and expectation, while the original experiments establish whether an actual trained system generalizes better. Running a deterministic mask table cannot establish a classification accuracy, a regularization benefit, or a throughput improvement.
What the MNIST evidence actually compares
The original paper's MNIST setting is permutation invariant: it does not exploit the 2D pixel layout with convolution or augmented image distortions. Table 2 reports several architectures and unit types rather than a single universally controlled dropout toggle. A three-hidden-layer network with 1,024 logistic units per layer reports 1.35% test error; switching that dropout model to ReLU reports 1.25%; adding max-norm constraints reports 1.06%. The listed standard logistic network has two 800-unit layers and 1.60% error, so that row is not an identical-architecture ablation of dropout alone. Paper, §6.1.1 and Table 2
A two-hidden-layer ReLU network with 8,192 units per layer, dropout and max-norm constraints reports 0.95%. The authors emphasize its more than 65 million parameters relative to 60,000 MNIST training cases. These dropout models retain hidden units with p=0.5 and inputs with p=0.8. Those settings belong to the reported experiment; they are not an argument that one rate should be copied into every modern architecture. Paper, §6.1.1
Appendix B.1 makes the evaluation procedure more concrete. The authors hold out 10,000 training images for hyperparameter selection, choose settings around one million weight updates, then combine that validation subset with training data and train the chosen setup for one million updates before evaluating the 10,000-image test set. The stated final momentum is 0.95 and the max-norm constraint c=2. Figure 4 separately examines six architectures formed from two, three, or four hidden layers and widths of 1,024 or 2,048. That provides evidence across architectural settings instead of letting the largest model stand in for every dropout configuration. Paper, Appendix B.1
Noise, capacity, and the training recipe
Dropout changes the objective sampled by stochastic training. A noisier gradient is not automatically an improvement: excessive masking can remove useful signal, slow learning, or produce underfitting. The paper discusses a coupled choice of retention probability, layer width, learning rate, momentum, and max-norm constraints. Reducing retention while keeping every other setting fixed need not preserve model capacity or optimization behavior. Paper, Appendix A
The co-adaptation explanation is a useful motivation, but an individual mask diagram does not prove what learned features represent. Demonstrating robustness requires trained-model comparisons, held-out data, and a specified evaluation procedure. Different dropout placements also inject noise at different parts of a computation; input masking, hidden-unit masking, channel masking, attention dropout, and stochastic depth should not be treated as one identical operator merely because all introduce randomness.
For implementation, I would first write down whether a parameter means “keep” or “drop,” whether scaling happens in training or inference, which axes share a mask, and how evaluation disables sampling. Then I would check the expected linear contribution and a fixed-mask gradient. Only after that would I compare validation behavior using the real training budget. The dropout concept provides a shorter introduction; Batch Normalization shows a different source of training-time batch dependence that also requires an explicit inference path.
