TL;DR
- Adam keeps exponential averages of gradients and squared gradients to combine signed direction with recent coordinatewise scale.
- Bias correction accounts for zero-initialized memory; it does not remove the effects of a changing gradient distribution.
- The timestep, both correction factors, and epsilon placement are essential to reproducing Algorithm 1.
- The paper's experiments support useful optimization behavior in their tested settings, while later work shows that Adam can fail to converge even in a convex setting.
- Use the scalar traces to check the arithmetic, then validate the exact optimizer behavior on the training workload.
Why gradient history matters
A stochastic gradient tells an optimizer which direction increases the current sampled loss. It does not tell the optimizer whether that direction is consistent across batches, whether one coordinate usually receives much larger gradients than another, or how much of a small running estimate comes from starting its memory at zero. Adam builds a practical update out of these three questions.
Kingma and Ba combine an exponential average of gradients with an exponential average of squared gradients, correct both for their zero initialization, and divide the first by the square root of the second. The same arithmetic runs independently for every parameter. This is why a short algorithm can accommodate coordinates with very different gradient histories without constructing a Hessian or a dense preconditioner. Selected paper, Algorithm 1 and §2
This review follows arXiv v9, revised January 30, 2017, of the paper published at ICLR 2015. Those dates describe different things. The interactive values are deterministic calculations on authored scalar gradients, not training runs, pretrained predictions, or latency measurements. The reference deliberately implements the clear Algorithm 1 update so that initialization, correction, and epsilon remain visible.
Two memories answer different questions
Let the current gradient be g. The first accumulator m is a weighted record of signed direction. If a coordinate repeatedly receives positive gradients, its average remains positive; if signs alternate, cancellation reduces that average. The second accumulator v records squared magnitude. Squaring before averaging means that positive and negative gradients both contribute positive values. Calling v “the variance” obscures this distinction: it is an estimate of the second raw moment, rather than an average of squared deviations from the mean. Paper, §2
The two decay factors determine how quickly old evidence fades. A larger beta gives old observations more influence and a longer effective memory. The paper's tested defaults are beta1 = 0.9 and beta2 = 0.999, with alpha = 0.001 and epsilon = 10⁻⁸. These are the paper's practical defaults, not a promise that every architecture or loss uses those values successfully. Keeping the two memories on different time scales is part of the method, rather than a requirement that they estimate the same statistic.
One zero gradient does not erase the past
Choose an authored gradient stream, then inspect both moment accumulators. No network is trained.
Starting view: step 5, just after the impulse at step 4. Reset returns to this view.
Step 5: g = 0; update Δ = 0.000491.
| t | g | m | v | m̂ | v̂ | Δ |
|---|---|---|---|---|---|---|
| 1 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000000 |
| 2 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000000 |
| 3 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000000 |
| 4 | 4.000 | 0.400 | 0.016 | 1.163 | 4.006 | 0.000581 |
| 5 | 0.000 | 0.360 | 0.016 | 0.879 | 3.203 | 0.000491 |
| 6 | 0.000 | 0.324 | 0.016 | 0.691 | 2.668 | 0.000423 |
| 7 | 0.000 | 0.292 | 0.016 | 0.559 | 2.286 | 0.000370 |
| 8 | 0.000 | 0.262 | 0.016 | 0.461 | 1.999 | 0.000326 |
Scroll the trace sideways to inspect every column.
The impulse stream is especially revealing. Three zeros leave the state exactly zero; a later nonzero gradient activates both memories. Subsequent zero gradients decay the memories instead of resetting them. Therefore, a parameter may continue moving when its current gradient is zero. This differs from an entirely zero history, which produces no update. It also differs from a parameter whose gradient is absent and which a framework chooses to skip. The explorer always supplies a numeric gradient at every step.
A changing sign does not immediately reverse the update either. The running first moment can retain the previous direction, while the second moment continues to record the recent scale. This can smooth noisy directions but also delay a response to a genuine change. The scalar table makes that tradeoff inspectable without equating one illustrative sequence with a convergence experiment.
Why zero initialization changes the early steps
An exponential average with infinite stationary history has weights that sum to one. Starting it at zero truncates that history. After t observations, the weights sum to one minus beta raised to t. For a constant gradient of one, the raw first-moment average is exactly that sum, not one. A small early average can therefore reflect missing history rather than small observed gradients. Paper, §3 and Eqs. 1–4
Adam divides m and v by their respective missing-mass factors. On the first observation, the corrected estimates are g and g². On a constant stream they remain those values, even while the uncorrected accumulators are still approaching them. The timestep must begin at one: using zero makes the correction denominator zero. State, step count, and update order are consequently part of the algorithm, not incidental bookkeeping.
Where did the missing EMA mass go?
For a constant gradient g=1, the zero-initialized average equals the sum of its finite-history weights.
Teal: observed weight mass · gray: missing initialization mass.
Lower bars show relative observed weights, oldest to newest.
raw m = 0.271000 · zero-history mass = 0.729000 · corrected m̂ = 1.000000
The derivation is exact for the zero-initialization factor under stationary moment expectations. It should not be expanded into the claim that the corrected average is always an unbiased estimate of the current gradient distribution. When the distribution changes, an exponentially weighted record still mixes earlier and later observations. The paper explicitly includes a nonstationary remainder in its derivation. Bias correction addresses how the memory was initialized; it cannot reveal gradients that have not been observed.
Correcting both moments also matters because they appear in different positions in the ratio. Simply saying that both are small and their errors cancel is generally wrong. With the default coefficients, the first and second moments accumulate at different rates, and the square root changes the second one's scaling. Omitting correction changes the early trajectory even on a perfectly constant gradient stream.
A coordinatewise normalized direction
After correction, Adam forms a parameter update proportional to m-hat divided by the square root of v-hat plus epsilon. The numerator measures signed tendency; the denominator supplies recent magnitude. If every observed gradient in a coordinate is multiplied by a positive constant, the numerator gains that factor and the square-root second moment gains the same factor. They cancel when epsilon is negligible. This helps explain why the method can accommodate coordinates with different gradient scales. Paper, §2.1
That property has limits. A finite additive epsilon breaks exact scale cancellation, especially for tiny gradients. Negative rescaling reverses the direction, and changing gradients through a new parameterization can alter more than their coordinatewise scale. Nor is the normalized ratio a probability, a confidence interval, or a Hessian inverse. Its statistical interpretation provides intuition for an update rule; it does not certify that the next step decreases a nonlinear training loss.
Scale cancellation has an epsilon boundary
Scale every gradient of a constant stream. With tiny epsilon, numerator and root-second-moment scale together.
m̂
1.000000
√v̂
1.000000
Δ / default Δ
1.000000
The position of epsilon is consequential. Algorithm 1 uses sqrt(v-hat) + epsilon. Writing sqrt(v-hat + epsilon) produces a different denominator. Section 2 also presents a rearranged form that moves correction factors into a time-dependent step size and uses a separately denoted epsilon. Copying one expression's epsilon into the other without accounting for the transformation changes the calculation. The supplied implementation follows Algorithm 1 directly rather than claiming all framework formulations are identical.
Epsilon also defines the all-zero boundary. With zero history, both moments and the numerator are zero, while the positive denominator keeps the update defined and equal to zero. There is no useful reason for the teaching implementation to accept epsilon = 0 and then silently produce an undefined 0/0. Its input checks make that boundary explicit. Production implementations additionally manage dtype, loss scaling, state precision, and accelerator kernels; none of those engineering choices is measured by this scalar example.
Read the algorithm beside executable arithmetic
The three rows below pair the relevant original-PDF excerpt with a mathematical reading and a canonical Python source region. The downloaded reference uses only the standard library and prints a deterministic sparse-gradient trace. It does not estimate gradients from a dataset or train a classifier.
1. Accumulate gradient direction and raw magnitude
Original paper
Algorithm 1 keeps two zero-initialized arrays with the same shape as the parameters.
Math and tensor shapes
The Python reference is scalar. A tensor implementation applies the same equations elementwise. v is not a centered variance: alternating +1 and −1 gradients still produce positive second-moment estimates. Gradient production, autodiff, mixed precision and distributed updates are omitted.
Python implementation
def moments(gradients, beta1=0.9, beta2=0.999):
"""Keep the first moment and the second RAW moment, initialized to zero."""
if not all(math.isfinite(x) for x in [beta1, beta2, *gradients]):
raise ValueError("Finite inputs required")
if not (0 <= beta1 < 1 and 0 <= beta2 < 1):
raise ValueError("Decay rates must be in [0,1)")
m, v, history = 0.0, 0.0, []
for t, g in enumerate(gradients, 1):
m = beta1 * m + (1 - beta1) * g
v = beta2 * v + (1 - beta2) * g * g
history.append((t, g, m, v))
return history2. Correct the missing initialization mass
Original paper
Section 3 derives the factor created by initializing exponential averages to zero.
Math and tensor shapes
For stationary moment expectations, division removes the initialization factor. For a changing distribution, it does not make the estimate an unbiased measurement of the instantaneous true gradient. The timestep begins at one and belongs to this optimizer state.
Python implementation
def correct(m, v, t, beta1, beta2):
"""Remove zero-initialization factors; not all nonstationary estimation bias."""
if not isinstance(t, int) or t < 1 or not (0 <= beta1 < 1 and 0 <= beta2 < 1):
raise ValueError("Positive timestep and decay rates in [0,1) required")
return m / (1 - beta1 ** t), v / (1 - beta2 ** t)3. Apply the normalized parameter update
Original paper
Algorithm 1 places epsilon outside the square root of the corrected second moment.
Math and tensor shapes
This is the clear Algorithm 1 form, rather than the rearranged update discussed in Section 2. Moving epsilon inside the root, or absorbing bias factors into alpha without transforming epsilon, changes the arithmetic. Positive epsilon makes an all-zero gradient history produce an exact zero update.
Python implementation
def adam_trace(gradients, alpha=0.001, beta1=0.9, beta2=0.999,
epsilon=1e-8, initial=0.0, corrected=True):
"""Epsilon is OUTSIDE sqrt, as in the selected paper's Algorithm 1."""
if not all(math.isfinite(x) for x in (alpha, epsilon, initial)):
raise ValueError("Finite hyperparameters required")
if alpha < 0 or epsilon <= 0:
raise ValueError("Nonnegative step size and positive epsilon required")
theta, result = initial, []
for t, g, m, v in moments(gradients, beta1, beta2):
mh, vh = correct(m, v, t, beta1, beta2) if corrected else (m, v)
delta = alpha * mh / (math.sqrt(vh) + epsilon)
theta -= delta
result.append(dict(t=t, g=g, m=m, v=v, mHat=mh, vHat=vh,
delta=delta, theta=theta))
return resultOriginal excerpts: Diederik P. Kingma, Jimmy Ba; v9 (2017-01-30); ICLR 2015 publication. Crops preserve original colors. PDF and crop hashes are recorded in the source manifest.
Run adam.py with Python 3.10 or later; the README records its scope and requirements. Its self-checks cover the first-step closed form, all-zero history, and momentum after an impulse. The independent JavaScript tests compare every printed scalar with the browser model. These establish agreement of the represented equations, not the performance of a full optimizer implementation.
What the evidence establishes
Section 6.1 trains L2-regularized multiclass logistic regression on MNIST's 784-dimensional images, using mini-batches of 128 and a step size decaying as 1/sqrt(t). Adam and Nesterov-momentum SGD show similar convergence in the reported training-negative-log-likelihood curve, both ahead of AdaGrad. On IMDB reviews represented by the 10,000 most frequent bag-of-words features, the sparse-feature experiment includes 50% feature dropout. Adam tracks AdaGrad's fast convergence while Nesterov SGD lags. These two cases already show why “Adam wins every comparison” is the wrong reading. Paper, §6.1 and Figure 1
The MNIST multilayer experiment uses two 1,000-unit ReLU hidden layers and mini-batches of 128. The paper separates an L2-regularized deterministic objective from a dropout objective; Figure 2 reports favorable progress for Adam, including comparison with the SFO quasi-Newton method on the deterministic case. On CIFAR-10, the network uses c64–c64–c128 convolution stages, 5×5 filters, 3×3 stride-two max pooling, and a 1,000-unit ReLU layer. Inputs are whitened, with dropout at the input and fully connected layer. Figure 3 shows fast early progress for Adam and AdaGrad, but later much slower AdaGrad convergence; the authors describe Adam's improvement over momentum SGD here as marginal. These are paper-reported learning curves, not newly reproduced measurements. Paper, §§6.2–6.3 and Figures 2–3
The correction ablation is unusually relevant to implementing the method: a VAE with a 500-unit softplus hidden layer and 50-dimensional spherical Gaussian latent variable is evaluated across beta1 in 0.9, beta2 in 0.9999, and a broad learning-rate range. Figure 4 reports early instability without correction when beta2 is close to one. Comparisons share initial parameters and use searched hyperparameters, rather than testing only each method's default. Those settings support useful, bounded conclusions about the original experiments; they do not establish a universal accuracy ranking or a hardware-independent speedup. Paper, §6.4 and Figure 4
The convergence discussion also needs historical context. The paper presents an online convex optimization regret analysis under boundedness and schedule assumptions. It is already narrower than a guarantee for arbitrary deep nonconvex networks. More importantly, Reddi, Kale, and Kumar's ICLR 2018 paper identifies problems in the earlier analysis and gives a convex setting where Adam fails to converge to the optimum. Their work motivates methods with longer memory, including AMSGrad. It would be inaccurate to present the original theorem as a universally valid convergence guarantee. On the Convergence of Adam and Beyond
This caveat does not erase Adam's practical contribution. It separates an influential, useful update rule from a claim that its empirical success follows from a general theorem. The running-square average can forget large past gradients, so its effective coordinate scaling need not behave like a monotonically accumulating AdaGrad denominator. The choice to forget is useful in changing environments and is also part of what makes the analysis subtle.
How I would use the paper
I read Algorithm 1 first, then trace an impulse and a constant stream before reading benchmark curves. Those two inputs catch several common implementation errors: confusing raw second moment with centered variance, applying the wrong timestep, omitting one bias correction, moving epsilon, or treating a zero current gradient as a request to clear state. A tiny scalar program can expose these errors without hiding them behind autodiff and a large training loop.
For a real training system, I would then inspect its optimizer's exact update order, epsilon convention, weight decay behavior, skipped-gradient policy, and state dtype. Adam and decoupled weight decay should not be conflated merely because a library exposes them through nearby options. None of those checks replaces validation on the actual architecture, batch schedule, precision, data, and stopping rule. The paper provides a reusable mechanism; choosing and evaluating a training procedure remains an empirical task.
Continue with gradient flow for the derivatives supplied to an optimizer, or Nadam for a related momentum modification. The useful connection is the division of responsibilities: backpropagation computes derivatives, while Adam transforms their history into parameter updates.
