TL;DR
- DDPM generates images by learning to reverse gradual Gaussian noising. Under the original 1000-step schedule, the forward terminal distribution is approximately Gaussian, with a small residual signal.
- A timestep-conditioned denoiser predicts added noise using a simplified MSE loss, avoiding the adversarial minimax dynamics of GAN training without guaranteeing successful optimization or full distribution coverage.
- The paper reports image sample quality competitive with contemporary GANs on its evaluated benchmarks; in its ablation, the simplified denoising objective improves sample quality while the full variational objective gives better likelihood codelengths.
- Original DDPM sampling requires 1000 sequential denoiser evaluations. This makes generation costly, but evaluation counts alone do not establish a fixed wall-clock slowdown relative to GANs or VAEs.
The Core Idea: Noise and Denoise
The central insight of diffusion models is elegantly simple: if you can systematically destroy information, you can learn to reverse that destruction.
The forward process q takes a clean data sample \mathbf{x}_0 and gradually adds Gaussian noise over T timesteps, producing a sequence \mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_T where each step makes the image slightly noisier. By the final step, \mathbf{x}_T is approximately Gaussian noise under the original 1000-step schedule, with a small residual signal. This approximation depends on the schedule and does not hold for every reduced step count.
The reverse process p_\theta learns to undo this destruction. A neural network (typically a U-Net) is trained to take a noisy image \mathbf{x}_t and predict the noise that was added, effectively learning to denoise one step at a time. At generation time, we start from pure noise \mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) and iteratively apply the learned denoiser, producing progressively cleaner images until we arrive at a realistic sample \mathbf{x}_0.
Each forward step is a simple Gaussian transition:
where \beta_t is a small noise variance that controls how much noise is added at step t. The signal is scaled by \sqrt{1-\beta_t} to keep the variance bounded, while fresh noise with variance \beta_t is injected.
The Forward Process: Destroying Information
The forward process is a fixed Markov chain — it has no learnable parameters. At each timestep t, the image is scaled down slightly and fresh Gaussian noise is added. The noise variance \beta_t follows a predetermined schedule, typically increasing linearly from \beta_1 = 10^{-4} to \beta_T = 0.02 over T = 1000 steps.
A critical property makes training efficient: thanks to the reparameterization trick, we can sample \mathbf{x}_t at any arbitrary timestep directly from \mathbf{x}_0 without running through all previous steps. Define \alpha_t = 1 - \beta_t and \bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s. Then:
This means we can write any noisy sample as a simple linear combination:
The coefficient \sqrt{\bar{\alpha}_t} controls how much of the original signal remains, while \sqrt{1-\bar{\alpha}_t} controls the noise amplitude. As t increases, \bar{\alpha}_t decreases toward zero, and the signal is progressively overwhelmed by noise. At t = T, \bar{\alpha}_T \approx 0 and the sample is essentially pure noise.
Noise Schedules: How Fast to Add Noise
The noise schedule — the sequence of \beta_t values across timesteps — determines how quickly information is destroyed during the forward process. This choice has a significant impact on both training efficiency and sample quality.
The linear schedule used in the original DDPM paper increases \beta_t linearly from 10^{-4} to 0.02. The timestep draw remains uniform, but \bar{\alpha}_t drops too quickly in early timesteps, a larger range of times has low signal-to-noise ratio. This changes which noise levels uniform timestep sampling emphasizes; it does not make the earliest timesteps less likely to be drawn.
The cosine schedule, introduced by Nichol and Dhariwal in their Improved DDPM paper (2021), addresses this by designing \bar{\alpha}_t directly as a cosine curve:
where s = 0.008 is a small offset that prevents \beta_t from being too small near t = 0. The cosine schedule preserves signal much longer in the early timesteps, giving the model more training signal at low noise levels where perceptual quality is determined.
The Reverse Process: Learning to Denoise
The reverse process is where learning happens. Given a noisy image \mathbf{x}_t and the current timestep t, a U-Net architecture predicts the noise \boldsymbol{\epsilon} that was added. The architecture uses sinusoidal timestep embeddings (similar to positional encodings in transformers) to condition the network on t, telling it how much noise to expect.
The U-Net is a natural fit for this task: its encoder-decoder structure with skip connections allows it to capture both global structure and fine-grained details. The encoder progressively downsamples the spatial dimensions to capture semantic content, while the decoder upsamples back to the original resolution using skip connections to preserve spatial detail. Self-attention layers at lower resolutions help the model reason about global image coherence.
The simplified training objective from the paper is a straightforward MSE loss on the predicted noise:
The expectation is over uniformly sampled timesteps t \sim \text{Uniform}(1, T), training images \mathbf{x}_0 \sim q(\mathbf{x}_0), and noise samples \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}). This simplified objective drops the weighting term from the full variational lower bound and changes the relative weighting of denoising tasks. In the paper’s ablation it improves sample quality, while optimizing the true variational bound gives better codelengths. The unweighted MSE by itself is not an asserted looser likelihood bound.
There are three common ways to parameterize what the network predicts:
-
\boldsymbol{\epsilon}-prediction (original DDPM): predict the noise that was added. The denoising update rule then uses this to compute \mathbf{x}_{t-1}.
-
\mathbf{x}_0-prediction: predict the clean image directly. The posterior mean formula then combines this with \mathbf{x}_t to find \mathbf{x}_{t-1}.
-
v-prediction (from progressive distillation): predict a velocity \mathbf{v} = \sqrt{\bar{\alpha}_t}\boldsymbol{\epsilon} - \sqrt{1-\bar{\alpha}_t}\mathbf{x}_0, a later parameterization whose training weighting differs when used with an unweighted MSE.
At a fixed timestep these predictions can be converted algebraically to the same reverse mean. Their unweighted training losses are not identical: conversion introduces timestep-dependent factors. The executable reference below uses original epsilon prediction; v-prediction remains later work.
Implementation
Training jumps directly to one noisy time per example; sampling traverses the reverse schedule. This reduced reference connects the sampled epsilon target to an actual update and then to the reverse mean, with every schedule coefficient and tensor reduction visible.
Authored seeded offline CPU mechanics in PyTorch float64: four synthetic vectors, three features, hidden width eight, eight timesteps and one SGD update. Original linear beta endpoints, epsilon prediction and fixed posterior beta-tilde variance; no x0 clipping. With T=8, terminal alpha_bar is about 0.9222 and SNR is about 11.86: substantial signal remains, rather than pure noise. There is no image U-Net, checkpoint, image-quality evidence or benchmark reproduction. Cosine schedules, DDIM, DPM-Solver and v-prediction remain later work.
1. Build the linear schedule and posterior variance
Original paper
The forward chain fixes beta; the reverse variance is a declared choice.
Math and tensor shapes
The endpoints are the original 0.0001 and 0.02, inclusive. Paper times 1…T map to Python indices 0…T−1. previous starts with alpha_bar_0=1, so beta_tilde_1=0. This reference chooses fixed posterior beta-tilde variance, one of the two options in §3.2. Eight steps are a reduction; the paper uses 1000.
beta, alpha, alpha_bar, previous and posterior_variance [8]
Python implementation
def schedule(steps):
"""Build original linear endpoints; index i means paper time t=i+1."""
if not isinstance(steps, int) or isinstance(steps, bool) or steps < 2:
raise ValueError("steps must be an integer >= 2")
beta = torch.linspace(1e-4, 0.02, steps, dtype=DTYPE)
alpha = 1.0 - beta
alpha_bar = torch.cumprod(alpha, dim=0)
previous = torch.cat((torch.ones(1, dtype=DTYPE), alpha_bar[:-1]))
posterior_variance = beta * (1.0 - previous) / (1.0 - alpha_bar)
return {"beta": beta, "alpha": alpha, "alpha_bar": alpha_bar,
"previous": previous, "posterior_variance": posterior_variance}2. Draw one timestep and one noise vector per example
Original paper
Training samples a random time rather than traversing the forward chain.
Math and tensor shapes
A local seeded CPU generator draws independent synthetic clean vectors, a timestep for each example and Gaussian noise. These vectors replace the paper’s image distribution. Holding a sampled batch fixed lets the test compare the same objective before and after SGD; it does not establish learned image semantics.
synthetic x0 [4,3], indices [4], epsilon [4,3]
Python implementation
def sample_training_batch(batch, dimensions, steps, generator):
"""Sample synthetic clean vectors, independent timesteps and noise."""
if any(not isinstance(n, int) or isinstance(n, bool) or n < 1
for n in (batch, dimensions, steps)):
raise ValueError("batch, dimensions and steps must be positive integers")
x0 = torch.randn(batch, dimensions, generator=generator, dtype=DTYPE)
indices = torch.randint(steps, (batch,), generator=generator)
epsilon = torch.randn(batch, dimensions, generator=generator, dtype=DTYPE)
return x0, indices, epsilon3. Jump directly to the selected noisy sample
Original paper
The forward marginal has a closed-form reparameterization.
Math and tensor shapes
Gather keeps each example’s timestep and adds the singleton feature axis explicitly. The signal and noise terms are computed once without simulating earlier steps. Independent boundary fixtures check the first and last times in the same batch; malformed indices and implicit tensor broadcasting are rejected.
gathered coefficients [4,1] broadcast over vectors [4,3]
Python implementation
def q_sample(x0, indices, epsilon, coefficients):
"""Sample x_t directly, without traversing earlier forward steps."""
check_pair(x0, epsilon)
alpha_bar = gather(coefficients["alpha_bar"], indices, x0.shape[0])
signal = alpha_bar.sqrt() * x0
noise = (1.0 - alpha_bar).sqrt() * epsilon
return signal + noise4. Condition a tiny denoiser on paper time
Original paper
The paper shares denoiser parameters across time.
Math and tensor shapes
This authored MLP concatenates a normalized scalar time. The paper instead uses an image U-Net with sinusoidal time embeddings and attention. The reference exposes the conditioning interface and preserves the caller’s random state during seeded initialization; it is not that architecture or a pretrained denoiser.
x_t [4,3] + time [4,1] → hidden [4,8] → prediction [4,3]
Python implementation
class EpsilonModel(nn.Module):
"""Tiny time-conditioned MLP; this is not the paper's image U-Net."""
def __init__(self, dimensions, hidden, steps, seed):
super().__init__()
if any(not isinstance(n, int) or isinstance(n, bool) or n < 1
for n in (dimensions, hidden, steps)):
raise ValueError("model dimensions and steps must be positive integers")
self.steps = steps
with torch.random.fork_rng(devices=[]):
torch.manual_seed(seed)
self.network = nn.Sequential(
nn.Linear(dimensions + 1, hidden, dtype=DTYPE), nn.Tanh(),
nn.Linear(hidden, dimensions, dtype=DTYPE))
def forward(self, xt, indices):
"""Concatenate normalized paper time and return predicted [B,D] noise."""
check_pair(xt, xt)
gather(torch.ones(self.steps, dtype=DTYPE), indices, xt.shape[0])
time = (indices.to(DTYPE) + 1.0)[:, None] / self.steps
return self.network(torch.cat((xt, time), dim=1))5. Minimize epsilon MSE with an actual SGD update
Original paper
The simplified objective drops the variational bound’s timestep weights.
Math and tensor shapes
Feature mean then batch mean follows the original code’s MSE convention. The paper writes a squared norm; dividing by the fixed D changes overall scale, not timestep weighting. One SGD step clears gradients, backpropagates, checks finite gradients and changes the parameters. Fixed-batch finite differences and an independent SGD oracle verify the update and a decrease in the same loss. This is not the full ELBO or its discrete decoder.
element errors [4,3] → per-example loss [4] → scalar []
Python implementation
def training_loss(model, xt, indices, epsilon):
"""Use feature-mean then batch-mean MSE, not the full weighted ELBO."""
check_pair(xt, epsilon)
prediction = model(xt, indices)
per_example = (prediction - epsilon).square().mean(dim=1)
return per_example.mean(), per_example
def training_step(model, xt, indices, epsilon, learning_rate=0.01):
"""Take one actual SGD update and expose gradients and changed state."""
if not 0 < learning_rate < float("inf"):
raise ValueError("learning rate must be finite and positive")
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
before = [parameter.detach().clone() for parameter in model.parameters()]
optimizer.zero_grad()
loss, per_example = training_loss(model, xt, indices, epsilon)
loss.backward()
gradients = [parameter.grad.detach().clone() for parameter in model.parameters()]
if not all(torch.isfinite(g).all() for g in gradients):
raise ValueError("nonfinite gradient")
optimizer.step()
with torch.no_grad():
after_loss, _ = training_loss(model, xt, indices, epsilon)
return {"loss_before": loss.detach(), "loss_after": after_loss,
"per_example": per_example.detach(), "before": before,
"gradients": gradients, "learning_rate": learning_rate}6. Turn predicted epsilon into the exact reverse mean
Original paper
The mean is the posterior mean with a reconstructed clean sample.
Math and tensor shapes
The exact beta coefficient matters: subtracting epsilon without these factors is not Equation 11. There is no x0 clipping here. An independent scalar oracle reconstructs x0 and substitutes it into Equation 7’s two-term posterior mean, checking equivalence at both boundary times and an interior time. A supplied epsilon fixture tests algebra, not denoiser quality.
beta, alpha and alpha_bar [4,1]; reverse mean [4,3]
Python implementation
def reverse_mean(xt, indices, epsilon_prediction, coefficients):
"""Evaluate Equation 11 directly, without clipping reconstructed x0."""
check_pair(xt, epsilon_prediction)
beta = gather(coefficients["beta"], indices, xt.shape[0])
alpha = gather(coefficients["alpha"], indices, xt.shape[0])
alpha_bar = gather(coefficients["alpha_bar"], indices, xt.shape[0])
return (xt - beta / (1.0 - alpha_bar).sqrt() * epsilon_prediction) / alpha.sqrt()7. Add supplied noise and visit every declared reverse time
Original paper
Sampling visits T…1 and returns the final mean without stochastic noise.
Math and tensor shapes
The caller supplies each noise tensor, indexed by the schedule index, so replay is exact. The final Python index 0 disables noise for every example. The full loop runs the updated MLP at all eight times and reports each state, time, noise flag and finiteness. Initial Gaussian noise is an authored sampling fixture: the tiny schedule’s forward terminal distribution still retains signal, so this is mechanics evidence rather than a validated generative model.
initial [4,3], supplied noises [8,4,3], eight states [4,3]
Python implementation
def sample_reverse_step(xt, indices, epsilon_prediction, noise, coefficients):
"""Use supplied noise and fixed beta-tilde; paper t=1 suppresses noise."""
check_pair(xt, noise)
mean = reverse_mean(xt, indices, epsilon_prediction, coefficients)
variance = gather(coefficients["posterior_variance"], indices, xt.shape[0])
active = (indices > 0)[:, None]
return mean + active * variance.sqrt() * noise
@torch.no_grad()
def sample_reverse_loop(model, initial, supplied_noises, coefficients):
"""Visit every declared timestep and return the complete state trace."""
steps = len(coefficients["beta"])
check_pair(initial, initial)
if supplied_noises.shape != (steps, *initial.shape):
raise ValueError("supplied noises must be [T,B,D]")
xt = initial.clone()
trace = []
for index in range(steps - 1, -1, -1):
indices = torch.full((xt.shape[0],), index, dtype=torch.long)
epsilon_prediction = model(xt, indices)
xt = sample_reverse_step(xt, indices, epsilon_prediction,
supplied_noises[index], coefficients)
trace.append({"index": index, "paper_time": index + 1,
"noise_enabled": index > 0, "state": xt.tolist(),
"finite": bool(torch.isfinite(xt).all())})
return xt, traceThe smoke prints the actual source SHA-256, seed, exact reductions, shapes, terminal SNR, losses, gradient and state norms, supplied noise and the full reverse-loop trace. The original implementation’s linear schedule, fixedsmall variance and MSE reduction are pinned for provenance; this CPU source is an authored reduction.
python -m pip install -r requirements-ddpm.txt python ddpm.py --smoke --seed 7
The eight-step trace verifies these operations. The original image model instead uses a trained U-Net and a 1000-step schedule whose forward terminal distribution is close to its Gaussian sampling prior.
Sampling: From Noise to Image
Once trained, generating an image follows a simple iterative procedure. Start with \mathbf{x}_T \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) and for each timestep from T down to 1:
where \mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) for t > 1 and \mathbf{z} = \mathbf{0} for t = 1. The noise term \sigma_t \mathbf{z} adds stochasticity to the sampling process, which helps with diversity.
The major drawback is speed: DDPM requires all T = 1000 sequential denoising steps. That is 1000 denoiser evaluations, compared with one generator or decoder evaluation for many GANs and VAEs. It is not a measured 1000× wall-clock difference: architecture, hardware and batching affect runtime. Two key follow-up works addressed this:
DDIM (Denoising Diffusion Implicit Models, Song et al. 2021) showed that the reverse process can be reformulated as solving an ordinary differential equation (ODE). This allows deterministic sampling and — crucially — enables skipping timesteps. With only 50 steps, DDIM can retain useful sample quality with far fewer evaluations; the quality trade-off depends on the dataset, model and chosen step count.
DPM-Solver (Lu et al. 2022) applies higher-order numerical ODE solvers (analogous to Runge-Kutta methods) to the diffusion ODE. By using second and third-order solvers, it achieves high-quality samples in as few as 10–20 steps — 50–100 times fewer denoiser evaluations than 1000 steps, rather than a universal wall-clock speedup.
How DDPM Compares
Generative Model Comparison
How DDPM compares to other generative model families across key dimensions. These are qualitative tendencies, not universal guarantees.
| Method | Training Stability | Mode Coverage | Sample Quality | Speed | Likelihood |
|---|---|---|---|---|---|
| DDPM | Stable MSE loss | Evaluate empirically | FID 3.17 | 1000 steps | Tractable ELBO |
| GAN | Mode collapse risk | Mode dropping | Sharp outputs | 1 forward pass | Implicit density |
| VAE | Stable ELBO | Posterior collapse | Often blurry | 1 forward pass | ELBO bound |
| Normalizing Flow | Exact likelihood | Bijective mapping | Architecture limits | Depends on depth | Exact log p(x) |
DDPM
Iterative denoising via learned reverse Markov chain
GAN
Adversarial min-max game between generator and discriminator
VAE
Encoder-decoder with latent Gaussian prior
Normalizing Flow
Invertible transforms with change-of-variables formula
Use DDPM when…
- You want high-quality samples without an adversarial game; mode coverage still requires empirical evaluation
- Training stability is critical — simple MSE loss, no adversarial dynamics
- You want a tractable likelihood bound for model comparison or density estimation
Consider alternatives when…
- Sampling speed is paramount — GANs and VAEs generate in a single forward pass
- You need real-time generation — 1000 sequential denoising steps is prohibitive
- You require exact likelihoods rather than bounds — normalizing flows are better suited
Key Results
Unconditional Image Generation
On CIFAR-10, DDPM achieves an FID score of 3.17, which at the time of publication was competitive with the best GAN results using the simplified denoising objective. The 3.17 score uses training-set reference statistics; the reported test-reference FID is 5.24. On 256×256 LSUN datasets, the model produces high-quality bedroom and church images with an FID of 4.90 for the larger LSUN Bedroom model (6.36 for the smaller model).
Connection to Variational Inference
DDPM is formally a hierarchical variational autoencoder with T latent variables. The forward process defines the approximate posterior q(\mathbf{x}_{1:T} | \mathbf{x}_0), and the reverse process defines the generative model p_\theta(\mathbf{x}_{0:T}). The training objective is derived from the evidence lower bound (ELBO):
The simplified MSE loss used in practice is a reweighted version of this bound that drops the per-timestep weighting factors. This reweighting empirically improves sample quality in the paper’s ablation, but the unweighted MSE is not itself a guaranteed likelihood bound. Likelihood evaluation still uses the full variational bound and discrete decoder terms.
Log-Likelihood
Table 1 reports a test codelength upper bound of 3.70 bits/dim for the model trained on the full variational objective, versus 3.75 for the simplified-loss model with FID 3.17. These are different trained models, and the paper notes that its codelengths were not competitive with the strongest likelihood models. The later Improved DDPM paper studies learned variance (predicting \sigma_t in addition to \boldsymbol{\epsilon}) alongside cosine schedules, changed training objectives and larger training runs. Its results are separate from this original fixed-variance reference. See Improved DDPM for the experiment-specific likelihood and sampling comparisons.
Key Takeaways
-
Diffusion as iterative refinement — generating images by gradually denoising from pure noise is a fundamentally different paradigm from adversarial training (GANs) or single-step decoding (VAEs), and it produces remarkably stable training dynamics.
-
The simplified objective works best — predicting the noise \boldsymbol{\epsilon} with a simple MSE loss improves sample quality in the paper’s ablation; the full variational objective gives better likelihood codelengths.
-
The forward process enables direct training — the reparameterization trick lets us jump to any timestep in O(1), making training efficient despite the 1000-step Markov chain.
-
Speed is the main weakness — 1000 sequential denoising steps makes generation slow, but this has been largely solved by DDIM and DPM-Solver, reducing the gap to 20–50 steps.
-
Likelihood-based training without a coverage guarantee — diffusion avoids an adversarial minimax game, but finite model capacity, data and optimization can still leave modes poorly represented. Neither the likelihood framework nor simplified MSE guarantees full data-distribution coverage.
Impact and Legacy
DDPM is the foundational paper behind the modern diffusion model revolution. While Sohl-Dickstein et al. (2015) introduced the theoretical framework, it was Ho, Jain, and Abbeel who demonstrated that diffusion models could generate images competitive with GANs — the dominant paradigm at the time.
The impact has been transformative. Stable Diffusion, DALL-E 2, Imagen, and Midjourney all build directly on the DDPM framework. Latent Diffusion Models (Rombach et al. 2022) applied the diffusion process in a compressed latent space rather than pixel space, enabling high-resolution generation with practical compute budgets. Classifier-free guidance (Ho & Salimans 2022) provided a way to control the quality-diversity trade-off, and text conditioning via CLIP or T5 embeddings enabled the text-to-image revolution.
The connection to score-based models deserves mention: Song and Ermon’s concurrent work on score matching showed that DDPM’s denoising objective is related to learning the score of the noisy marginal \nabla_{\mathbf{x}_t} \log q_t(\mathbf{x}_t). Under squared error, the optimal epsilon predictor satisfies \nabla_{\mathbf{x}_t}\log q_t(\mathbf{x}_t) = -\mathbb{E}[\boldsymbol{\epsilon}|\mathbf{x}_t]/\sqrt{1-\bar{\alpha}_t}; this is a time-dependent noisy-data score, not simply the clean-data score. Song et al.’s subsequent unification via stochastic differential equations (Score SDE) showed that both perspectives are instances of the same continuous-time framework, providing a deeper theoretical foundation.
From image generation to video synthesis, audio generation, molecular design, and robotic planning — diffusion models have become the default generative framework across machine learning. DDPM’s contribution was showing that the simplest possible training objective, applied to the simplest possible noise-and-denoise pipeline, was enough to produce state-of-the-art results.
Related Reading
- GAN — the adversarial generative paradigm diffusion models largely displaced, trading GANs’ unstable minimax game for a stable denoising objective
- VAE — the other classic likelihood-based generative model; diffusion can be viewed as a deep hierarchy of denoising latent-variable steps
- Attention Is All You Need — the transformer architecture used in modern diffusion model backbones like DiT
- Vision Transformer — ViT architecture that inspired the Diffusion Transformer (DiT)
- VICReg — another approach to learning without labels, using variance-invariance-covariance regularization
- DINO — self-supervised vision transformers that learn representations through self-distillation
