Skip to main content

KL Divergence in Machine Learning

Learn KL divergence for machine learning: measure distribution differences in VAEs, knowledge distillation, and variational inference.

KL Divergence: Measuring Distribution Differences

Kullback-Leibler (KL) divergence quantifies how one probability distribution differs from another. It is a cornerstone of modern machine learning — variational autoencoders use it to regularize latent spaces, knowledge distillation uses it to transfer knowledge between models, and variational inference uses it to approximate intractable posteriors.

KL divergence answers a simple question: if the true data follows distribution P, how much information do we waste by encoding it using distribution Q instead?

The Weather Forecaster Analogy

Imagine two weather prediction systems for a city. One system, P, perfectly reflects the actual weather frequencies. The other, Q, is a forecaster's model that may not match reality. If we used Q's probability assignments to build our encoding scheme (how many bits per weather event), we'd waste extra "surprise bits" every time reality deviates from Q's predictions. KL divergence measures exactly this waste.

Choose a tropical Sunny Region where sun dominates and the forecast is uniform, a monsoon Rainy Region whose forecast is biased toward sun, or Perfect Match with identical probabilities. Press Play to encode each event and compare optimal and actual bit costs; larger mismatches waste more bits.

Weather encoding
Step 0 / 4
EventReality (P)Forecaster (Q)
Sunny
60%
25%
Cloudy
25%
25%
Rainy
10%
25%
Snowy
5%
25%
Reality (P)Forecaster’s model (Q)
Encoding cost per event

Optimal → actual bits; event contribution in nats.

Encoded 0 of 4 events.

Mathematical Definition

Discrete Distributions

For discrete probability distributions P and Q over the same events:

D_{KL}(P \| Q) = \sum_{x} P(x) \log\frac{P(x)}{Q(x)}

Continuous Distributions

For continuous probability densities:

D_{KL}(P \| Q) = \int_{-\infty}^{\infty} p(x) \log\frac{p(x)}{q(x)} \, dx

Information-Theoretic Interpretation

Helpful background

Cross-entropy loss

Connect probability assigned to the target with the classification loss.

Open full page

KL divergence equals the expected extra bits needed to encode data from P using a code optimized for Q:

D_{KL}(P \| Q) = H(P, Q) - H(P)

Where H(P, Q) is the cross-entropy between P and Q, and H(P) is the entropy of P. When P = Q, cross-entropy equals entropy, and KL divergence is zero — no wasted bits. This is the only case where KL divergence equals zero.

Adjust the two distributions and watch how the three divergence measures — forward KL, reverse KL, and Jensen-Shannon — respond in real-time. The teal shading shows finite positive forward-KL integrand contributions. KL values use analytic formulas for the selected families; support mismatch is infinite even outside the displayed range. JS uses a separately disclosed numerical integration.

KL explorer
Distribution P
0.0
1.0
Distribution Q
2.0
1.5
P and Q probability densities with KL contribution shading
P (true distribution)Q (model)KL contribution

KL values are analytic (nats), including infinite support mismatch. JS uses 12,000 midpoint bins over a tail-truncated range, normalized to unit mass; the plot shows only x ∈ [−6, 8]. Reverse KL > Forward KL: Q has mass where P is low. Q overestimates some regions.

Forward vs Reverse KL: The Crucial Asymmetry

KL divergence is not symmetric: D_{KL}(P \| Q) \neq D_{KL}(Q \| P). This asymmetry has profound practical consequences.

Forward KL — KL(P||Q) penalizes assigning too little Q density where P has mass. A restricted unimodal approximation to a multimodal target often broadens to cover the modes, hence “mass-covering” or “mean-seeking.” The exact optimum depends on the allowed Q family. Maximum likelihood minimizes forward KL up to terms independent of the fitted parameters.

Reverse KL — KL(Q||P) penalizes Q mass where P density is very low or zero. A restricted unimodal Q can prefer one separated mode rather than bridge a low-density gap. This is a common behavior in variational inference, not a guarantee: a sufficiently expressive family can match every mode.

Watch a single Gaussian Q approximate a bimodal target distribution. Switch the KL direction and press Play to follow the fitting behavior:

Forward and reverse KL
Step 0 / 25
Single Gaussian Q fitting the bimodal target P
P (bimodal target)Q (fitting — mean-seeking)Penalty shading

Fit step 0 of 25.

KL in Practice: VAE Latent Spaces

In Variational Autoencoders, the loss function has two terms: reconstruction loss and KL divergence. The KL term pulls the encoder's approximate posterior q(z|x) toward the prior p(z) = \mathcal{N}(0, I).

Without prior regularization, the learned posterior can differ substantially from the distribution used for generation. Sampling the prior may then visit regions poorly supported during training; this is a risk, not an inevitable geometric pattern.

KL regularization pressures each posterior toward the prior while reconstruction encourages preserving information. A continuous decoder gives continuous pixel outputs, but semantic validity and disentanglement are not guaranteed. The weight \beta controls this tradeoff: \beta = 1 is the standard VAE, while \beta > 1 (beta-VAE) changes the relative weight of regularization; its effects depend on the model and optimization.

The next scatter is an authored family of four diagonal Gaussian posteriors, not a trained VAE. Its control moves their means toward zero and standard deviations toward one by an explicit formula. The readout averages their analytic KL to N(0,I); no reconstruction or generation-quality metric is measured.

Authored Gaussian posterior family
0.0
Animate to:
Step 0 / 20
VAE latent points and standard-normal reference circles
Class AClass BClass CClass DN(0,1), dashed reference

Authored interpolation, not VAE training: qᵢ = N(μᵢ/(1+0.8β), σ²I), σ = 1−0.4/(1+0.8β). Each color samples one explicitly defined posterior. The readout averages their analytic KL to N(0,I); it is not the KL of the aggregate mixture or a reconstruction score.

Interpolation step 0 of 20.

Comparing Divergence Measures

KL divergence is just one way to measure distributional differences. Different measures have different mathematical properties that make them suitable for different tasks.

Divergence measures
KL(P||Q)
Forward KL divergence
Symmetric? poor
KL(P||Q) ≠ KL(Q||P)
Metric? poor
No triangle inequality
Bounded? poor
Can be infinite
Gradients moderate
Probability-coordinate derivative −P/Q diverges as Q → 0 with P > 0; parameter gradients depend on the model
Best use: Maximum likelihood, density estimation
KL(Q||P)
Reverse KL divergence
Symmetric? poor
Same asymmetry issue
Metric? poor
No triangle inequality
Bounded? poor
Can be infinite
Gradients excellent
Depends on parameterization and support; no universal stability guarantee
Best use: Variational inference, compression
JS Divergence
Jensen-Shannon divergence
Symmetric? excellent
JS(P||Q) = JS(Q||P)
Metric? moderate
√JS is a metric
Bounded? excellent
Between 0 and ln(2)
Gradients moderate
Can saturate for distant distributions
Best use: Original GAN training
Wasserstein
Earth mover’s distance
Symmetric? excellent
W(P,Q) = W(Q,P)
Metric? excellent
True metric space
Bounded? poor
Unbounded
Gradients excellent
Can remain informative with separated supports; not everywhere differentiable (W₁ of point masses is |θ|)
Best use: WGAN, optimal transport
Total Variation
Maximum probability difference
Symmetric? excellent
TV(P,Q) = TV(Q,P)
Metric? excellent
True metric space
Bounded? excellent
Between 0 and 1
Gradients poor
Uninformative with non-overlapping supports
Best use: Statistical testing, bounds

When to Use KL Divergence

Choose Forward KL When:

  • Performing maximum likelihood estimation — you want the model to cover all data modes
  • Training density estimation models where missing modes is worse than extra spread
  • Doing knowledge distillation — the student should learn all of the teacher's knowledge

Choose Reverse KL When:

  • Training VAEs or variational models
  • Performing variational inference — you want a tight approximation of a single mode
  • Compressing information — better to be precise about what you model than to spread thin
  • The true posterior is complex but you need a simple parametric approximation

Consider Alternatives When:

  • Distributions may have non-overlapping supports (use Wasserstein distance)
  • You need a symmetric measure (use Jensen-Shannon divergence)
  • Training GANs where gradient quality matters (use Wasserstein distance)
  • You need a true metric satisfying the triangle inequality (use Total Variation or Wasserstein)

Common Pitfalls

1. Zero Probability Blow-up

When Q assigns zero probability to an event that P considers possible, P(x) \log \frac{P(x)}{Q(x)} \to \infty. For true support mismatch the divergence is infinite, not a numerical accident. Smoothing or clipping changes the distributions or objective; report that approximation explicitly. Log-space arithmetic handles small positive probabilities without discarding divergent terms.

2. Wrong Direction Choice

Using forward KL when you want a focused approximation (or reverse KL when you want full coverage) leads to poor results. Match the direction to your objective: maximum likelihood uses forward KL, variational inference uses reverse KL.

3. KL Collapse in VAEs

When the decoder is too powerful, the KL term drives to zero and the encoder ignores the input — known as posterior collapse. Solutions include KL annealing (gradually increasing the KL weight during training), free bits (setting a minimum KL threshold per dimension), or using more expressive posterior families.

4. Numerical Instability

Computing \log \frac{P(x)}{Q(x)} in probability space can overflow. Always work in log-space: compute \log P(x) - \log Q(x) directly from log-probabilities. PyTorch's F.kl_div expects log-probabilities as input for exactly this reason.

If you found this explanation helpful, consider sharing it with others.

Mastodon