Skip to main content

VAE Latent Space: Understanding Variational Autoencoders

Explore VAE latent space in deep learning. Learn variational autoencoder encoding, decoding, interpolation, and the reparameterization trick.

A variational autoencoder learns to compress data into a continuous, probabilistic latent space — whose coordinates can be decoded into generated outputs. Continuity of the decoder does not guarantee that every output is plausible. Unlike a plain autoencoder, a VAE encodes each input to a distribution, and a regularizer keeps that space organized.

Interactive VAE latent space

The explorer below is a real VAE — a 2-D-latent model trained in PyTorch on the 1,797 real sklearn 8×8 handwritten digits. Its decoder weights run live in your browser, so every glyph is the trained network turning a latent z back into pixels. Move the pointer or keyboard-accessible coordinate sliders, interpolate between digits, or sample the prior. The provenance disclosure identifies the retained export and the missing historical training metadata.

VAE latent space: a trained decoder

Explore the exported decoder, encoded examples, interpolation, and prior sampling. The browser evaluates the fixed learned weights; it does not retrain the VAE.

A 2-D-latent VAE trained in PyTorch on 1,797 real handwritten digits. The decoder runs live in your browser — every 8×8 glyph is the trained network mapping a latent z back to pixels.

0123456789
decode(z) — z = (-0.92, -1.23)

Move a pointer on the plane, or use these keyboard-accessible coordinates.

-0.92
-1.23

Decoder coordinates z1 -0.92, z2 -1.23.

2
latent dim
2.60
Prototype posterior KL
Encode a real digit
1. A real decoder, run live
Step 1 / 5

Every digit here is the real PyTorch decoder evaluated in your browser — img = sigmoid(W₂·relu(W₁z + b₁) + b₂). The faint field behind the dots is the decoder swept across the whole latent plane; move the pointer or the latent-coordinate sliders and the 8×8 decoder output changes continuously. One trained network, not a lookup table.

Training and artifact provenance

The retained exporter declares 1,797 sklearn digits, 2 latent dimensions, 64 hidden units, Adam at 0.001, 600 epochs, batch size 128, seeds 0, and KL warm-up to β=1 over 100 epochs. It uses sigmoid/BCE reconstruction on intensities normalized to [0,1].

Current exported artifact SHA-256: d703c84b185811a03e32d6a574bb2a5f491af82f84d7a172dbacabd225b00d32

The original checkpoint, training dependency versions, and source-data hash were not retained. Current source/artifact hashes identify the retained files but do not prove training-run reproducibility. This revision preserves the decoder bytes and does not rerun training.

Probabilistic encoding

Instead of a single point, the encoder maps each input to a Gaussian over the latent space:

q_\phi(z \mid x) = \mathcal{N}\big(\mu_\phi(x),\; \sigma_\phi^2(x)\big)

The mean \mu_\phi(x) is where the digit lands; the variance \sigma_\phi^2(x) is how much wiggle room it claims. In the explorer those are the dot and the dashed ring on the highlighted prototype.

The reparameterization trick

An opaque sample operation does not directly provide a pathwise derivative. Reparameterization expresses the sample using parameter-independent noise, so gradients can flow through \mu and \sigma:

z = \mu + \sigma \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)
python
def forward(self, x):
    mu, logvar = self.encode(x)
    z = mu + torch.exp(0.5 * logvar) * torch.randn_like(mu)  # reparameterize
    recon = self.decode(z)
    bce = F.binary_cross_entropy(recon, x, reduction="sum")  # reconstruction
    kl  = -0.5 * torch.sum(1 + logvar - mu**2 - logvar.exp())  # KL to N(0,I)
    return bce + beta * kl                                     # negative ELBO when beta == 1

The loss: reconstruction vs. KL

Helpful background

KL divergence

Explain the asymmetry of KL and its relationship to entropy and cross-entropy.

Open full page

VAEs maximize a lower bound on the log-likelihood (the ELBO), which splits into a term that wants faithful reconstructions and a term that regularizes the latent space toward a standard-normal prior:

\mathcal{L} = \underbrace{\mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)]}_{\text{reconstruction}} \;-\; \underbrace{D_{KL}\big(q_\phi(z|x)\,\|\,p(z)\big)}_{\text{regularization}}

For two Gaussians the KL has a closed form, which is why it is cheap to compute every step:

D_{KL}\big(\mathcal{N}(\mu,\sigma^2)\,\|\,\mathcal{N}(0,1)\big) = \tfrac{1}{2}\sum_{j} \big(\mu_j^2 + \sigma_j^2 - \log \sigma_j^2 - 1\big)

The KL term penalizes posterior means far from zero and variances that differ from one. Reconstruction supplies a competing incentive to preserve information. This pressure does not forbid islands of encoded means, force semantic organization, or guarantee that the aggregate posterior matches the prior.

Why the space is useful

  • Continuity. The ReLU-and-sigmoid decoder is continuous, so a continuous latent line gives continuous pixel values. Some intermediate images can remain ambiguous or implausible (the Interpolate lens).
  • Generation. The prior is p(z) = \mathcal{N}(0, I), so you can sample z from it and decode generated pixel probabilities; novelty and semantic validity are not guaranteed (the Generate lens).
  • Structure. Inspect whether particular directions change stroke width, slant, or style in this decoder. Such associations are observations, not guarantees of disentanglement.

Practical notes

β-VAE. Scaling the KL term changes the pressure toward the prior. It can change reconstruction and representation structure; disentanglement is not guaranteed:

\mathcal{L}_\beta = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - \beta \cdot D_{KL}\big(q_\phi(z|x)\,\|\,p(z)\big)

Posterior collapse. If the KL term dominates (or the decoder is too strong), the encoder gives up and returns the prior for every input — q_\phi(z|x) \approx p(z). KL warm-up and free-bits are mitigation techniques whose effectiveness depends on the setup.

Likelihood and softness. This exporter uses sigmoid outputs and binary cross-entropy, the Bernoulli reconstruction objective applied to grayscale intensities normalized to [0,1]. It does not use a Gaussian likelihood. The figure displays pixel probabilities as intensities, which can look soft; the narrow latent bottleneck also limits reconstruction. Other VAEs may choose Gaussian or richer observation models.

References

  • Kingma & Welling. "Auto-Encoding Variational Bayes" (the original VAE)
  • Higgins et al. "β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework"
  • Rezende et al. "Stochastic Backpropagation and Approximate Inference in Deep Generative Models"

If you found this explanation helpful, consider sharing it with others.

Mastodon