GAN: Generative Adversarial Networks

TL;DR

• GANs pit two networks against each other: a Generator G maps random noise to synthetic samples, and a Discriminator D tries to tell real data from G’s fakes. Alternating updates train the pair; indistinguishability is an idealized optimum, not a guaranteed outcome.
• The training objective is a minimax game — D maximizes the conditional log likelihood of real/fake labels, while G learns through D. The practical non-saturating G objective differs from minimax descent.
• At the theoretical optimum the generated distribution exactly matches the data distribution and the optimal D outputs 1/2 wherever the shared density is positive. This nonparametric result does not guarantee convergence of finite neural networks.
• The 2014 paper established the adversarial framework with MNIST, face and CIFAR-10 experiments; later GAN architectures developed high-fidelity image synthesis.

The two-player game

The central insight of Goodfellow et al. is to frame generation as a game between two neural networks rather than as a likelihood-maximization problem. The Generator G takes a latent noise vector z \sim p_z(z) and maps it to a synthetic sample G(z). The Discriminator D takes any sample — real or generated — and outputs a scalar probability that the sample came from the real data distribution.

The two networks are trained in alternating steps with different parameter updates. D wants to assign high probability to real samples and low probability to generated ones. G wants to produce samples that fool D into assigning high probability. This produces the minimax objective:

D maximizes both terms: it wants D(x) \to 1 for real samples (making \log D(x) large) and D(G(z)) \to 0 for fakes (making \log(1 - D(G(z))) large). G minimizes the second term, pushing D(G(z)) \to 1 so the discriminator is fooled. In practice, early training uses a non-saturating variant — G maximizes \log D(G(z)) rather than minimizing \log(1 - D(G(z))) — which provides stronger gradients when G is far from the optimum.

Implementation

The loop diagram introduces the two players, but the executable step makes the boundary between their updates precise. Algorithm 1 updates D first and then draws fresh noise for G. The reference below uses the non-saturating alternative in Section 3: it minimizes negative log D(G(z)), rather than the original minimax generator term. Watch which parameters and gradients remain fixed in each half of the step.

def synthetic_data(batch=4, latent=2, seed=7):
    """Draw an equal mixture at (-1,-1)/(1,1), with isotropic stddev 0.15.

    Real samples and the two noise minibatches use independent local RNGs.
    The generator update receives fresh noise, as in Algorithm 1.
    """
    positive_integer(batch, "batch")
    positive_integer(latent, "latent")
    real_rng = torch.Generator().manual_seed(seed)
    d_rng = torch.Generator().manual_seed(seed + 1)
    g_rng = torch.Generator().manual_seed(seed + 2)
    mode = 2 * torch.randint(0, 2, (batch, 1), generator=real_rng) - 1
    real = mode + 0.15 * torch.randn(
        batch, 2, generator=real_rng, dtype=torch.float64
    )
    z_d = torch.randn(batch, latent, generator=d_rng, dtype=torch.float64)
    z_g = torch.randn(batch, latent, generator=g_rng, dtype=torch.float64)
    return real, z_d, z_g

def generator(latent=2, hidden=4, seed=7):
    """Construct G: [B,Z] -> [B,2], preserving global RNG state."""
    positive_integer(latent, "latent")
    positive_integer(hidden, "hidden")
    with torch.random.fork_rng(devices=[]):
        torch.manual_seed(seed)
        return nn.Sequential(
            nn.Linear(latent, hidden), nn.Tanh(), nn.Linear(hidden, 2)
        ).double()

def discriminator(hidden=4, seed=8):
    """Construct logits a: [B,2] -> [B,1]; D(x) = sigmoid(a(x))."""
    positive_integer(hidden, "hidden")
    with torch.random.fork_rng(devices=[]):
        torch.manual_seed(seed)
        return nn.Sequential(
            nn.Linear(2, hidden), nn.Tanh(), nn.Linear(hidden, 1)
        ).double()


def logistic_loss(logits, target):
    """Return mean BCE-with-logits for a scalar binary target."""
    matrix(logits, 1, "logits")
    if target not in (0, 1):
        raise ValueError("target must be 0 or 1")
    return F.binary_cross_entropy_with_logits(
        logits, torch.full_like(logits, target), reduction="mean"
    )

def discriminator_objective(g, d, real, z_d):
    """Minimize negative Equation 1 in D, with G detached."""
    matrix(real, 2, "real")
    matrix(z_d, g[0].in_features, "z_d")
    if real.shape[0] != z_d.shape[0]:
        raise ValueError("real and noise batch sizes must agree")
    fake = g(z_d).detach()
    real_logits, fake_logits = d(real), d(fake)
    loss = logistic_loss(real_logits, 1) + logistic_loss(fake_logits, 0)
    return loss, real_logits, fake_logits

def discriminator_update(g, d, real, z_d, learning_rate=0.05):
    """Clear D gradients and take one actual D-only SGD step."""
    if not math.isfinite(learning_rate) or learning_rate <= 0:
        raise ValueError("learning_rate must be finite and positive")
    optimizer = torch.optim.SGD(d.parameters(), lr=learning_rate)
    optimizer.zero_grad(set_to_none=True)
    loss, real_logits, fake_logits = discriminator_objective(g, d, real, z_d)
    loss.backward()
    grad = gradients(d)
    optimizer.step()
    return loss.detach(), grad, real_logits.detach(), fake_logits.detach()

@contextmanager
def frozen_discriminator(d):
    """Freeze D parameters across forward/backward, retaining the path to G."""
    flags = [p.requires_grad for p in d.parameters()]
    try:
        for p in d.parameters():
            p.requires_grad_(False)
        yield
    finally:
        for p, flag in zip(d.parameters(), flags):
            p.requires_grad_(flag)


def generator_objective(g, d, z_g):
    """Re-forward fresh fakes through updated D; minimize -log D(G(z))."""
    matrix(z_g, g[0].in_features, "z_g")
    fake = g(z_g)
    logits = d(fake)  # Neither detach() nor no_grad(): G needs this derivative.
    return logistic_loss(logits, 1), fake, logits

def alternating_step(g, d, real, z_d, z_g, learning_rate=0.05):
    """Execute D first, then non-saturating G, returning full state evidence."""
    matrix(z_g, g[0].in_features, "z_g")
    g_before, d_before = parameters(g), parameters(d)
    g_grads_before = [
        None if p.grad is None else p.grad.clone() for p in g.parameters()
    ]
    d_loss, d_grad, real_logits, fake_logits = discriminator_update(
        g, d, real, z_d, learning_rate
    )
    g_after_d, d_after_d = parameters(g), parameters(d)
    g_untouched = all(
        (old is None and p.grad is None)
        or (old is not None and p.grad is not None and torch.equal(old, p.grad))
        for old, p in zip(g_grads_before, g.parameters())
    )
    d_grad_after_d = [
        None if p.grad is None else p.grad.clone() for p in d.parameters()
    ]
    optimizer = torch.optim.SGD(g.parameters(), lr=learning_rate)
    optimizer.zero_grad(set_to_none=True)
    flags_before = [p.requires_grad for p in d.parameters()]
    with frozen_discriminator(d):
        g_loss, fake, g_logits = generator_objective(g, d, z_g)
        flags_during = [p.requires_grad for p in d.parameters()]
        g_loss.backward()
        g_grad = gradients(g)
        optimizer.step()
    d_grads_untouched = all(
        (old is None and p.grad is None)
        or (old is not None and p.grad is not None and torch.equal(old, p.grad))
        for old, p in zip(d_grad_after_d, d.parameters())
    )
    return {
        "d_loss": float(d_loss),
        "g_loss": float(g_loss.detach()),
        "real_logits": real_logits.tolist(),
        "d_fake_logits": fake_logits.tolist(),
        "g_fake_logits": g_logits.detach().tolist(),
        "generated_samples_before_g_update": fake.detach().tolist(),
        "g_before": g_before.tolist(),
        "g_after_d": g_after_d.tolist(),
        "g_after_g": parameters(g).tolist(),
        "g_gradient": g_grad.tolist(),
        "d_before": d_before.tolist(),
        "d_after_d": d_after_d.tolist(),
        "d_after_g": parameters(d).tolist(),
        "d_gradient": d_grad.tolist(),
        "g_gradients_untouched_by_d": g_untouched,
        "d_gradients_untouched_by_g": d_grads_untouched,
        "d_flags_before": flags_before,
        "d_flags_during_g": flags_during,
        "d_flags_after": [p.requires_grad for p in d.parameters()],
        "finite_gradients": bool(
            torch.isfinite(g_grad).all() and torch.isfinite(d_grad).all()
        ),
        "g_gradient_norm": float(torch.linalg.vector_norm(g_grad)),
        "d_gradient_norm": float(torch.linalg.vector_norm(d_grad)),
        "g_parameter_change": float(
            torch.linalg.vector_norm(parameters(g) - g_before)
        ),
        "d_parameter_change": float(
            torch.linalg.vector_norm(d_after_d - d_before)
        ),
    }

python -m pip install -r requirements-gan.txt
python gan.py --smoke --seed 7

That numerical trace verifies one alternating update. Distribution matching requires a different argument: the next result assumes an optimal discriminator in a nonparametric setting, rather than the tiny D just updated here.

Learning the data distribution

Section 4 analyzes the idealized nonparametric game. For a fixed G and wherever the denominator is positive, the optimal discriminator is:

where p_g is the distribution induced by G. Substituting D^* back into the objective yields the global minimum at p_g = p_{\text{data}}, where the objective value equals -\log 4. At this point D^*(x) = 1/2 wherever the shared density is positive. Where both densities vanish, the criterion places no constraint on D. This is an optimal-discriminator statement, not a measured property of the toy network.

The proof also shows that the objective is equivalent to minimizing the Jensen–Shannon divergence between p_{\text{data}} and p_g:

The plot below illustrates the theorem with a handcrafted interpolation between densities and the corresponding analytic D*. It does not optimize neural networks or demonstrate learned convergence. The finite-capacity training loop can fail even though the idealized criterion has the stated optimum.

Mode collapse

GANs can be difficult to train. One practical failure is mode collapse: G can exploit the current D by producing samples from a small subset of the data distribution and repeating them, rather than covering its full diversity. D may learn to reject those outputs, G may shift to a different mode, and the two networks can cycle without covering the whole distribution.

Mode collapse is a practical failure of the alternating optimization and finite models, not the optimum of the idealized distribution-matching criterion. A weak or changing D can supply gradients that favor a narrow subset of outputs. Later work explored techniques such as minibatch discrimination, unrolled GANs, spectral normalization (SN-GAN), and Wasserstein distance training (WGAN); none guarantees that every practical run covers every mode. The dots below are crafted scenarios, not samples from this one-step reference.

Why it mattered

GANs offered a new way to fit a sampler through a learned discriminator without explicitly evaluating the generator’s density. Earlier generative families used different likelihoods, latent-variable approximations and sampling procedures; they did not all optimize a pixel-level reconstruction loss. The 2014 work demonstrated the potential of adversarial training, while later architectures and larger-scale training improved image fidelity.

The cascade of follow-on work was immediate. DCGAN (2015) stabilized GAN training with convolutional architectures and batch normalization. pix2pix (2017) extended GANs to conditional image-to-image translation. StyleGAN and StyleGAN2 (2019–2020) produced photorealistic face synthesis indistinguishable from photographs at 1024×1024 resolution. BigGAN (2018) brought class-conditional ImageNet generation to high fidelity.

The GAN framework also reframed generation as adversarial learning — a design pattern that found applications well beyond images, including text generation, domain adaptation, data augmentation, and neural rendering.

Diffusion models such as DDPM (2020) provide an alternative that removes the adversarial two-player game. DDPM develops a variational likelihood bound and a practical denoising loss; these objectives do not guarantee complete mode coverage in a finite trained model. A GAN generates a sample in one forward pass, while iterative diffusion sampling uses multiple steps. Relative latency depends on the model, sampler, number of steps and hardware, so neither family has a universal speed advantage.

Related Reading

• DDPM — denoising diffusion probabilistic models, an alternative trained without an adversarial discriminator
• Latent Diffusion — diffusion in a compressed latent space, enabling high-resolution generation at practical compute cost
• VAE — variational autoencoders, an earlier probabilistic generative framework that GANs were benchmarked against
• Flow Matching — learns continuous vector fields along chosen conditional probability paths; the resulting marginal trajectories need not be straight