Generative Adversarial Networks

pytorch
GAN
Author

Gregor Cerar

Published

2023-10-10

Abstract

This post implements a multilayer perceptron GAN and a Deep Convolutional GAN (DCGAN) in PyTorch, then trains both on MNIST and Fashion-MNIST. The goal is to connect the GAN objective to a working training loop and compare how the two architectures behave. In these single-seed runs, the dense model had smoother loss curves, while the convolutional model produced samples that looked sharper to me.

Introduction

Generative Adversarial Networks (GANs) were introduced by Goodfellow et al. (Goodfellow et al. 2014). A GAN consists of two neural networks: the generator, which creates samples, and the discriminator, which estimates whether a sample came from the training data rather than the generator. Training alternates between improving the discriminator and improving the generator. When training succeeds, the generated samples become harder to distinguish from real ones.

As an analogy, consider two kids, one drawing counterfeit money (“Generator”) and another assessing its realism (“Discriminator”). Over time, the counterfeit drawings become increasingly convincing.

Vanilla GAN

The most fundamental variant of GAN is the “vanilla” GAN, where “vanilla” signifies the model in its original and most straightforward form rather than a flavor. To better understand its mechanism, I’ve illustrated its structure on Figure 1.

Figure 1: GAN architecture
  • Generator \(G(z; w_g)\) takes random noise \(z\) as input and produces fabricated data \(x_f\).
    • \(z\) is a latent vector sampled from the standard normal distribution used in the experiments below.
    • \(w_g\) denotes generator neural network weights.
    • \(x_f\) is a fabricated data sample meant for the discriminator.
  • Discriminator \(D(x; w_d)\) differentiates between real and generated data.
    • \(x\) is either a real sample \(x_r \sim p_\textrm{data}\) or a generated sample \(x_f = G(z; w_g)\) with \(z \sim p_z\).
    • \(w_d\) denotes discriminator neural network weights.

Objective Function

In the equations below, \(D(x)\) is the probability that \(x\) came from the training data. The implementation produces an unbounded logit and lets BCEWithLogitsLoss apply the sigmoid transformation internally.

The discriminator minimizes the binary cross-entropy loss

\[ \mathcal{L}_D = -\frac{1}{2}\left(\mathbb{E}_{x \sim p_\textrm{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]\right). \]

The code uses the non-saturating generator loss, which asks the discriminator to classify generated samples as real:

\[ \mathcal{L}_G = -\mathbb{E}_{z \sim p_z}[\log D(G(z))]. \]

This non-saturating objective has the same ideal solution as the minimax objective but provides stronger gradients when the discriminator confidently rejects early generator samples (Goodfellow et al. 2014).

Minimax Game

The original GAN value function is a two-player minimax game:

\[ \min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_\textrm{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]. \]

In essence:

  • Discriminator: Maximizes the probability assigned to real samples and minimizes it for generated samples.
  • Generator: Changes the generated distribution so that the discriminator assigns higher probability to its samples.

The minimax equation describes the theoretical game. The alternating optimization loop below is a practical approximation, and it uses the non-saturating generator loss defined above.

Implementation Setup

The implementation uses PyTorch for the models and training loop, Matplotlib for figures, a fixed random seed for reproducibility, and a shared helper for initializing model weights.

from collections.abc import Callable, Sequence
from pathlib import Path
from typing import Final

import joblib
import numpy as np
import torch
from matplotlib import pyplot as plt
from torch import Tensor, nn, optim
from torch.utils.data import ConcatDataset, DataLoader, Dataset
from torchinfo import summary
from torchvision import transforms as T
from torchvision.utils import make_grid
from tqdm import tqdm

SEED: Final[int] = 42

PROJECT_PATH = Path.cwd()
FIGURE_PATH = PROJECT_PATH / "figures"
DATASET_PATH = Path.home() / "datasets"

# Common constants for all experiments
IMG_DIM: Final[tuple[int, int, int]] = (1, 28, 28)
device = torch.device("cpu")

if torch.cuda.is_available():
    device = torch.device("cuda")
def weights_init(module: nn.Module) -> None:
    if isinstance(module, nn.Conv2d | nn.ConvTranspose2d):
        nn.init.normal_(module.weight, 0.0, 0.02)
        if module.bias is not None:
            nn.init.constant_(module.bias, 0.0)

    elif isinstance(module, nn.BatchNorm1d | nn.BatchNorm2d):
        nn.init.normal_(module.weight, 1.0, 0.02)
        if module.bias is not None:
            nn.init.constant_(module.bias, 0.0)

    elif isinstance(module, nn.Linear):
        nn.init.normal_(module.weight, 0.0, 0.02)
        if module.bias is not None:
            nn.init.constant_(module.bias, 0.0)

Generator

The dense generator maps a 100-dimensional noise vector through fully connected layers, then reshapes the result into a 1x28x28 image. Its final tanh activation matches the normalized image range of -1 to 1.

class Generator(nn.Module):
    def __init__(self, out_dim: Sequence[int], nz: int = 100, ngf: int = 256, alpha: float = 0.2):
        """
        :param out_dim: output image dimension / shape
        :param nz: size of the latent z vector $z$
        :param ngf: size of feature maps (units in the hidden layers) in the generator
        :param alpha: negative slope of leaky ReLU activation
        """
        super().__init__()
        self.out_dim = out_dim
        self.model = nn.Sequential(
            nn.Linear(nz, ngf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Linear(ngf, 2 * ngf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Linear(2 * ngf, 4 * ngf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Linear(4 * ngf, int(np.prod(self.out_dim))),
            nn.Tanh(),
        )

    def forward(self, x: Tensor) -> Tensor:
        x = self.model(x)
        x = torch.reshape(x, (x.size(0), *self.out_dim))
        return x


summary(Generator(out_dim=(1, 28, 28)), input_size=[128, 100])
==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
Generator                                [128, 1, 28, 28]          --
├─Sequential: 1-1                        [128, 784]                --
│    └─Linear: 2-1                       [128, 256]                25,856
│    └─LeakyReLU: 2-2                    [128, 256]                --
│    └─Linear: 2-3                       [128, 512]                131,584
│    └─LeakyReLU: 2-4                    [128, 512]                --
│    └─Linear: 2-5                       [128, 1024]               525,312
│    └─LeakyReLU: 2-6                    [128, 1024]               --
│    └─Linear: 2-7                       [128, 784]                803,600
│    └─Tanh: 2-8                         [128, 784]                --
==========================================================================================
Total params: 1,486,352
Trainable params: 1,486,352
Non-trainable params: 0
Total mult-adds (Units.MEGABYTES): 190.25
==========================================================================================
Input size (MB): 0.05
Forward/backward pass size (MB): 2.64
Params size (MB): 5.95
Estimated Total Size (MB): 8.63
==========================================================================================

Discriminator

The discriminator flattens each 1x28x28 image and passes it through a dense binary classifier that returns one unbounded logit. BCEWithLogitsLoss combines the sigmoid transformation and binary cross-entropy calculation for numerical stability.

class Discriminator(nn.Module):
    def __init__(self, input_dim: Sequence[int], ndf: int = 128, alpha: float = 0.2):
        super().__init__()
        self.model = nn.Sequential(
            nn.Linear(int(np.prod(input_dim)), 4 * ndf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Dropout(0.3),
            nn.Linear(4 * ndf, 2 * ndf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Dropout(0.3),
            nn.Linear(2 * ndf, ndf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Dropout(0.3),
            nn.Linear(ndf, 1),
        )

    def forward(self, x: Tensor) -> Tensor:
        x = torch.reshape(x, (x.size(0), -1))
        return self.model(x)


summary(Discriminator(input_dim=(1, 28, 28)), input_size=[128, 1, 28, 28])
==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
Discriminator                            [128, 1]                  --
├─Sequential: 1-1                        [128, 1]                  --
│    └─Linear: 2-1                       [128, 512]                401,920
│    └─LeakyReLU: 2-2                    [128, 512]                --
│    └─Dropout: 2-3                      [128, 512]                --
│    └─Linear: 2-4                       [128, 256]                131,328
│    └─LeakyReLU: 2-5                    [128, 256]                --
│    └─Dropout: 2-6                      [128, 256]                --
│    └─Linear: 2-7                       [128, 128]                32,896
│    └─LeakyReLU: 2-8                    [128, 128]                --
│    └─Dropout: 2-9                      [128, 128]                --
│    └─Linear: 2-10                      [128, 1]                  129
==========================================================================================
Total params: 566,273
Trainable params: 566,273
Non-trainable params: 0
Total mult-adds (Units.MEGABYTES): 72.48
==========================================================================================
Input size (MB): 0.40
Forward/backward pass size (MB): 0.92
Params size (MB): 2.27
Estimated Total Size (MB): 3.59
==========================================================================================

Training Loop

The training process is iterative:

  • Update Discriminator: Keep the Generator parameters unchanged while improving the Discriminator’s detection of real vs. fake.
  • Update Generator: Keep the Discriminator parameters unchanged while improving the Generator’s ability to deceive. The Discriminator remains in training mode, so stateful layers such as batch normalization still update their running statistics.

Under the idealized assumptions in the original analysis, the equilibrium has \(p_g = p_\textrm{data}\) and \(D(x) = \frac{1}{2}\). Finite neural networks trained by alternating gradient updates are not guaranteed to reach that point. In practice, training may oscillate, diverge, or collapse, so I train for a fixed number of epochs and inspect both the samples and losses.

Note

I keep both networks in training mode during parameter updates. The .eval() and .train() methods are not performance switches: they change the behavior of layers such as batch normalization and dropout. Switching modes within each update made these runs diverge.

def train_step(
    generator: nn.Module,
    discriminator: nn.Module,
    optim_G: optim.Optimizer,
    optim_D: optim.Optimizer,
    criterion: Callable[[torch.Tensor, torch.Tensor], torch.Tensor],
    real_data: torch.Tensor,
    noise_dim: int,
    device: torch.device,
) -> tuple[float, float]:
    batch_size = real_data.size(0)
    real_data = real_data.to(device, non_blocking=True)

    ### Train Discriminator
    optim_D.zero_grad(set_to_none=True)

    noise = torch.randn(batch_size, noise_dim, device=device)

    output_real = discriminator(real_data)
    real_labels = torch.ones_like(output_real)
    loss_D_real = criterion(output_real, real_labels)

    fake_data = generator(noise)
    output_fake = discriminator(fake_data.detach())
    fake_labels = torch.zeros_like(output_fake)
    loss_D_fake = criterion(output_fake, fake_labels)

    loss_D = (loss_D_real + loss_D_fake) / 2

    loss_D.backward()
    optim_D.step()

    ### Train Generator
    optim_G.zero_grad(set_to_none=True)

    # Freeze D params so autograd does not waste work computing their grads
    for p in discriminator.parameters():
        p.requires_grad_(False)

    # Reuse the existing batch so G's BatchNorm statistics update only once
    output_fake = discriminator(fake_data)
    target_for_g = torch.ones_like(output_fake)
    loss_G = criterion(output_fake, target_for_g)

    loss_G.backward()
    optim_G.step()

    for p in discriminator.parameters():
        p.requires_grad_(True)

    return loss_G.detach().item(), loss_D.detach().item()

Experiments

All experiments use the same learning rate, optimizer \(\beta\) parameters, batch size, number of epochs, and latent dimension. Each result is one seeded run. The plots show raw per-batch losses, while the videos show a fixed set of latent vectors within each run. I therefore treat the comparison as descriptive rather than as a benchmark.

OPTIMIZER_LR = 0.0002
L2_NORM = 1e-5
OPTIMIZER_BETAS = (0.5, 0.999)
N_EPOCHS = 100
BATCH_SIZE = 128
g = torch.Generator()
g.manual_seed(SEED)

loader_kwargs = {
    "num_workers": joblib.cpu_count(only_physical_cores=True),
    "pin_memory": True,
    "shuffle": True,
    "batch_size": BATCH_SIZE,
    "prefetch_factor": 2,
    "persistent_workers": True,
    "worker_init_fn": seed_worker,
    "generator": g,
}

MNIST Digits Dataset

MNIST contains 70,000 grayscale 28x28 images of handwritten digits from 10 classes. Its small images and simple subject matter make it practical for comparing the two GAN architectures without a large computational budget.

In total, the dataset contains 70,000 grayscale images of handwritten digits (from 0 to 9). Each image is 28x28 pixels. I combine the official training and test splits because these experiments are qualitative and do not estimate held-out performance.

def get_mnist_dataset(transform: T.Compose | None = None) -> Dataset:
    from torchvision.datasets import MNIST

    root = str(DATASET_PATH)
    trainset = MNIST(root=root, train=True, download=True, transform=transform)
    testset = MNIST(root=root, train=False, download=True, transform=transform)
    # Combine train and test dataset for more samples.
    dataset = ConcatDataset([trainset, testset])
    return dataset
NOISE_DIM = 100
transform = T.Compose([T.ToTensor(), T.Normalize(0.5, 0.5)])

dataset = get_mnist_dataset(transform=transform)
dataloader = DataLoader(dataset, **loader_kwargs)

# set seed for random generators
set_random_seed(seed=SEED)

# benchmark_noise is used for the animation to show how output evolve on the same vector
benchmark_noise = torch.randn(16 * 16, NOISE_DIM, device=device)

generator = Generator(out_dim=IMG_DIM, nz=NOISE_DIM).to(device)
generator.apply(weights_init)

discriminator = Discriminator(input_dim=IMG_DIM).to(device)
discriminator.apply(weights_init)

optimizer_G = optim.AdamW(
    generator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

optimizer_D = optim.AdamW(
    discriminator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

criterion = nn.BCEWithLogitsLoss().to(device)
animation: list[np.ndarray] = []

g_losses: list[float] = []
d_losses: list[float] = []

for _ in tqdm(range(N_EPOCHS), unit="epochs"):
    generator.train()
    discriminator.train()

    for samples_real, _ in dataloader:
        g_loss, d_loss = train_step(
            generator,
            discriminator,
            optimizer_G,
            optimizer_D,
            criterion,
            samples_real,
            NOISE_DIM,
            device,
        )

        g_losses.append(g_loss)
        d_losses.append(d_loss)

    generator.eval()
    with torch.inference_mode():
        images = generator(benchmark_noise)
        images = images.cpu()

        images = make_grid(images, nrow=16, normalize=True)
        images = images.permute(1, 2, 0).contiguous().numpy()

        animation.append(images)
100%|██████████| 100/100 [03:16<00:00,  1.96s/epochs]

Generator and Discriminator loss evolution over epochs using Vanilla GAN on the MNIST digit dataset.

Both losses start noisy, then settle into a stable band within the first 10,000 batches and stay there for the rest of training, without either one collapsing to zero.

Fashion-MNIST Dataset

To see how the same setup behaves beyond handwritten digits, I repeated the run on Fashion-MNIST. It contains 70,000 grayscale 28x28 images from 10 clothing categories, including shirts, trousers, dresses, coats, and footwear.

NOISE_DIM: int = 100
def get_mnist_fashion_dataset(transform: T.Compose | None = None) -> Dataset:
    from torchvision.datasets import FashionMNIST

    root = str(DATASET_PATH)
    trainset = FashionMNIST(root=root, train=True, download=True, transform=transform)
    testset = FashionMNIST(root=root, train=False, download=True, transform=transform)
    # Combine train and test dataset for more samples.
    dataset = ConcatDataset([trainset, testset])
    return dataset
transform = T.Compose([T.ToTensor(), T.Normalize(0.5, 0.5)])

data = get_mnist_fashion_dataset(transform=transform)
dataloader = DataLoader(data, **loader_kwargs)

# set seed for random generators
set_random_seed(seed=SEED)

# benchmark_noise is used for the animation to show how output evolve on same vector
benchmark_noise = torch.randn(16 * 16, NOISE_DIM, device=device)

generator = Generator(out_dim=IMG_DIM, nz=NOISE_DIM).to(device)
generator.apply(weights_init)

discriminator = Discriminator(input_dim=IMG_DIM).to(device)
discriminator.apply(weights_init)

optimizer_G = optim.AdamW(
    generator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

optimizer_D = optim.AdamW(
    discriminator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

criterion = nn.BCEWithLogitsLoss().to(device)
animation = []

g_losses, d_losses = [], []
for _ in tqdm(range(N_EPOCHS), unit="epochs"):
    generator.train()
    discriminator.train()

    for samples_real, _ in dataloader:
        g_loss, d_loss = train_step(
            generator,
            discriminator,
            optimizer_G,
            optimizer_D,
            criterion,
            samples_real,
            NOISE_DIM,
            device,
        )

        g_losses.append(g_loss)
        d_losses.append(d_loss)

    generator.eval()
    with torch.inference_mode():
        images = generator(benchmark_noise)
        images = images.cpu()

        images = make_grid(images, nrow=16, normalize=True)
        images = images.permute(1, 2, 0).contiguous().numpy()

        animation.append(images)
100%|██████████| 100/100 [03:19<00:00,  1.99s/epochs]

Generator and Discriminator loss evolution over epochs using Vanilla GAN on Fashion-MNIST.

The pattern looks similar to the digits run: an early noisy phase, then a stable band for the remainder of training.

DCGAN

The Deep Convolutional Generative Adversarial Network (DCGAN) replaces dense hidden layers with strided convolutions and transposed convolutions, which preserve the grid structure of image data (Radford et al. 2016). The architecture also uses batch normalization, except at the generator output and discriminator input, with ReLU activations in the generator and leaky ReLU activations in the discriminator.

In the single runs below, the DCGAN loss curves are noisier than the vanilla GAN curves, with recurring spikes rather than a stable band. Loss alone is an unreliable proxy for sample quality, so I also inspected the generated samples. The DCGAN samples looked sharper to me, but that observation is qualitative and does not establish that the architecture is generally better.

Setting Up DCGANs

The setup mirrors the vanilla GAN: same device selection, same weight initialization helper, same MNIST and Fashion-MNIST datasets. What changes is the Generator and Discriminator themselves, now built from convolutional and transposed-convolutional layers instead of dense layers.

Generator

The DCGAN Generator replaces dense layers with a stack of transposed convolutions, upsampling the noise vector step by step into a full image instead of reshaping a single flat output.

  • Input: Random noise, reshaped to a \(1 \times 1\) spatial vector.
  • Architecture: Transposed convolutions with batch normalization and ReLU activations, each layer roughly doubling the spatial resolution.
  • Output: A full image, using “tanh” for activation, same as the vanilla GAN.
  • Objective: Same as the vanilla GAN: generate data indistinguishable from real by the Discriminator.
class Generator(nn.Module):
    def __init__(self, nz: int = 100, ngf: int = 32, nc: int = 1):
        """
        :param nz: size of the latent z vector
        :param ngf: size of feature maps in generator
        :param nc: number of channels in the training images.
        """
        super().__init__()
        self.layers = nn.Sequential(
            nn.ConvTranspose2d(nz, 4 * ngf, 4, 1, 0, bias=False),
            nn.BatchNorm2d(4 * ngf),
            nn.ReLU(inplace=True),
            nn.ConvTranspose2d(4 * ngf, 2 * ngf, 3, 2, 1, bias=False),
            nn.BatchNorm2d(2 * ngf),
            nn.ReLU(inplace=True),
            nn.ConvTranspose2d(2 * ngf, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(inplace=True),
            nn.ConvTranspose2d(ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh(),
        )

    def forward(self, x: Tensor) -> Tensor:
        x = torch.reshape(x, (x.size(0), -1, 1, 1))
        return self.layers(x)


summary(Generator(), input_size=(128, 100))
==========================================================================================
Layer (type:depth-idx)                   Output Shape              Param #
==========================================================================================
Generator                                [128, 1, 28, 28]          --
├─Sequential: 1-1                        [128, 1, 28, 28]          --
│    └─ConvTranspose2d: 2-1              [128, 128, 4, 4]          204,800
│    └─BatchNorm2d: 2-2                  [128, 128, 4, 4]          256
│    └─ReLU: 2-3                         [128, 128, 4, 4]          --
│    └─ConvTranspose2d: 2-4              [128, 64, 7, 7]           73,728
│    └─BatchNorm2d: 2-5                  [128, 64, 7, 7]           128
│    └─ReLU: 2-6                         [128, 64, 7, 7]           --
│    └─ConvTranspose2d: 2-7              [128, 32, 14, 14]         32,768
│    └─BatchNorm2d: 2-8                  [128, 32, 14, 14]         64
│    └─ReLU: 2-9                         [128, 32, 14, 14]         --
│    └─ConvTranspose2d: 2-10             [128, 1, 28, 28]          512
│    └─Tanh: 2-11                        [128, 1, 28, 28]          --
==========================================================================================
Total params: 312,256
Trainable params: 312,256
Non-trainable params: 0
Total mult-adds (Units.GIGABYTES): 1.76
==========================================================================================
Input size (MB): 0.05
Forward/backward pass size (MB): 24.26
Params size (MB): 1.25
Estimated Total Size (MB): 25.56
==========================================================================================

Discriminator

The DCGAN Discriminator mirrors the Generator’s structure in reverse: convolutions instead of transposed convolutions, progressively downsampling the image into a single score.

  • Input: Either a real image or one produced by the Generator.
  • Architecture: Strided convolutions with batch normalization and leaky ReLU activations, each layer roughly halving the spatial resolution.
  • Output: A raw logit, evaluated by the same BCEWithLogitsLoss as the vanilla GAN.
  • Objective: Same as the vanilla GAN: recognize real data and identify fake data from the Generator.
class Discriminator(nn.Module):
    def __init__(self, ndf: int = 32, nc: int = 1, alpha: float = 0.2):
        super().__init__()
        self.layers = nn.Sequential(
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Conv2d(ndf, 2 * ndf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(2 * ndf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Conv2d(2 * ndf, 4 * ndf, 3, 2, 1, bias=False),
            nn.BatchNorm2d(4 * ndf),
            nn.LeakyReLU(alpha, inplace=True),
            nn.Conv2d(4 * ndf, 1, 4, 1, 0, bias=False),
        )

    def forward(self, x: Tensor) -> Tensor:
        x = self.layers(x)
        x = torch.reshape(x, (x.size(0), -1))
        return x


summary(Discriminator(), input_size=(BATCH_SIZE, 1, 28, 28))

Experiments

The learning rate, optimizer betas, batch size, number of epochs, latent dimension, and data loader settings match the vanilla GAN runs. The model architecture and capacity differ by design. Because each result comes from one seeded run and the sample comparison is visual, the observations below describe these runs rather than isolate a general architecture effect.

MNIST Digits Dataset

Same dataset as the vanilla GAN run: 70,000 grayscale handwritten digit images, 28x28 pixels each.

NOISE_DIM = 100

transform = T.Compose(
    [
        T.ToTensor(),
        T.Normalize(0.5, 0.5),
    ]
)

data = get_mnist_dataset(transform)
dataloader = DataLoader(data, **loader_kwargs)

# set seed for random generators
set_random_seed()

# benchmark_noise is used for the animation to show how output evolve on same vector
benchmark_noise = torch.randn(16 * 16, NOISE_DIM, device=device)

generator = Generator(nz=NOISE_DIM, ngf=32, nc=IMG_DIM[0]).to(device)
generator.apply(weights_init)

discriminator = Discriminator(ndf=32, nc=IMG_DIM[0]).to(device)
discriminator.apply(weights_init)

optimizer_G = optim.AdamW(
    generator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

optimizer_D = optim.AdamW(
    discriminator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

criterion = nn.BCEWithLogitsLoss().to(device)
animation = []

g_losses, d_losses = [], []
for _ in tqdm(range(N_EPOCHS), unit="epochs"):
    generator.train()
    discriminator.train()

    for samples_real, _ in dataloader:
        g_loss, d_loss = train_step(
            generator,
            discriminator,
            optimizer_G,
            optimizer_D,
            criterion,
            samples_real,
            NOISE_DIM,
            device,
        )

        g_losses.append(g_loss)
        d_losses.append(d_loss)

    generator.eval()
    with torch.inference_mode():
        images = generator(benchmark_noise)
        images = images.cpu()

        images = make_grid(images, nrow=16, normalize=True)
        images = images.permute(1, 2, 0).contiguous().numpy()

        animation.append(images)
100%|██████████| 100/100 [03:46<00:00,  2.26s/epochs]

Generator and Discriminator loss evolution over epochs using DCGAN on the MNIST digit dataset.

Unlike the vanilla GAN, the generator loss keeps spiking throughout the entire run instead of settling into a stable band, even though the generated samples in the video below still look reasonable.

Fashion-MNIST Dataset

I use the same Fashion-MNIST dataset as before to see whether the subjectively sharper samples from the digit run also appear on a harder dataset.

NOISE_DIM = 100

transform = T.Compose(
    [
        T.ToTensor(),
        T.Normalize(0.5, 0.5),
    ]
)

data = get_mnist_fashion_dataset(transform)
dataloader = DataLoader(data, **loader_kwargs)

# set seed for random generators
set_random_seed()

# benchmark_noise is used for the animation to show how output evolve on same vector
benchmark_noise = torch.randn(16 * 16, NOISE_DIM, device=device)

generator = Generator(nz=NOISE_DIM, ngf=32, nc=IMG_DIM[0]).to(device)
generator.apply(weights_init)

discriminator = Discriminator(ndf=32, nc=IMG_DIM[0]).to(device)
discriminator.apply(weights_init)

optimizer_G = optim.AdamW(
    generator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

optimizer_D = optim.AdamW(
    discriminator.parameters(),
    lr=OPTIMIZER_LR,
    betas=OPTIMIZER_BETAS,
    weight_decay=L2_NORM,
    fused=True,
)

criterion = nn.BCEWithLogitsLoss().to(device)
animation = []

g_losses, d_losses = [], []
for _ in tqdm(range(N_EPOCHS), unit="epochs"):
    generator.train()
    discriminator.train()

    for samples_real, _ in dataloader:
        g_loss, d_loss = train_step(
            generator,
            discriminator,
            optimizer_G,
            optimizer_D,
            criterion,
            samples_real,
            NOISE_DIM,
            device,
        )

        g_losses.append(g_loss)
        d_losses.append(d_loss)

    generator.eval()
    with torch.inference_mode():
        images = generator(benchmark_noise)
        images = images.cpu()

        images = make_grid(images, nrow=16, normalize=True)
        images = images.permute(1, 2, 0).contiguous().numpy()

        animation.append(images)
100%|██████████| 100/100 [03:46<00:00,  2.27s/epochs]

Generator and Discriminator loss evolution over epochs using DCGAN on Fashion-MNIST.

Same pattern as the digits run, and if anything, the spikes grow larger later in training rather than settling down.

Conclusion

A GAN trains a generator and discriminator with competing objectives. The theoretical equilibrium explains the target behavior, but alternating gradient updates do not guarantee that training will reach it. The loss curves in these runs show that distinction clearly: the dense GAN settled into a relatively stable band, while the DCGAN remained noisy and produced recurring spikes.

The DCGAN samples looked sharper to me despite their less stable losses, which reinforces a practical lesson: GAN losses are useful diagnostics but not direct measures of sample quality. This is a qualitative result from one seed, not evidence that DCGAN is always better. A stronger comparison would repeat each run across several seeds and evaluate sample quality and diversity with quantitative metrics.

The main implementation lesson is to keep the mathematical value function separate from the loss used in code. Here, the discriminator minimizes binary cross-entropy and the generator uses the non-saturating objective for stronger gradients.

References

Goodfellow, Ian J., Jean Pouget-Abadie, Mehdi Mirza, et al. 2014. “Generative Adversarial Nets.” Advances in Neural Information Processing Systems 27.
Radford, Alec, Luke Metz, and Soumith Chintala. 2016. “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.” International Conference on Learning Representations.

Reuse