Chapter 41 β Generative Vision
βUnderstanding vision at its deepest means being able to generate it. Generative models close the loop between perception and creation.β
The Generative Vision Landscape
Planned content: Three families: GANs (adversarial), VAEs (variational), Diffusion (score-based). Each models the data distribution differently. Recent convergence around diffusion models. Applications: text-to-image, image editing, super-resolution, inpainting, video generation, 3D generation, synthetic training data.
π Planned diagram: Generative model taxonomy β GAN (discriminator-generator adversarial) vs. VAE (encoder-decoder probabilistic) vs. Diffusion (iterative denoising) β conceptual comparison.
GANs: Generative Adversarial Networks
Planned content: Minimax game: generator G creates fake images, discriminator D distinguishes real/fake. Objective:
min_G max_D E[log D(x)] + E[log(1 - D(G(z)))]. Training instability (mode collapse, vanishing gradients). DCGAN: convolutional architecture. Progressive GAN: growing resolution during training. StyleGAN2/3: style-based generator, high-quality portraits. BigGAN: class-conditional at ImageNet scale.
π Planned diagram: GAN training loop β noise z β G β fake image β D competes with real images β gradient back to both G and D.
π Planned diagram: StyleGAN2 architecture β mapping network (z β w) + synthesis network (constant input β styled convolutions β image), with AdaIN/modulation.
VAEs: Variational Autoencoders
Planned content: Encoder:
q(z|x)β ΞΌ, Ο. Reparameterization trick:z = ΞΌ + Ο * Ξ΅. Decoder:p(x|z). ELBO loss: reconstruction + KL divergence. VQ-VAE: discrete latent codes via vector quantization. VQ-VAE-2: hierarchical latent codes for high resolution. Used as perceptual compression in latent diffusion.
π Planned diagram: VAE architecture β encoder
[B, 3, H, W]β ΞΌ[B, D]+ log Ο[B, D]β sample z[B, D]β decoder β reconstruction[B, 3, H, W]. KL loss on latent distribution.
DDPM: Denoising Diffusion Probabilistic Models
Planned content: Forward process: gradually add Gaussian noise over T=1000 steps.
q(x_t | x_{t-1}) = N(x_t; β(1-Ξ²_t) x_{t-1}, Ξ²_t I). Closed form:q(x_t | x_0) = N(x_t; βαΎ±_t x_0, (1-αΎ±_t) I). Reverse process: denoise step by step. Network Ξ΅_ΞΈ predicts the noise at each step. Training: MSE on predicted vs. actual noise. Sampling: T denoising steps. Connection to score matching.
\[\mathcal{L}_{DDPM} = E_{t, x_0, \epsilon}\left[\|\epsilon - \epsilon_\theta(\sqrt{\bar\alpha_t} x_0 + \sqrt{1-\bar\alpha_t}\epsilon, t)\|^2\right]\]π Planned diagram: DDPM forward and reverse process β x_0 (clean) β x_1 β β¦ β x_T (pure noise) forward, and reverse denoising x_T β β¦ β x_0 with U-Net noise predictor.
DDIM: Faster Sampling
Planned content: DDIM: deterministic non-Markovian reverse process. Skip T β 50 steps while maintaining quality. PLMS, DPM-Solver: even faster samplers. The quality-speed tradeoff in sampling steps. CFG (Classifier-Free Guidance):
Ξ΅_guided = Ξ΅_uncond + w * (Ξ΅_cond - Ξ΅_uncond). Guidance scale w.
π Planned diagram: CFG mechanism β conditional (text-guided) noise prediction pushed further from unconditional prediction, amplifying text alignment.
Stable Diffusion: Latent Diffusion Models
Planned content: Key insight: diffusion in latent space (4Γ smaller) instead of pixel space. VQ-VAE encoder:
[B, 3, 512, 512]β[B, 4, 64, 64]latent. U-Net diffusion in latent space. CLIP text encoder for text conditioning via cross-attention. Decoder:[B, 4, 64, 64]β[B, 3, 512, 512]. SDXL: two text encoders (CLIP-L + OpenCLIP-G), larger U-Net. SD3: DiT-based (transformer backbone).
π Planned diagram: Stable Diffusion architecture β CLIP text encoder β text embeddings β U-Net (with cross-attention on text) operating on latent
[B, 4, 64, 64]β VAE decoder β image.
ControlNet: Conditioned Generation
Planned content: Add spatial conditioning (pose, depth, canny, segmentation map, normal map) to Stable Diffusion. ControlNet architecture: copies SD encoder weights, connects via zero-conv layers. Training only the ControlNet copy. Enables precise control over generated image structure.
π Planned diagram: ControlNet β condition image (pose/depth/edge) + text prompt β modified U-Net with ControlNet branch β conditioned generation.
Video Generation
Planned content: Make-A-Video, Imagen Video, CogVideoX. Sora: diffusion transformer (DiT) for video. Challenges: temporal consistency, long duration. Factored space-time attention. Training data: long video + text captions.
Evaluating Generative Models
Planned content: FID: lower = better, measures distributional distance. IS: higher = better (quality + diversity). Precision/Recall/Density/Coverage. LPIPS: perceptual similarity. CLIP-Score for text-image alignment. Human preference studies (ELO, A/B tests).
Next: Chapter 42 β The Frontier β
Last updated: May 2026