← Back to Table of Contents

Chapter 17 β€” CNN Fundamentals

β€œThe convolutional neural network is the inductive bias that made deep learning work for images β€” and understanding it is understanding the field’s foundation.”

The Convolution Operation

Planned content: 2D convolution: filter slides over input, element-wise multiply + sum. Input [B, C_in, H, W], filter [C_out, C_in, k_h, k_w], output [B, C_out, H', W']. Padding: valid (no pad), same (pad to maintain size). Stride: step size. Output size formula: H' = (H + 2P - k_h) / S + 1. Bias term. Parameter count: C_out * C_in * k_h * k_w + C_out.

πŸ“Š Planned diagram: 2D convolution animation β€” single filter sliding over a 5Γ—5 feature map with stride 1, showing element-wise multiply-accumulate.

πŸ“Š Planned diagram: Output spatial size as a function of input size, kernel size, padding, and stride.

\[H' = \left\lfloor \frac{H + 2P - k_h}{S} \right\rfloor + 1, \qquad W' = \left\lfloor \frac{W + 2P - k_w}{S} \right\rfloor + 1\]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import torch
import torch.nn as nn

# Standard 2D convolution
conv = nn.Conv2d(
    in_channels=64,
    out_channels=128,
    kernel_size=3,
    stride=1,
    padding=1,
    bias=True
)
# Parameters: 128 * 64 * 3 * 3 + 128 = 73,856

x = torch.randn(8, 64, 56, 56)   # [B=8, C_in=64, H=56, W=56]
y = conv(x)                        # [B=8, C_out=128, H=56, W=56]  (same padding)

Receptive Field

Planned content: The receptive field of a unit: which input pixels influence its output. Grows with depth. Stack of 3Γ—3 convs: RF after L layers = 2L + 1. A 7Γ—7 conv = 3 stacked 3Γ—3 convs but 3Γ— more parameters. Effective receptive field vs. theoretical RF. Why deep narrow networks > shallow wide networks for CV.

πŸ“Š Planned diagram: Receptive field growth β€” visualization of which input pixels affect a single neuron at depth 1, 2, 3 with 3Γ—3 conv.

Depth-wise and Separable Convolutions

Planned content: Standard conv: C_out * C_in * k * k params. Depthwise separable: factorize into depthwise (C_in groups, C_in * k * k) + pointwise (C_out * C_in * 1 * 1). ~8-9Γ— fewer parameters and FLOPs for 3Γ—3 conv. MobileNet’s core building block.

πŸ“Š Planned diagram: Depthwise separable convolution decomposition β€” standard vs. DW+PW, with parameter counts compared.

1
2
3
4
5
6
7
# Depthwise separable convolution
dw = nn.Conv2d(64, 64, 3, padding=1, groups=64)   # depthwise: groups=C_in
pw = nn.Conv2d(64, 128, 1)                         # pointwise: 1Γ—1 conv

x = torch.randn(8, 64, 56, 56)   # [8, 64, 56, 56]
x = dw(x)                          # [8, 64, 56, 56]  depthwise
x = pw(x)                          # [8, 128, 56, 56] pointwise

Pooling Strategies

Planned content: Max pooling: take max in each window. Average pooling. Global average pooling (GAP): [B, C, H, W] β†’ [B, C, 1, 1] β†’ squeeze β†’ [B, C] β€” replaces FC layers. Why GAP over fully-connected reduces parameters and enables variable input sizes. Adaptive pooling.

πŸ“Š Planned diagram: Global average pooling β€” [B, C, H, W] β†’ spatial mean per channel β†’ [B, C] β€” showing how it aggregates spatial information.

Normalization Layers

Planned content: Batch Normalization (BN): normalize over [B, H, W] per channel β†’ stable training, fast convergence. Learnable Ξ³ and Ξ². Training vs. inference behavior (running stats). Layer Normalization (LN): normalize over [C, H, W] per sample β†’ better for transformers. Instance Normalization (IN): normalize over [H, W] per channel per sample β†’ style transfer. Group Normalization (GN): normalize over groups of channels β†’ small batch safe.

πŸ“Š Planned diagram: Four normalization methods β€” which axes are normalized for each, illustrated as a 4D tensor [B, C, H, W] with colored normalization regions.

Activation Functions

Planned content: ReLU: simple, dead neuron problem. Leaky ReLU: small negative slope. PReLU: learned slope. ELU, SELU. GELU: smooth approximation, used in transformers. SiLU/Swish: self-gated, used in EfficientNet, Swin. Mish. Hard variants (HardSwish, HardSigmoid) for mobile.

πŸ“Š Planned diagram: Activation function comparison curves β€” ReLU, GELU, SiLU, Mish on the same axes.

Dilated (Atrous) Convolutions

Planned content: (Covered in depth in Ch. 4 for segmentation context) β€” theoretical foundation here: dilation rate r inserts r-1 zeros between filter elements. Effective kernel size: k_eff = k + (k-1)(r-1). Maintains spatial resolution while expanding receptive field. Gridding artifact for large dilation rates.

Inductive Biases of CNNs

Planned content: Translation equivariance (not invariance). Weight sharing across spatial locations. Local connectivity. Spatial hierarchy. Why these biases are beneficial for natural images but restrictive for long-range dependencies. Comparison with ViT’s weaker inductive biases (more data required).

πŸ“Š Planned diagram: Translation equivariance visualization β€” same object at different positions produces same features (shifted).

Next: Chapter 18 β€” Classification Architectures β†’


Last updated: May 2026