← Back to Table of Contents

Chapter 20 β€” Segmentation Architectures

β€œFrom coarse features to pixel-perfect boundaries β€” the architecture is the bridge between semantic understanding and spatial precision.”

FCN: The Founding Architecture (2014)

Planned content: Fully Convolutional Network: replace FC layers with 1Γ—1 conv for dense prediction. Skip connections from pool3, pool4 to decoder. FCN-32s, FCN-16s, FCN-8s variants β€” improving spatial detail with each. Why FCN was revolutionary despite rough output.

πŸ“Š Planned diagram: FCN architecture β€” VGG backbone with FC layers replaced by 1Γ—1 conv, bilinear upsample, skip additions. FCN-32s vs. FCN-8s comparison.

U-Net: The Universal Segmentation Architecture (2015)

Planned content: Encoder-decoder with dense skip connections. Every encoder level connects directly to corresponding decoder level. Works extremely well with small datasets. Cropping in original (no padding). U-Net++: nested skip connections. UΒ²-Net: residual U-blocks. Attention U-Net.

πŸ“Š Planned diagram: U-Net architecture β€” full tensor shapes at every stage, symmetric structure, skip connections visualized. Left: encoder (contracting), right: decoder (expanding).

DeepLab Series (v1β†’v3+)

Planned content: DeepLab v1: atrous conv + CRF post-processing. V2: multi-scale ASPP. V3: improved ASPP + global pooling. V3+: decoder with low-level features from backbone for sharper boundaries. Xception backbone. Backbone-neck-head design.

πŸ“Š Planned diagram: DeepLabV3+ architecture β€” Xception encoder β†’ ASPP module β†’ decoder merging low-level features β†’ bilinear upsample β†’ output [B, C, H, W].

Mask R-CNN for Instance Segmentation

Planned content: (Covered in Ch. 5) β€” architecture deep dive here: FPN backbone, RPN, RoIAlign, mask head (FCN predicting [C, 28, 28] per RoI). Decoupling classification and segmentation (class-agnostic mask head works best).

Panoptic FPN

Planned content: Extend Mask R-CNN with a semantic segmentation branch. Share FPN backbone between instance (Mask R-CNN) and semantic (FCN) heads. Merge predictions: instances override stuff predictions. Panoptic quality evaluation.

Mask2Former: Universal Segmentation (2021)

Planned content: Unified architecture for semantic, instance, and panoptic segmentation. Pixel decoder (multi-scale FPN features) + transformer decoder (object queries). Masked cross-attention: each query only attends to its predicted foreground region. Top-K queries. Task-specific heads on top.

πŸ“Š Planned diagram: Mask2Former architecture β€” backbone β†’ pixel decoder (multi-scale features) β†’ transformer decoder with masked attention β†’ per-query class + mask predictions.

SAM: Segment Anything (2023)

Planned content: (Architecture details covered in Ch. 5) β€” training at scale: SA-1B dataset (1B masks). Interactive annotation engine. Zero-shot. SAM 2 for video with streaming memory.

OneFormer: One Transformer for All Segmentation

Planned content: Task-conditioned architecture: a task token ([semantic], [instance], [panoptic]) conditions all attention layers. Single model trained jointly on all three tasks. Outperforms specialized models in some settings.

SegFormer

Planned content: Hierarchical transformer encoder (Mix Transformer, MiT-B0 to B5) + lightweight MLP decoder. No positional encoding needed β€” interpolation at test time. Fast and accurate. NV-NVIDIA official HuggingFace model.

πŸ“Š Planned diagram: SegFormer architecture β€” MiT encoder producing 4 feature scales β†’ MLP decoder concatenating and projecting β†’ segmentation output.

Architecture Selection Guide

Planned content: When to use U-Net (medical, satellite, limited data), DeepLabV3+ (general semantic), Mask2Former (panoptic/instance), SegFormer (efficient transformer), SAM (promptable zero-shot).

πŸ“Š Planned table: Segmentation architecture comparison β€” task support (semantic/instance/panoptic), backbone, ADE20K mIoU, COCO panoptic PQ, parameters, speed.

Next: Chapter 21 β€” Pose Architectures β†’


Last updated: May 2026