πŸ‘οΈ Computer Vision β€” The Complete Guide

From Pixels to Predictions: Understanding, building, and deploying ML-based computer vision systems. A comprehensive reference for practitioners, engineers, and researchers navigating the modern CV landscape.

Who This Is For

You understand Python and have touched PyTorch or TensorFlow. You’ve used a pre-trained model, maybe fine-tuned one. But you want to truly understand what is happening inside: how a convolutional filter fires, why NMS is tricky, how ArcFace creates separable embedding spaces, how YOLO’s anchor-free head works, and what makes a transformer see an image differently than a CNN. This guide takes you from first principles β€” images as tensors β€” all the way to foundation models, generative systems, and the research frontier.

πŸ“‹ Table of Contents

# Chapter What You’ll Learn
Overview & Foundations Β  Β 
0 Grand Overview The full CV landscape β€” tasks, history, benchmarks, and how every piece connects
1 Image Fundamentals Images as tensors [B, C, H, W], color spaces, preprocessing pipelines, camera models
Use Cases Β  Β 
2 Image Classification Softmax, cross-entropy, fine-grained recognition, zero-shot classification, ImageNet
3 Object Detection Two-stage vs one-stage, anchors, NMS deep dive, IoU variants, mAP
4 Semantic Segmentation Encoder-decoder, dilated convolutions, ASPP, loss functions, mIoU
5 Instance Segmentation Mask R-CNN, panoptic segmentation, SAM, query-based methods
6 Object Tracking Kalman filter, Hungarian algorithm deep dive, Re-ID, SORT β†’ ByteTrack
7 Pose Estimation Keypoints, heatmaps, PAF deep dive, top-down vs bottom-up, 3D pose
8 Face Recognition Detection, alignment, embedding, ArcFace/CosFace/SphereFace deep dive
9 Depth Estimation Stereo, monocular depth, LiDAR fusion, depth completion
10 Optical Flow Lucas-Kanade, FlowNet, RAFT, video stabilization
11 Video Understanding Action recognition, temporal models (3D conv, transformers), action localization
12 3D Vision Point clouds, NeRF, 3D Gaussian splatting, 3D object detection, SLAM
13 Anomaly Detection One-class learning, PatchCore, flow-based, reconstruction-based
14 OCR & Document Understanding Text detection, CRNN+CTC, end-to-end OCR, LayoutLM
15 Medical Imaging Segmentation, classification, domain challenges, foundation models (MedSAM)
16 Multimodal Vision Tasks VQA, image captioning, visual grounding, text-to-image generation
Architectures Β  Β 
17 CNN Fundamentals Convolution math, receptive fields, normalization, depthwise & dilated convs
18 Classification Architectures ResNet, MobileNet, EfficientNet, ViT, Swin, ConvNeXt β€” with param/FLOP tables
19 Detection Architectures YOLO (v1β†’v11), Faster R-CNN, DETR, RT-DETR, FCOS, CenterNet
20 Segmentation Architectures FCN, U-Net, DeepLab v3+, Mask R-CNN, Mask2Former, SAM
21 Pose Architectures Stacked Hourglass, HRNet, ViTPose, OpenPose, DEKR, WholeBody
22 Face Architectures MTCNN, RetinaFace, MobileFaceNet, ArcFace, AdaFace
23 Tracking Architectures SORT, DeepSORT, FairMOT, ByteTrack, StrongSORT, TrackFormer
24 Vision Transformers ViT, DeiT, Swin, MAE, DINO, DINOv2 β€” attention in 2D
25 Multimodal Architectures CLIP, BLIP-2, LLaVA, Grounding DINO, SAM 2, Flamingo
Optimization Β  Β 
26 Optimization Overview The accuracy-speed-size triangle, profiling, optimization taxonomy
27 Quantization PTQ, QAT, INT8/FP8, mixed precision, per-channel vs per-tensor
28 Pruning & Sparsification Structured/unstructured pruning, lottery ticket, movement pruning, ASP
29 Knowledge Distillation Feature mimicking, attention transfer, task-specific KD, self-distillation
30 Neural Architecture Search RL/evolutionary/gradient-based NAS, DARTS, Once-for-All, EfficientNet
31 Efficient Inference Graph fusion, ONNX, TensorRT, OpenVINO, TFLite, CoreML, NCNN
Data & Training Β  Β 
32 Datasets ImageNet, COCO, ADE20K, LFW, Kinetics, KITTI, MVTec β€” every major benchmark
33 Data Augmentation MixUp, CutMix, Mosaic, AutoAugment, Copy-Paste, TTA
34 Training Recipes Loss functions, AdamW/LARS/SAM optimizers, LR schedules, EMA, bag-of-tricks
35 Self-Supervised Learning SimCLR, MoCo, BYOL, MAE, DINO β€” learning without labels
Deployment & Ecosystem Β  Β 
36 Deployment ONNX export, serving (Triton/TorchServe), edge (Jetson/mobile), batching
37 Ecosystem & Libraries timm, MMDetection, Detectron2, Ultralytics, OpenCV, Hugging Face, Roboflow
38 Evaluation & Benchmarks mAP computation, OKS, HOTA, FID, Rank-1, FLOP counting
Frontier Β  Β 
39 Vision–Language Models CLIP ecosystem, GPT-4V, Gemini Vision, VQA, grounding, generation
40 Foundation Models SAM, DINOv2, Florence-2, InternVL β€” universal visual features
41 Generative Vision GANs, VAEs, DDPM, Stable Diffusion, ControlNet, video generation
42 The Frontier World models, 3DGS, embodied AI, neuromorphic vision, open problems
Appendices Β  Β 
A History of Computer Vision Full timeline from Hubel & Wiesel to SAM 2 β€” every milestone and paradigm shift
B Math Foundations Linear algebra, convolutions as matrix ops, Fourier, probability for CV practitioners
C Glossary Every key term, acronym, and metric defined in one place

πŸ—ΊοΈ Learning Paths

Recommended Learning Path
πŸ‘οΈ Ch 0–1: Grand Overview & Image Fundamentals
🎯 Ch 2–16: Use Cases β€” The Full Task Taxonomy
πŸ—οΈ Ch 17–25: Architectures β€” CNN to Multimodal
⚑ Ch 26–31: Optimization β€” Speed, Size, Efficiency
πŸ“¦ Ch 32–35: Data, Augmentation & Self-Supervised
πŸš€ Ch 36–38: Deployment, Ecosystem & Evaluation
πŸ”¬ Ch 39–42: Frontier β€” VLMs, Foundation Models, Generation

Path A: β€œI want to build a CV application” (8 chapters)

  1. 00 β€” Grand Overview β€” the big picture
  2. 01 β€” Image Fundamentals β€” images as tensors
  3. 02 β€” Image Classification β€” the simplest task
  4. 03 β€” Object Detection β€” the most deployed task
  5. 18 β€” Classification Architectures β€” choosing your backbone
  6. 19 β€” Detection Architectures β€” YOLO and friends
  7. 33 β€” Data Augmentation β€” training data strategy
  8. 36 β€” Deployment β€” getting to production

Path B: β€œI want to go deep on a specific task” (task chapter β†’ architecture chapter β†’ eval chapter)

  • Detection: 03 β†’ 19 β†’ 38
  • Segmentation: 04 + 05 β†’ 20 β†’ 38
  • Face Recognition: 08 β†’ 22 β†’ 38
  • Tracking: 06 β†’ 23 β†’ 38
  • Pose Estimation: 07 β†’ 21 β†’ 38

Path C: β€œI want to optimize for edge/mobile” (7 chapters)

  1. 17 β€” CNN Fundamentals β€” understand what you’re optimizing
  2. 18 β€” Classification Architectures β€” MobileNet, EfficientNet
  3. 26 β€” Optimization Overview β€” the landscape
  4. 27 β€” Quantization β€” INT8 and beyond
  5. 28 β€” Pruning β€” structured pruning
  6. 29 β€” Knowledge Distillation β€” compress with teachers
  7. 31 β€” Efficient Inference β€” TensorRT, TFLite, CoreML

Path D: β€œI want to understand vision transformers and foundation models” (6 chapters)

  1. 24 β€” Vision Transformers β€” ViT to DINOv2
  2. 25 β€” Multimodal Architectures β€” CLIP, BLIP-2, LLaVA
  3. 35 β€” Self-Supervised Learning β€” MAE, DINO
  4. 39 β€” Vision–Language Models β€” CLIP ecosystem
  5. 40 β€” Foundation Models β€” SAM, DINOv2
  6. 41 β€” Generative Vision β€” diffusion models

Path E: β€œI want the research frontier” (full guide)

Read chapters 0 through 42 in order. Appendices A, B, and C provide depth on history, math, and terminology.

πŸ“š Prerequisites

  • Python β€” NumPy, basic PyTorch tensor operations
  • Linear algebra β€” matrix multiplication, eigenvectors, SVD
  • ML basics β€” loss functions, gradient descent, backpropagation, CNNs
  • Probability β€” softmax, cross-entropy, Gaussian distributions
  • Command line β€” terminal, pip/conda, CUDA setup

🧭 How This Guide Is Organized

This guide spans six dimensions of computer vision:

Six Dimensions of Computer Vision
🎯
Use Cases
Classification, detection, segmentation, tracking, pose, face, depth, flow, video, 3D, anomaly, OCR, medical, multimodal
πŸ—οΈ
Architectures
CNNs, ResNets, MobileNets, EfficientNets, ViT, Swin, YOLO, DETR, HRNet, MobileFaceNet, CLIP, BLIP-2
⚑
Optimization
Quantization (QAT/PTQ), pruning, sparsification, knowledge distillation, NAS, efficient inference runtimes
πŸ“¦
Data & Training
Benchmark datasets, augmentation strategies, loss functions, optimizers, self-supervised pre-training
πŸš€
Deployment
ONNX, TensorRT, OpenVINO, TFLite, Triton, edge (Jetson / mobile), serving patterns
πŸ”¬
Frontier
VLMs, foundation models, generative vision, 3D scene representations, embodied AI, open problems

πŸ“ Changelog

Date Changes
May 2026 Initial structure β€” 43 chapters + 3 appendices

Last updated: May 2026