← Back to Table of Contents

Chapter 23 β€” Tracking Architectures

β€œTracking architectures encode assumptions about motion, appearance, and identity β€” choosing the wrong one costs IDs.”

SORT: Simple Online and Realtime Tracking (2016)

Planned content: Kalman filter for motion prediction + Hungarian algorithm for IoU-based assignment. No appearance features. 7-line core algorithm. Surprisingly competitive on pedestrian tracking. Limitations: ID switches under occlusion. FPS: 260 Hz (detector not included).

πŸ“Š Planned diagram: SORT flow β€” predict Kalman states β†’ detect β†’ compute IoU matrix β†’ Hungarian assignment β†’ update states β†’ manage track lifecycle.

DeepSORT (2017)

Planned content: Add appearance descriptor (Re-ID network) to SORT. Two-distance metric: IoU distance + cosine distance on appearance embeddings. Cascade matching: prefer recently-seen tracks. Track state: confirmed vs. tentative. 20Hz with deep features.

πŸ“Š Planned diagram: DeepSORT metric fusion β€” IoU distance matrix [M, N] + appearance cosine distance [M, N] β†’ element-wise gating β†’ cascade assignment.

FairMOT (2020)

Planned content: Joint detection and re-ID in a single network. CenterNet detector head + re-ID embedding head. One-shot tracker. Feature at detected center = re-ID embedding. Fairer competition between detection and re-ID gradients.

πŸ“Š Planned diagram: FairMOT head β€” shared DLA backbone β†’ detection branch (heatmap + offset + size) + re-ID branch (128D embedding per center).

ByteTrack: Every Detection Counts (2022)

Planned content: Key insight: low-confidence detections (0.1-0.5) still carry useful identity information β€” use them in a second matching round. Two-round matching: high-confidence detections first β†’ unmatch tracks β†’ low-confidence detections second. Dramatically reduces missed detections (FN β†’ ID switches).

πŸ“Š Planned diagram (flowchart): ByteTrack two-round matching β€” high-conf detections + all tracks β†’ Hungarian round 1 β†’ remaining tracks + low-conf detections β†’ Hungarian round 2 β†’ new tracks from unmatched high-conf.

StrongSORT (2022)

Planned content: Improved re-ID model (OSNet) + EMA (exponential moving average) for appearance features + motion compensation (camera motion). AFLink: inter-frame non-linear link prediction to fill gaps. GSSI: Gaussian-smoothed state interpolation.

OC-SORT: Observation-Centric SORT (2022)

Planned content: Problem with standard Kalman: when a track is lost (occluded), virtual observations cause drift in state estimation. OC-SORT: re-initialize Kalman with actual observations when track is recovered. Virtual observations only for motion compensation.

πŸ“Š Planned diagram: OC-SORT observation-centric re-correction β€” showing Kalman state drift during occlusion vs. observation-anchored correction.

Transformer-Based Trackers

Planned content: TrackFormer: extend DETR with track queries (propagated object queries from t-1). MOTR: end-to-end multi-object tracking with motion-aware transformer. Advantages: no explicit IoU/appearance matching, learned association. Challenges: slower, hard to debug.

πŸ“Š Planned diagram: TrackFormer β€” DETR object queries for new detections + track queries (re-used from t-1) β†’ joint update β†’ maintained object identity.

Metrics Deep Dive

Planned content: MOTA = 1 - (FN + FP + IDSW) / GT. IDF1 = 2 * IDTP / (2*IDTP + IDFP + IDFN). HOTA = √(DetA Γ— AssA). Why HOTA is more balanced. MOT17, MOT20, DanceTrack, SportsMOT benchmarks.

πŸ“Š Planned diagram: HOTA decomposition visualization β€” DetA vs AssA space, showing where different trackers fall.

Next: Chapter 24 β€” Vision Transformers β†’


Last updated: May 2026