Chapter 6 β Object Tracking
βDetection is per frame. Tracking is across time β maintaining identity through motion, occlusion, and crowding.β
The Tracking Problem
Planned content: Multi-Object Tracking (MOT) problem formulation. Input: sequence of frames. Output: per-frame set of tracks, each with consistent ID, bounding box, and class. SOT (single object) vs. MOT (multiple objects). Online vs. offline tracking. Detection-based tracking (tracking-by-detection) vs. joint detection+tracking.
π Planned diagram: MOT problem β sequence of frames with bounding boxes, showing track continuity through occlusion and re-identification.
The Tracking-by-Detection Paradigm
Planned content: Step 1: detect objects in every frame. Step 2: associate detections across frames. The two-step pipeline: detector β association module. Strengths: modular, benefits from strong detectors. Weakness: detector failures propagate. SORT, DeepSORT, ByteTrack all follow this paradigm.
π Planned diagram (flowchart): Tracking-by-detection pipeline β frame t β detector β detections β association with tracks from t-1 β updated tracks β output.
Kalman Filter for Motion Prediction
Planned content: State vector
[cx, cy, s, r, vc_x, vc_y, vs](centroid, area scale, aspect ratio, velocities). Predict step: propagate state with constant velocity model. Update step: correct with new measurement. Covariance matrices (Q for process noise, R for measurement noise). When to predict vs. update. Why Kalman filter is fast and effective despite being linear.
π Planned diagram: Kalman filter cycle β state prediction β measurement β update, showing uncertainty ellipses before and after measurement.
| $$\hat{x}_{t | t-1} = F \hat{x}_{t-1 | t-1}, \qquad P_{t | t-1} = F P_{t-1 | t-1} F^T + Q$$ |
| $$K_t = P_{t | t-1} H^T (H P_{t | t-1} H^T + R)^{-1}$$ | Β | Β |
| $$\hat{x}_{t | t} = \hat{x}_{t | t-1} + K_t (z_t - H \hat{x}_{t | t-1})$$ | Β |
β Hungarian Algorithm β Deep Dive
Planned content: The association problem: given M tracks and N detections, find the optimal matching that minimizes total cost (e.g., 1 - IoU). This is the classic Linear Assignment Problem (LAP). Hungarian algorithm: find minimum-cost perfect matching in a bipartite graph. Complexity O(nΒ³). How cost matrices are built. Threshold-based acceptance: only accept matches below a cost threshold. Unmatched detections β new tracks. Unmatched tracks β lost state.
π Planned diagram (flowchart): Hungarian algorithm step-by-step β cost matrix construction β row/column reduction β find zero coverage β assignment extraction.
π Planned diagram: Bipartite matching visualization β tracks (left nodes) matched to detections (right nodes) with cost-weighted edges. Before and after assignment.
π Planned diagram: Cost matrix example with IoU values β showing how the matrix is built and what the optimal assignment looks like.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from scipy.optimize import linear_sum_assignment
import numpy as np
def associate(tracks, detections, iou_threshold=0.3):
"""
tracks: List of track boxes [M, 4]
detections: List of detection boxes [N, 4]
returns: (matches, unmatched_tracks, unmatched_detections)
"""
if len(tracks) == 0 or len(detections) == 0:
return [], list(range(len(tracks))), list(range(len(detections)))
iou_matrix = compute_iou_matrix(tracks, detections) # [M, N]
cost_matrix = 1 - iou_matrix # [M, N]
# Hungarian algorithm: find optimal assignment
row_ind, col_ind = linear_sum_assignment(cost_matrix) # [K], [K]
matches, unmatched_t, unmatched_d = [], [], []
# Filter out weak matches
for r, c in zip(row_ind, col_ind):
if iou_matrix[r, c] >= iou_threshold:
matches.append((r, c))
else:
unmatched_t.append(r)
unmatched_d.append(c)
unmatched_t += [i for i in range(len(tracks)) if i not in row_ind]
unmatched_d += [j for j in range(len(detections)) if j not in col_ind]
return matches, unmatched_t, unmatched_d
Track Lifecycle
Planned content: Track states: Tentative (newly created, not yet confirmed) β Active (confirmed, being matched) β Lost (missed for max_age frames) β Deleted. Minimum hits to confirm a track. Maximum age before deletion. Birth, confirmation, occlusion handling, re-identification.
π Planned diagram (state machine): Track lifecycle state machine β Tentative β Active β Lost β Deleted, with transition conditions.
Appearance Features and Re-ID
Planned content: Pure motion (IoU) fails under occlusion and fast motion. Appearance embeddings: re-ID network extracts a feature vector per detection crop. Cosine similarity in embedding space for long-range matching. Gallery management: rolling average of appearance features per track. DeepSORT: adds appearance cost to Kalman IoU cost.
π Planned diagram: Re-ID pipeline β detection crop
[1, 3, 128, 64]β Re-ID backbone β embedding[1, 512]β cosine similarity with track gallery.
SORT, DeepSORT, ByteTrack β A Comparison
Planned content: SORT: Kalman + Hungarian on IoU only. Simple and fast. DeepSORT: add appearance feature matching. ByteTrack: match ALL detections (even low-confidence) β two-round matching prevents missed detections. StrongSORT: improved Re-ID + motion model. OC-SORT: observation-centric Kalman. BoTrack. Analysis of where each method improves.
π Planned table: SORT family comparison β method, detector used, matching strategy, whether Re-ID is used, MOT17 HOTA score.
Tracking Evaluation Metrics
Planned content: MOTA (Multiple Object Tracking Accuracy) β penalizes FP, FN, ID switches. MOTP (precision). IDF1 (identity F1 score). HOTA (Higher Order Tracking Accuracy) β decomposes into detection and association quality. How HOTA resolves disagreements between MOTA and IDF1.
π Planned diagram: HOTA decomposition β DetA vs. AssA, HOTA = geometric mean.
Next: Chapter 7 β Pose Estimation β
Last updated: May 2026