← Back to Table of Contents

Chapter 6 β€” Object Tracking

β€œDetection is per frame. Tracking is across time β€” maintaining identity through motion, occlusion, and crowding.”

The Tracking Problem

Planned content: Multi-Object Tracking (MOT) problem formulation. Input: sequence of frames. Output: per-frame set of tracks, each with consistent ID, bounding box, and class. SOT (single object) vs. MOT (multiple objects). Online vs. offline tracking. Detection-based tracking (tracking-by-detection) vs. joint detection+tracking.

πŸ“Š Planned diagram: MOT problem β€” sequence of frames with bounding boxes, showing track continuity through occlusion and re-identification.

The Tracking-by-Detection Paradigm

Planned content: Step 1: detect objects in every frame. Step 2: associate detections across frames. The two-step pipeline: detector β†’ association module. Strengths: modular, benefits from strong detectors. Weakness: detector failures propagate. SORT, DeepSORT, ByteTrack all follow this paradigm.

πŸ“Š Planned diagram (flowchart): Tracking-by-detection pipeline β€” frame t β†’ detector β†’ detections β†’ association with tracks from t-1 β†’ updated tracks β†’ output.

Kalman Filter for Motion Prediction

Planned content: State vector [cx, cy, s, r, vc_x, vc_y, vs] (centroid, area scale, aspect ratio, velocities). Predict step: propagate state with constant velocity model. Update step: correct with new measurement. Covariance matrices (Q for process noise, R for measurement noise). When to predict vs. update. Why Kalman filter is fast and effective despite being linear.

πŸ“Š Planned diagram: Kalman filter cycle β€” state prediction β†’ measurement β†’ update, showing uncertainty ellipses before and after measurement.

$$\hat{x}_{t t-1} = F \hat{x}_{t-1 t-1}, \qquad P_{t t-1} = F P_{t-1 t-1} F^T + Q$$
$$K_t = P_{t t-1} H^T (H P_{t t-1} H^T + R)^{-1}$$ Β  Β 
$$\hat{x}_{t t} = \hat{x}_{t t-1} + K_t (z_t - H \hat{x}_{t t-1})$$ Β 

⭐ Hungarian Algorithm β€” Deep Dive

Planned content: The association problem: given M tracks and N detections, find the optimal matching that minimizes total cost (e.g., 1 - IoU). This is the classic Linear Assignment Problem (LAP). Hungarian algorithm: find minimum-cost perfect matching in a bipartite graph. Complexity O(nΒ³). How cost matrices are built. Threshold-based acceptance: only accept matches below a cost threshold. Unmatched detections β†’ new tracks. Unmatched tracks β†’ lost state.

πŸ“Š Planned diagram (flowchart): Hungarian algorithm step-by-step β€” cost matrix construction β†’ row/column reduction β†’ find zero coverage β†’ assignment extraction.

πŸ“Š Planned diagram: Bipartite matching visualization β€” tracks (left nodes) matched to detections (right nodes) with cost-weighted edges. Before and after assignment.

πŸ“Š Planned diagram: Cost matrix example with IoU values β€” showing how the matrix is built and what the optimal assignment looks like.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
from scipy.optimize import linear_sum_assignment
import numpy as np

def associate(tracks, detections, iou_threshold=0.3):
    """
    tracks:     List of track boxes [M, 4]
    detections: List of detection boxes [N, 4]
    returns: (matches, unmatched_tracks, unmatched_detections)
    """
    if len(tracks) == 0 or len(detections) == 0:
        return [], list(range(len(tracks))), list(range(len(detections)))

    iou_matrix = compute_iou_matrix(tracks, detections)  # [M, N]
    cost_matrix = 1 - iou_matrix                         # [M, N]

    # Hungarian algorithm: find optimal assignment
    row_ind, col_ind = linear_sum_assignment(cost_matrix)  # [K], [K]

    matches, unmatched_t, unmatched_d = [], [], []

    # Filter out weak matches
    for r, c in zip(row_ind, col_ind):
        if iou_matrix[r, c] >= iou_threshold:
            matches.append((r, c))
        else:
            unmatched_t.append(r)
            unmatched_d.append(c)

    unmatched_t += [i for i in range(len(tracks))     if i not in row_ind]
    unmatched_d += [j for j in range(len(detections)) if j not in col_ind]

    return matches, unmatched_t, unmatched_d

Track Lifecycle

Planned content: Track states: Tentative (newly created, not yet confirmed) β†’ Active (confirmed, being matched) β†’ Lost (missed for max_age frames) β†’ Deleted. Minimum hits to confirm a track. Maximum age before deletion. Birth, confirmation, occlusion handling, re-identification.

πŸ“Š Planned diagram (state machine): Track lifecycle state machine β€” Tentative β†’ Active β†’ Lost β†’ Deleted, with transition conditions.

Appearance Features and Re-ID

Planned content: Pure motion (IoU) fails under occlusion and fast motion. Appearance embeddings: re-ID network extracts a feature vector per detection crop. Cosine similarity in embedding space for long-range matching. Gallery management: rolling average of appearance features per track. DeepSORT: adds appearance cost to Kalman IoU cost.

πŸ“Š Planned diagram: Re-ID pipeline β€” detection crop [1, 3, 128, 64] β†’ Re-ID backbone β†’ embedding [1, 512] β†’ cosine similarity with track gallery.

SORT, DeepSORT, ByteTrack β€” A Comparison

Planned content: SORT: Kalman + Hungarian on IoU only. Simple and fast. DeepSORT: add appearance feature matching. ByteTrack: match ALL detections (even low-confidence) β€” two-round matching prevents missed detections. StrongSORT: improved Re-ID + motion model. OC-SORT: observation-centric Kalman. BoTrack. Analysis of where each method improves.

πŸ“Š Planned table: SORT family comparison β€” method, detector used, matching strategy, whether Re-ID is used, MOT17 HOTA score.

Tracking Evaluation Metrics

Planned content: MOTA (Multiple Object Tracking Accuracy) β€” penalizes FP, FN, ID switches. MOTP (precision). IDF1 (identity F1 score). HOTA (Higher Order Tracking Accuracy) β€” decomposes into detection and association quality. How HOTA resolves disagreements between MOTA and IDF1.

πŸ“Š Planned diagram: HOTA decomposition β€” DetA vs. AssA, HOTA = geometric mean.

Next: Chapter 7 β€” Pose Estimation β†’


Last updated: May 2026