← Back to Table of Contents

Chapter 36 β€” Deployment

β€œA model that isn’t deployed doesn’t exist. Deployment is where ML meets reality.”

Model Export Pipeline

Planned content: PyTorch β†’ TorchScript (graph capture via tracing or scripting) β†’ ONNX β†’ TensorRT/.tflite/.mlmodel. Common gotchas: dynamic shapes, custom ops, control flow. Testing exported model vs. PyTorch model numerically.

πŸ“Š Planned diagram (flowchart): Model export decision tree β€” server GPU β†’ TensorRT; Intel CPU β†’ OpenVINO; Mobile β†’ TFLite (Android) / CoreML (iOS); Jetson β†’ TensorRT; FPGA β†’ custom.

Serving Infrastructure

Planned content: Triton Inference Server (NVIDIA): multi-model server, backends (TensorRT, ONNX, PyTorch, TF), dynamic batching, streaming. TorchServe: PyTorch-native model serving. FastAPI + uvicorn for custom REST endpoints. gRPC for high-throughput service. Model ensembles (preprocessing + model + postprocessing as pipeline).

πŸ“Š Planned diagram: Triton serving architecture β€” client request β†’ Triton server β†’ backend (TensorRT engine) β†’ response. Dynamic batching accumulation illustration.

Edge Deployment

Planned content: Jetson Orin: NVIDIA Xavier + GPU + DLA, TensorRT optimized, 275 TOPS. Raspberry Pi 4 + Coral TPU: EdgeTPU INT8. Qualcomm NPU: Snapdragon AI Engine, SNPE SDK. Rockchip RK3588: RKNN toolkit. Apple M-series ANE via CoreML. Power consumption vs. throughput tradeoffs.

πŸ“Š Planned diagram: Edge device comparison β€” TOPS, memory bandwidth, power (W), supported precisions, SDK ecosystem.

Latency vs. Throughput

Planned content: Latency: time for single request. Throughput: requests per second at sustained load. Little’s Law: throughput = N/latency (N = concurrent requests). Batch size effect: latency increases with batch, throughput increases and saturates. SLA-based optimization: optimize throughput subject to p99 latency < X ms.

πŸ“Š Planned diagram: Latency-throughput tradeoff curve β€” showing how batching affects both, with optimal operating point for different SLAs.

Preprocessing and Postprocessing in Pipeline

Planned content: Don’t neglect preprocessing latency: image decode (JPEG) + resize + normalize can be 30-50% of total latency. NVJPEG for GPU-accelerated JPEG decode. TensorRT preprocessing via CUDA kernels. Postprocessing (NMS, keypoint decode) also needs optimization.

Model Versioning and A/B Testing

Planned content: Model registry (MLflow, W&B Artifacts). Versioning strategy. Shadow mode: run new model in parallel, compare outputs. Canary deployment: 5% traffic β†’ validate β†’ 100%. Feature flags. Rollback procedures. Monitoring in production.

Monitoring in Production

Planned content: Data drift: input distribution shifts over time (new camera, lighting changes, season). Concept drift: label distribution changes. Performance monitoring: latency, throughput, error rate. Accuracy monitoring: human spot-check or auto-evaluation. Alerting and automated retraining triggers.

πŸ“Š Planned diagram: Production monitoring dashboard β€” input feature distribution over time, model confidence distribution, latency p50/p95/p99.

Next: Chapter 37 β€” Ecosystem & Libraries β†’


Last updated: May 2026