Chapter 36 β Deployment
βA model that isnβt deployed doesnβt exist. Deployment is where ML meets reality.β
Model Export Pipeline
Planned content: PyTorch β TorchScript (graph capture via tracing or scripting) β ONNX β TensorRT/.tflite/.mlmodel. Common gotchas: dynamic shapes, custom ops, control flow. Testing exported model vs. PyTorch model numerically.
π Planned diagram (flowchart): Model export decision tree β server GPU β TensorRT; Intel CPU β OpenVINO; Mobile β TFLite (Android) / CoreML (iOS); Jetson β TensorRT; FPGA β custom.
Serving Infrastructure
Planned content: Triton Inference Server (NVIDIA): multi-model server, backends (TensorRT, ONNX, PyTorch, TF), dynamic batching, streaming. TorchServe: PyTorch-native model serving. FastAPI + uvicorn for custom REST endpoints. gRPC for high-throughput service. Model ensembles (preprocessing + model + postprocessing as pipeline).
π Planned diagram: Triton serving architecture β client request β Triton server β backend (TensorRT engine) β response. Dynamic batching accumulation illustration.
Edge Deployment
Planned content: Jetson Orin: NVIDIA Xavier + GPU + DLA, TensorRT optimized, 275 TOPS. Raspberry Pi 4 + Coral TPU: EdgeTPU INT8. Qualcomm NPU: Snapdragon AI Engine, SNPE SDK. Rockchip RK3588: RKNN toolkit. Apple M-series ANE via CoreML. Power consumption vs. throughput tradeoffs.
π Planned diagram: Edge device comparison β TOPS, memory bandwidth, power (W), supported precisions, SDK ecosystem.
Latency vs. Throughput
Planned content: Latency: time for single request. Throughput: requests per second at sustained load. Littleβs Law: throughput = N/latency (N = concurrent requests). Batch size effect: latency increases with batch, throughput increases and saturates. SLA-based optimization: optimize throughput subject to p99 latency < X ms.
π Planned diagram: Latency-throughput tradeoff curve β showing how batching affects both, with optimal operating point for different SLAs.
Preprocessing and Postprocessing in Pipeline
Planned content: Donβt neglect preprocessing latency: image decode (JPEG) + resize + normalize can be 30-50% of total latency. NVJPEG for GPU-accelerated JPEG decode. TensorRT preprocessing via CUDA kernels. Postprocessing (NMS, keypoint decode) also needs optimization.
Model Versioning and A/B Testing
Planned content: Model registry (MLflow, W&B Artifacts). Versioning strategy. Shadow mode: run new model in parallel, compare outputs. Canary deployment: 5% traffic β validate β 100%. Feature flags. Rollback procedures. Monitoring in production.
Monitoring in Production
Planned content: Data drift: input distribution shifts over time (new camera, lighting changes, season). Concept drift: label distribution changes. Performance monitoring: latency, throughput, error rate. Accuracy monitoring: human spot-check or auto-evaluation. Alerting and automated retraining triggers.
π Planned diagram: Production monitoring dashboard β input feature distribution over time, model confidence distribution, latency p50/p95/p99.
Next: Chapter 37 β Ecosystem & Libraries β
Last updated: May 2026