Chapter 31 β Efficient Inference
βA model that runs in 100ms on GPU is useless on a Raspberry Pi. Efficient inference is about bridging training and deployment realities.β
Graph-Level Optimization
Planned content: Operator fusion: fuse Conv + BN + ReLU into a single kernel (eliminates intermediate memory writes). Constant folding: pre-compute BN parameters into conv weights. Dead code elimination. Layout optimization (NCHW β NHWC for ARM). Graph partitioning. Kernel auto-tuning (TVM, Halide).
π Planned diagram: Operator fusion β Conv β BN β ReLU without fusion (3 memory round-trips) vs. fused (1 kernel, 1 memory round-trip).
ONNX: Universal Model Exchange
Planned content: Open Neural Network Exchange: IR for ML models. Export from PyTorch/TensorFlow β optimize β deploy on any runtime. ONNX operators. Dynamic vs. static shapes. onnxruntime: optimized ONNX inference (CPU, GPU, Edge). ONNX simplifier.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import torch
import torch.onnx
model = ... # your trained PyTorch model
dummy_input = torch.randn(1, 3, 224, 224) # [B, C, H, W]
torch.onnx.export(
model,
dummy_input,
"model.onnx",
opset_version=17,
input_names=["input"], # name inputs
output_names=["output"], # name outputs
dynamic_axes={"input": {0: "batch_size"}, "output": {0: "batch_size"}}
)
TensorRT: NVIDIA GPU Optimization
Planned content: TensorRT: optimize ONNX models for NVIDIA GPUs. Layer fusion, kernel auto-selection, precision calibration (FP16, INT8). Build engine from ONNX β serialize as .plan file β deserialize and run. Builder API. ICalibrator for INT8. Layer-by-layer timing profiles. Triton Inference Server for serving.
π Planned diagram (flowchart): TensorRT workflow β ONNX model β TensorRT builder β optimized engine
.planβ runtime inference with tensors[B, C, H, W].
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import tensorrt as trt
import numpy as np
# Build TensorRT engine from ONNX
logger = trt.Logger(trt.Logger.WARNING)
builder = trt.Builder(logger)
network = builder.create_network(1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH))
parser = trt.OnnxParser(network, logger)
with open("model.onnx", "rb") as f:
parser.parse(f.read())
config = builder.create_builder_config()
config.set_flag(trt.BuilderFlag.FP16) # enable FP16
engine = builder.build_serialized_network(network, config)
OpenVINO: Intel Optimization
Planned content: Intelβs inference toolkit: optimize for Intel CPUs, iGPUs, Movidius VPU, FPGA. Model Optimizer: convert ONNX/TF/PyTorch to IR (.xml + .bin). Post-Training Optimization Tool (POT) for INT8. OpenVINO Runtime for deployment. NNCF for QAT-ready models.
TFLite and CoreML: Mobile Deployment
Planned content: TFLite: TensorFlow Lite for Android/iOS/embedded. XNNPACK backend. INT8 quantization with representative dataset. Flatbuffer format. CoreML: Appleβs on-device inference. ANE (Apple Neural Engine) acceleration. coremltools: convert PyTorch/ONNX β CoreML.
π Planned diagram: Mobile deployment pipeline β PyTorch β ONNX β TFLite/CoreML β Android/iOS device inference.
NCNN and MNN: Ultra-Lightweight Mobile
Planned content: NCNN (Tencent): ARM-optimized C++ inference library. No dependencies. Supports many operators. Widely used in mobile vision apps. MNN (Alibaba): multi-backend mobile inference. Vulkan GPU support.
Batching Strategies
Planned content: Static batching: fixed batch size B. Dynamic batching (Triton): accumulate requests up to max_batch or timeout. Continuous batching for LLMs. Optimal batch size for throughput vs. latency tradeoff. Memory bandwidth bound vs. compute bound transition point.
π Planned diagram: Throughput vs. latency vs. batch size curve β showing knee point where throughput saturates and latency inflects.
Benchmark: Latency vs. Accuracy on Common Platforms
Planned content: ResNet-50, MobileNetV3, EfficientNet-B0 on A100 GPU, Jetson Orin, iPhone 14 ANE, Raspberry Pi 4. Comparison of FP32, FP16, INT8. Tools: netron (visualization), trtexec (TensorRT benchmark), OpenVINO benchmark_app.
π Planned table: Model Γ platform Γ precision β latency (ms), throughput (img/s), accuracy drop (%).
Next: Chapter 32 β Datasets β
Last updated: May 2026