Chapter 30 β Neural Architecture Search
βIf human intuition designed ResNet and EfficientNet, imagine what an algorithm optimizing for your specific hardware and dataset could find.β
Manual vs. Automated Architecture Design
Planned content: Manual design: domain experts iterating on intuitions (VGG β ResNet β DenseNet). Time-consuming, biased toward human-intuitive structures. NAS: define a search space, search strategy, and performance estimator β automate the architecture design process.
π Planned diagram: NAS framework β search space (building blocks, connections) + search strategy (controller, evolutionary, gradient) + performance estimation (training, proxy) β optimal architecture.
Search Strategies
Planned content: Reinforcement Learning (NASNet, MnasNet): RNN controller generates architectures, reward = validation accuracy. Slow (GPU-years). Evolutionary (AmoebaNet): population of architectures, mutate and select best. DARTS: Differentiable Architecture Search β relax discrete choices to continuous (weighted sum of operations), optimize with gradient. One-shot NAS: train a supernet, derive subnetworks by weight sharing.
π Planned diagram: DARTS mixed operation β edge in search graph = weighted sum of candidate operations (conv3Γ3, conv5Γ5, skip, etc.), weights learned by gradient.
DARTS: Differentiable Architecture Search
\[\bar{o}^{(i,j)}(x) = \sum_{o \in \mathcal{O}} \frac{\exp(\alpha_o^{(i,j)})}{\sum_{o'} \exp(\alpha_{o'}^{(i,j)})} \cdot o(x)\]Planned content: Continuous relaxation: replace discrete architecture choices with softmax over operations. Bilevel optimization: alternately update architecture weights Ξ± (on validation set) and operation weights W (on training set). Derive final architecture by argmax of Ξ±. Limitations: instability, skip connection dominance.
MobileNetV3 and Platform-Aware NAS
Planned content: MnasNet: optimize directly for on-device latency Γ accuracy. Look-up table for layer latency on target hardware. ProxylessNAS: directly optimize on target hardware with latency measured. Platform-aware: same NAS run on GPU vs. mobile gives different architectures.
EfficientNet: NAS + Compound Scaling
Planned content: (Architecture covered in Ch. 18) β NAS process here: MnasNet-style NAS to find EfficientNet-B0 baseline. Then compound scaling to derive B1-B7. The scaling coefficients Ξ±, Ξ², Ξ³ found by grid search.
Once-For-All: Train Once, Deploy Anywhere
Planned content: Train a single supernet that supports many subnetworks of varying depth/width/kernel size/resolution. Progressive shrinking training: gradually reduce subnetwork size. At deployment: look up accuracy-latency table for your hardware, pick the right subnetwork. No retraining needed.
π Planned diagram: Once-for-All supernet β single training produces a family of models at different efficiency points, deployed to phone/edge/server by choosing subnetwork.
NAS for Detection and Segmentation
Planned content: SpineNet: NAS backbone for detection with cross-scale connections. DetNAS: NAS for one-stage detector. SqueezeNAS for segmentation. Auto-DeepLab: NAS for semantic segmentation.
Next: Chapter 31 β Efficient Inference β
Last updated: May 2026