Neural Architecture Search
Neural architecture search is the automated design of neural network architectures using search algorithms, reinforcement learning, or gradient-based methods to discover models that meet target accuracy, latency, and size constraints.
Neural architecture search (NAS) is the automated process of designing neural network topologies by searching over a defined space of candidate architectures and evaluating them against an objective that typically combines accuracy with hardware-aware constraints such as latency, memory footprint, and energy. NAS sits inside the broader field of automated machine learning (AutoML) and has produced several state-of-the-art image classification, object detection, and on-device models.
Search components
Every NAS system is characterised by three design choices.
The search space defines the set of architectures the algorithm may consider. Macro spaces select entire network topologies; cell-based spaces fix the outer skeleton and search only for a repeated building block (the "cell") that is then stacked. Hardware-aware spaces include layers and operators with known latency profiles for a target device.
The search strategy decides how to explore the space. Reinforcement learning approaches treat architecture choice as a sequence of actions taken by an agent rewarded with validation accuracy. Evolutionary algorithms mutate and recombine populations of architectures. Gradient-based approaches such as DARTS relax the discrete choice over operators into a continuous mixture and optimise it jointly with weights. Bayesian optimisation and random search remain strong baselines.
The evaluation strategy determines how each candidate is scored. Naive evaluation — training each candidate from scratch — is prohibitively expensive. Weight sharing, supernet training, performance predictors, low-fidelity proxies (smaller datasets, fewer epochs), and zero-cost proxies based on initialisation statistics have all been developed to reduce evaluation cost.
Landmark systems
NASNet (Zoph and Le, 2017) used reinforcement learning to discover cells later stacked into competitive ImageNet models. AmoebaNet applied evolutionary search to similar effect. ENAS introduced weight sharing across the search, slashing compute requirements. DARTS made the entire search differentiable and brought NAS within reach of single-GPU users. EfficientNet, while not strictly searched, combined an architecture discovered by NAS with a principled compound scaling rule that became influential across vision. MnasNet and ProxylessNAS targeted mobile latency directly. Once-for-All trained a single supernet from which many specialised subnetworks could be sliced without retraining.
Hardware-aware NAS
Modern NAS rarely optimises pure accuracy. Hardware-aware variants incorporate the latency or energy of candidate architectures measured on real devices — phones, edge accelerators, embedded GPUs — into the search objective. This has produced families of models with smooth accuracy–latency Pareto frontiers, allowing engineers to pick the best architecture for a deployment target without retraining.
Limitations
NAS has well-known shortcomings. Search remains computationally costly compared with manual design. Reported gains over strong baselines are sometimes small, and several studies have shown that random search within a well-designed space rivals more elaborate strategies. The transferability of discovered architectures across tasks and datasets varies. As a result, manual design — informed by NAS results — continues to dominate frontier model development, particularly in the transformer family.
Current trends
Recent work focuses on three directions. Foundation-model NAS searches over transformer hyperparameters such as depth, width, attention head count, and tokenisation choices for language and vision. Multi-objective NAS jointly optimises accuracy, latency, memory, and energy across a fleet of target devices. Green NAS explicitly measures and reduces the carbon footprint of the search procedure itself. Open-source frameworks including NNI (Microsoft), Auto-Keras, AutoGluon (AWS), and Google's Vertex AI Vizier expose NAS to practitioners without requiring custom infrastructure.
| System | Year | Search method | Notable result | |---|---|---|---| | NASNet | 2017 | Reinforcement learning | Strong ImageNet accuracy | | AmoebaNet | 2018 | Evolutionary | Matched NASNet at less compute | | DARTS | 2018 | Gradient-based | Single-GPU search feasibility | | EfficientNet | 2019 | Compound scaling + search | Pareto-leading vision family | | MnasNet | 2019 | Mobile-aware RL | Latency-targeted mobile model | | Once-for-All | 2020 | Supernet slicing | Many subnets without retraining |
References
- Zoph, B. and Le, Q. (2017). Neural Architecture Search with Reinforcement Learning. ICLR.
- Real, E. et al. (2019). Regularized Evolution for Image Classifier Architecture Search. AAAI.
- Liu, H., Simonyan, K., Yang, Y. (2019). DARTS: Differentiable Architecture Search. ICLR.
- Tan, M. and Le, Q. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML.
- Cai, H. et al. (2020). Once-for-All: Train One Network and Specialize it for Efficient Deployment. ICLR.