Neural Architecture Search

Neural architecture search (NAS) is the automated process of designing neural network topologies by searching over a defined space of candidate architectures and evaluating them against an objective that typically combines accuracy with hardware-aware constraints such as latency, memory footprint, and energy. NAS sits inside the broader field of automated machine learning (AutoML) and has produced several state-of-the-art image classification, object detection, and on-device models.

Search components

Every NAS system is characterised by three design choices.

The search space defines the set of architectures the algorithm may consider. Macro spaces select entire network topologies; cell-based spaces fix the outer skeleton and search only for a repeated building block (the "cell") that is then stacked. Hardware-aware spaces include layers and operators with known latency profiles for a target device.

The search strategy decides how to explore the space. Reinforcement learning approaches treat architecture choice as a sequence of actions taken by an agent rewarded with validation accuracy. Evolutionary algorithms mutate and recombine populations of architectures. Gradient-based approaches such as DARTS relax the discrete choice over operators into a continuous mixture and optimise it jointly with weights. Bayesian optimisation and random search remain strong baselines.

The evaluation strategy determines how each candidate is scored. Naive evaluation — training each candidate from scratch — is prohibitively expensive. Weight sharing, supernet training, performance predictors, low-fidelity proxies (smaller datasets, fewer epochs), and zero-cost proxies based on initialisation statistics have all been developed to reduce evaluation cost.

Landmark systems

NASNet (Zoph and Le, 2017) used reinforcement learning to discover cells later stacked into competitive ImageNet models. AmoebaNet applied evolutionary search to similar effect. ENAS introduced weight sharing across the search, slashing compute requirements. DARTS made the entire search differentiable and brought NAS within reach of single-GPU users. EfficientNet, while not strictly searched, combined an architecture discovered by NAS with a principled compound scaling rule that became influential across vision. MnasNet and ProxylessNAS targeted mobile latency directly. Once-for-All trained a single supernet from which many specialised subnetworks could be sliced without retraining.

Hardware-aware NAS

Modern NAS rarely optimises pure accuracy. Hardware-aware variants incorporate the latency or energy of candidate architectures measured on real devices — phones, edge accelerators, embedded GPUs — into the search objective. This has produced families of models with smooth accuracy–latency Pareto frontiers, allowing engineers to pick the best architecture for a deployment target without retraining.

Limitations

NAS has well-known shortcomings. Search remains computationally costly compared with manual design. Reported gains over strong baselines are sometimes small, and several studies have shown that random search within a well-designed space rivals more elaborate strategies. The transferability of discovered architectures across tasks and datasets varies. As a result, manual design — informed by NAS results — continues to dominate frontier model development, particularly in the transformer family.

Current trends

Recent work focuses on three directions. Foundation-model NAS searches over transformer hyperparameters such as depth, width, attention head count, and tokenisation choices for language and vision. Multi-objective NAS jointly optimises accuracy, latency, memory, and energy across a fleet of target devices. Green NAS explicitly measures and reduces the carbon footprint of the search procedure itself. Open-source frameworks including NNI (Microsoft), Auto-Keras, AutoGluon (AWS), and Google's Vertex AI Vizier expose NAS to practitioners without requiring custom infrastructure.

| System | Year | Search method | Notable result | |---|---|---|---| | NASNet | 2017 | Reinforcement learning | Strong ImageNet accuracy | | AmoebaNet | 2018 | Evolutionary | Matched NASNet at less compute | | DARTS | 2018 | Gradient-based | Single-GPU search feasibility | | EfficientNet | 2019 | Compound scaling + search | Pareto-leading vision family | | MnasNet | 2019 | Mobile-aware RL | Latency-targeted mobile model | | Once-for-All | 2020 | Supernet slicing | Many subnets without retraining |

NAS is used selectively in Malaysia, primarily by R&D teams that need on-device or latency-bounded models. Local system integrators delivering vision inspection — ViTrox, Pentamaster, Greatech, Aemulus — typically rely on NAS-derived backbones (EfficientNet, MobileNet, MnasNet) selected from public model zoos rather than running NAS in-house, because the engineering and compute investment is large.

Academic groups conducting NAS research include Universiti Malaya, Universiti Sains Malaysia, Universiti Teknologi Malaysia, and Universiti Kebangsaan Malaysia. Research collaborations through MIMOS Berhad, MDEC's AI Sandbox, and MOSTI-funded programmes provide compute access. HRD Corp-funded AutoML training delivered by Microsoft, AWS, Google, and local providers introduces engineers to NAS concepts even where in-house search is not feasible.

For Malaysian enterprises, the practical pathway to NAS benefits runs through cloud AutoML services: Google Vertex AI, Amazon SageMaker Autopilot, and Azure Machine Learning all expose hosted NAS or related architecture-tuning capabilities. PDPA requirements and BNM's outsourcing guidance shape the choice between cloud-hosted search and on-premises pipelines, especially for banks such as Maybank, CIMB, Hong Leong, and Public Bank, and for Petronas Digital, TM, Maxis, CelcomDigi, and AirAsia's AI teams.

1. Zoph, B. and Le, Q. (2017). Neural Architecture Search with Reinforcement Learning. ICLR.
2. Real, E. et al. (2019). Regularized Evolution for Image Classifier Architecture Search. AAAI.
3. Liu, H., Simonyan, K., Yang, Y. (2019). DARTS: Differentiable Architecture Search. ICLR.
4. Tan, M. and Le, Q. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML.
5. Cai, H. et al. (2020). Once-for-All: Train One Network and Specialize it for Efficient Deployment. ICLR.