AlexNet

AlexNet is a deep convolutional neural network developed at the University of Toronto by Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton. Entered in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012 under the team name SuperVision, it produced a winning top-5 error rate of 15.3 per cent against 26.2 per cent for the runner-up — a margin without precedent in the competition's history. The result is widely treated as the starting point of the modern deep learning era, after which convolutional networks moved from a niche approach to the default method in computer vision.^[1]

History

Through the 2000s, image recognition was dominated by hand-engineered features such as SIFT and HOG paired with linear classifiers, and many researchers believed convolutional networks could not be trained at the scale ImageNet demanded — roughly 1.2 million high-resolution images across 1,000 categories. Krizhevsky, then a doctoral student, re-implemented a convolutional network in CUDA so it could run on consumer graphics processors, and tuned the design throughout 2012.^[2]

The team submitted its entry on 30 September 2012. The gap between AlexNet and every classical machine-learning entry was so large that it reordered the field almost immediately: within months, major technology companies were building dedicated deep-learning research groups. In early 2013 the three authors founded DNNresearch, a company holding the AlexNet code and models, which Google acquired later that year; Hinton and Sutskever subsequently joined Google.

Architecture and techniques

AlexNet processes 224 by 224 pixel colour images through five convolutional layers followed by three fully connected layers, with 60 million parameters in total. Training used two NVIDIA GTX 580 GPUs with 3 GB of memory each, with the network split across the pair in a model-parallel arrangement — a design constraint that shaped the architecture, including a split in the middle of the network's feature layers. Full training on the ImageNet data set took several days.

The paper introduced or popularised a set of techniques that became standard practice in deep learning:

• ReLU nonlinearity in place of saturating activations such as tanh, which accelerated training.
• Dropout in the fully connected layers to reduce co-adaptation of neurons, in one of its first large-scale validations.
• Data augmentation, including horizontal flips, random cropping and colour jitter derived from principal components of the training pixels, to expand the effective size of the data set.
• Overlapping pooling and local response normalisation, which the authors argued helped generalisation.

Impact on artificial intelligence

AlexNet cut ImageNet top-5 error from 26.2 per cent to 15.3 per cent in a single year — a larger one-year improvement than the preceding decade of incremental progress had delivered. Successive architectures such as VGGNet, GoogLeNet and ResNet pushed the figure below 4 per cent by 2015, a level commonly compared with human performance on the same task, and convolutional networks went on to underpin object detection, medical imaging, autonomous driving and most modern image-generation systems. The work also established the practical habits of the field: training on GPUs, using ReLU and dropout as defaults, and treating benchmark leaderboards as evidence of progress.^[3]

>See Also

• ImageNet
• Convolutional Neural Network
• Computer Vision
• Deep Learning
• Residual Network