Autoencoder

An autoencoder is a neural network architecture that learns to copy its input to its output by passing the data through a constrained intermediate layer known as the latent space or bottleneck. Because the network cannot simply memorise the identity function when the latent representation is smaller than the input, training forces it to discover compact, informative features. Autoencoders are unsupervised models in the sense that no labels are required; the input itself serves as the target during training.

The architecture is composed of two functions trained jointly. The encoder maps an input vector x to a latent vector z, typically through a stack of fully connected or convolutional layers with progressively smaller dimensions. The decoder maps z back to a reconstruction x'. Training minimises a reconstruction loss, most commonly mean squared error for continuous data or binary cross-entropy for binary or normalised inputs. Backpropagation adjusts both encoder and decoder parameters simultaneously so that the latent representation captures the structure necessary to rebuild the input as accurately as possible.

Variants

Several variants extend the basic formulation to address specific weaknesses. A denoising autoencoder is trained to reconstruct a clean input from a corrupted version, forcing the encoder to learn features that are robust to noise. A sparse autoencoder adds a penalty on the activations of the hidden layer so that only a small number of units are active for any given input, mimicking the sparse coding behaviour observed in biological neurons. A contractive autoencoder penalises the Frobenius norm of the encoder Jacobian, making the latent representation insensitive to small changes in the input.

The variational autoencoder (VAE) is a probabilistic extension in which the encoder outputs the parameters of a distribution rather than a deterministic vector. Sampling from this distribution and applying the decoder yields a generative model from which new data can be synthesised. VAEs are trained by maximising a variational lower bound on the data likelihood, combining a reconstruction term with a Kullback-Leibler divergence that regularises the latent distribution toward a prior, typically a standard normal.

Applications

In dimensionality reduction, autoencoders provide a non-linear alternative to principal component analysis. The latent representation can be used as a compact feature vector for downstream classifiers, search systems, or clustering algorithms. In anomaly detection, an autoencoder is trained on normal examples; samples that produce high reconstruction error at inference time are flagged as anomalies, a technique widely deployed in fraud monitoring, predictive maintenance, and network intrusion detection.

Image and audio compression benefit from convolutional autoencoders that exploit the spatial or temporal structure of the data. In recommender systems, autoencoders model user-item interaction matrices, predicting missing entries from learned latent factors. In drug discovery and chemistry, autoencoders learn continuous representations of molecular graphs that can be optimised for desired properties.

Comparison with other models

| Model | Output | Training signal | Typical use | |---|---|---|---| | Autoencoder | Reconstruction | Reconstruction loss | Compression, features | | VAE | Probabilistic sample | ELBO | Generation, smooth latent | | GAN | Generated sample | Adversarial loss | Photorealistic generation | | Diffusion | Iteratively denoised sample | Noise prediction loss | High-fidelity generation |

While modern generative models such as diffusion and large transformer-based decoders have surpassed classical autoencoders on photorealistic generation, autoencoders remain widely used as components within larger systems, notably as the perceptual compression stage of latent diffusion models such as Stable Diffusion.

References

1. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533-536.
2. Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P. A. (2008). Extracting and composing robust features with denoising autoencoders. ICML.
3. Kingma, D. P., and Welling, M. (2013). Auto-Encoding Variational Bayes. arXiv:1312.6114.
4. Bank Negara Malaysia. (2023). Risk Management in Technology (RMiT) Policy Document. https://www.bnm.gov.my.