Overfitting

Overfitting occurs when a machine learning model captures not only the underlying pattern in its training data but also the random noise and idiosyncrasies particular to that sample. Such a model achieves very low error on the data it was trained on yet fails to generalise, producing markedly worse predictions when presented with new examples. Overfitting is one of the central obstacles in applied machine learning, and managing it is a routine part of model development.

How overfitting arises

A model has a certain capacity, loosely the richness of the relationships it can represent. When capacity is large relative to the amount and quality of training data, the model has enough flexibility to memorise individual data points rather than infer the general rule that produced them. A high-degree polynomial fitted to a handful of points, for example, can pass exactly through every point while oscillating wildly between them. The fit looks perfect on the training set and is useless for prediction.

Several conditions encourage overfitting: an overly complex model architecture, too few training examples, noisy or mislabelled data, training for too many iterations, and the presence of features that are only spuriously correlated with the target. Deep neural networks, with millions or billions of parameters, are especially prone to it unless deliberately constrained.

Detecting overfitting

The standard diagnostic is to compare performance on data the model was trained on against performance on a held-out validation or test set. A widening gap, where training accuracy keeps improving while validation accuracy stalls or declines, is the signature of overfitting. Plotting both curves against training time produces the familiar learning curve used to decide when to stop training.

Cross-validation, in which the data is repeatedly partitioned into training and validation folds, gives a more robust estimate of how well a model will generalise and reduces the chance of being misled by a single fortunate or unfortunate split.

The bias-variance tradeoff

Overfitting is best understood through the bias-variance decomposition of prediction error. A model that overfits has low bias but high variance: it is highly sensitive to the particular training sample, so small changes in the data produce large changes in the fitted model. Underfitting is the mirror image, with high bias and low variance. The practitioner seeks the middle ground where total error is minimised. This tradeoff frames most of the techniques used to control overfitting.

Preventing and reducing overfitting

A range of methods address overfitting. Gathering more representative training data is the most direct, since a richer sample makes memorisation harder and the true pattern more evident. Regularisation techniques such as L1 and L2 penalties discourage large parameter values and thereby simplify the learned function. In neural networks, dropout randomly disables units during training, while early stopping halts optimisation once validation performance ceases to improve. Reducing model size, pruning features, and using data augmentation to synthetically expand the training set are also widely applied. Ensemble methods such as bagging reduce variance by averaging many models.

The table below summarises common remedies and the mechanism by which each helps.

| Technique | Mechanism | | --- | --- | | More training data | Makes memorisation harder, clarifies signal | | L1 / L2 regularisation | Penalises complex parameter configurations | | Dropout | Prevents co-adaptation of neurons | | Early stopping | Stops before noise is fitted | | Data augmentation | Expands effective dataset size | | Cross-validation | Detects poor generalisation early |

Overfitting is a practical concern for Malaysian organisations deploying machine learning on relatively small or skewed local datasets. A fraud-detection model trained on a single bank's historical transactions, for instance, may overfit to patterns specific to that institution and degrade when applied across a broader customer base. Bank Negara Malaysia's guidance on the responsible use of analytics in financial services emphasises model validation and ongoing monitoring, both of which directly target overfitting and model drift.

The Malaysian AI talent pool, supported by training initiatives funded through HRD Corp and university programmes at Universiti Malaya and Universiti Teknologi Malaysia, increasingly covers sound model evaluation practice as a core competency. Public agencies under the MyDigital Blueprint that experiment with predictive analytics for healthcare or agriculture face the same risk, since locally collected datasets are often limited in size.

Vendors operating in the region, including local technology firms and multinationals offering managed AI platforms through MDEC-recognised digital hubs, typically build cross-validation and held-out testing into their delivery process. As Malaysia expands its data-centre capacity and domestic AI services, disciplined evaluation that guards against overfitting remains essential to trustworthy deployment in banking, telecommunications and the public sector.

1. Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
2. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning. MIT Press.
3. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.