Logistic Regression

Overview

Logistic regression is a supervised learning algorithm used to estimate the probability that an input belongs to a particular class. Despite its name it is a classification method rather than a regression method in the usual sense. It is one of the most widely used models in statistics, epidemiology, finance and machine learning because it is simple, fast to train, and produces interpretable, probabilistic outputs.

In its basic form, logistic regression handles binary classification, distinguishing between two outcomes such as fraud or legitimate, default or repay, disease or healthy. Extensions handle problems with more than two classes.

How it works

Logistic regression computes a weighted sum of the input features, the same linear combination used in linear regression, and then passes the result through the logistic or sigmoid function. The sigmoid maps any real number to a value between 0 and 1, which can be interpreted as a probability. In plain notation, the model computes z = w*x + b and then p = 1 / (1 + e^(-z)).

A decision threshold, commonly 0.5, converts the probability into a class label. The model parameters — the weights and the bias — are estimated from training data by maximising the likelihood of the observed labels, which is equivalent to minimising a loss called binary cross-entropy or log loss. Because this loss is convex, optimisation by gradient descent reliably finds the global optimum.

Multiclass extension

For problems with more than two classes, multinomial logistic regression, also called softmax regression, replaces the sigmoid with the softmax function to produce a probability distribution over all classes. An alternative is the one-versus-rest scheme, which trains a separate binary classifier for each class.

Regularisation and interpretation

To prevent overfitting, logistic regression is often trained with L1 or L2 regularisation, which penalise large weights. L1 regularisation can drive some weights to zero, performing implicit feature selection. A key strength of the model is interpretability: each weight indicates the direction and strength of a feature's influence on the outcome, and exponentiating a weight yields an odds ratio that domain experts can reason about.

Strengths and limitations

Logistic regression is computationally cheap, robust on linearly separable problems, and resistant to overfitting when regularised. Its main limitation is that it models a linear decision boundary in feature space, so it struggles with complex non-linear relationships unless features are engineered or transformed. For highly non-linear problems, tree ensembles or neural networks usually perform better, but logistic regression remains a strong, interpretable baseline.

Applications

The method underpins credit scoring, medical diagnosis, marketing response models, spam filtering and click-through-rate prediction. It also forms the final layer of many neural networks, where a sigmoid or softmax output effectively reproduces logistic regression on learned features.

References

1. Cox, D. R. (1958). The Regression Analysis of Binary Sequences. Journal of the Royal Statistical Society, Series B.
2. Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning. Springer.
3. Bank Negara Malaysia. (2024). Model risk management guidance for financial institutions.