Recommendation System

A recommendation system (also called a recommender system or recommendation engine) is a class of information filtering algorithm designed to predict the preferences or interests of a user and proactively present relevant items from a large catalogue. Recommendation systems are among the most economically significant AI applications in deployment: they drive a substantial fraction of content consumption on streaming platforms, product discovery on e-commerce sites, and engagement on social networks. Netflix has estimated that its recommendation system is responsible for approximately 80% of the content watched on its platform; Amazon attributes roughly 35% of its revenue to recommendations.[^1] The core computational challenge is predicting which items, from catalogues numbering in the millions, a given user will engage with, based on sparse and noisy signals about past behaviour.

Approaches

Collaborative Filtering

Collaborative filtering is the foundational approach to recommendation, based on the principle that users who have agreed in the past tend to agree in the future. It analyses the recorded interactions between users and items — purchases, ratings, clicks, play counts — to identify patterns that allow preferences to be transferred between users who have not interacted with the same items.

The user-item interaction matrix is the central data structure: rows represent users, columns represent items, and each cell holds an interaction signal (such as a star rating or a binary purchase indicator). Because most users interact with only a tiny fraction of available items, this matrix is extremely sparse — typically over 99% empty. The task of collaborative filtering is to fill in the missing entries: predicting which items a user would rate highly if they encountered them.

Matrix factorisation techniques such as Singular Value Decomposition (SVD) and Alternating Least Squares (ALS) decompose the interaction matrix into latent factor representations of users and items in a low-dimensional embedding space, where the dot product of a user vector and an item vector approximates the expected rating. These low-dimensional representations capture latent concepts such as genre preferences, price sensitivity, or brand loyalty without requiring explicit labels for these dimensions.[^2]

Content-Based Filtering

Content-based filtering recommends items similar to those a user has previously engaged with, using features of the items themselves rather than the behaviour of other users. A music recommendation system might recommend tracks with similar tempo, key, instrumentation, and genre to songs the user has liked. A news recommendation system might represent articles as TF-IDF or embedding vectors and recommend articles with high cosine similarity to recently read content. Content-based approaches are less susceptible to the cold-start problem for new items — since a new product can be described by its features even before any user has purchased it — but they are limited by the quality of item feature representations and tend to produce less serendipitous recommendations (recommending more of the same rather than novel discoveries).

Hybrid Systems

Most production recommendation systems combine collaborative and content-based signals, alongside additional signals such as recency, popularity, geographic location, time of day, and contextual factors (e.g., browsing session, device type). These hybrid systems can balance the strengths of each approach — the personalisation depth of collaborative filtering with the cold-start resilience of content-based methods.

Deep Learning for Recommendations

Since approximately 2016, deep learning has become the dominant paradigm in large-scale recommendation systems. Neural collaborative filtering replaces the dot product of matrix factorisation with a neural network capable of learning more complex user-item interaction functions. Two-tower architectures learn separate deep encoders for users and items, enabling efficient retrieval from large item catalogues through approximate nearest-neighbour search over embedding spaces. Session-based recommendation models — often using recurrent neural networks or transformers — model the sequential dynamics of user behaviour within a single browsing session. TikTok's recommendation algorithm, widely credited with making short-video content consumption highly addictive, is a deep learning system that incorporates video content features, user behaviour sequences, and real-time engagement signals.[^3]

The Cold-Start Problem

A persistent challenge in recommendation systems is the cold-start problem: the difficulty of making good recommendations for new users (no interaction history) or new items (no engagement signals). Approaches include asking new users to explicitly rate a set of seed items during onboarding, using demographic or contextual signals as a proxy for preference, applying content-based methods for new items, and leveraging transfer learning from related domains. Contextual bandits — a class of reinforcement learning algorithms — are commonly deployed to balance exploration (exposing new items to gather signal) with exploitation (recommending known high-engagement items).

Evaluation Metrics

Recommendation quality is measured using a combination of offline and online metrics. Offline metrics applied to held-out interaction data include Precision@K (fraction of top-K recommendations the user actually engaged with), Recall@K (fraction of all items the user would engage with that appear in the top-K), NDCG (Normalised Discounted Cumulative Gain, which weights relevant items higher when they appear earlier in the ranked list), and Mean Average Precision (MAP). Online A/B testing measures real-world impact on business metrics including click-through rate (CTR), conversion rate, session length, and revenue per session.

References

1. Gomez-Uribe, C. A., & Hunt, N. (2016). The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems, 6(4), 1–19.
2. Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix Factorization Techniques for Recommender Systems. IEEE Computer, 42(8), 30–37.
3. Zhang, S., Yao, L., Sun, A., & Tay, Y. (2019). Deep Learning Based Recommender System: A Survey and New Perspectives. ACM Computing Surveys, 52(1), 1–38.
4. Lazada Group. (2024). Lazada Announces Suite of GenAI Features to Transform Shopping and Seller Experiences in Southeast Asia. PR Newswire.