Recommender Systems and Ranking

Recommendation and search ranking power feeds, shopping, streaming, jobs and ads, so they appear constantly in applied-ML interviews. The question is usually open-ended ("design a recommender for X"), and a strong answer uses the standard multi-stage funnel, names the signals and metrics, and acknowledges feedback loops and cold start.

1. Framing the problem

Clarify first:

• What is being recommended (items, people, content, queries)? To whom, and where in the product?
• What signal counts as success: click, watch time, purchase, return visits, long-term retention?
• Explicit feedback (ratings) is rare and biased. Implicit feedback (views, clicks, dwell time, purchases) is abundant but noisy: a missing interaction is not a dislike, it may just be unseen.
• Constraints: latency (tens of milliseconds), catalogue size (millions), freshness, fairness and business rules.

2. Core approaches

Popularity and rules

Recommend the most popular items, perhaps per segment or time window. It is a strong baseline and a fallback for new users. Always build it first.

Content-based filtering

Represent items by features (text, tags, image embeddings, price) and users by the items they liked, then recommend similar items. It handles new items well and is transparent, but it traps users in a narrow bubble and needs good item features.

Collaborative filtering

Use patterns across many users: "people like you also liked". No item features needed.

• Neighbourhood methods: user-user or item-item similarity (cosine over interaction vectors). Item-item is usually more stable and cacheable.
• Matrix factorisation: approximate the user-item matrix $R\approx U{V}^{\top }$ with low-dimensional user and item vectors. A predicted score is the dot product $u_i\cdot v_j$, trained on observed entries with regularisation (ALS or SGD). For implicit feedback, weight observed positives heavily and treat unobserved entries as weak negatives.

import numpy as np

rng = np.random.default_rng(0)
n_users, n_items, k = 30, 20, 3
U_true = rng.normal(size=(n_users, k)); V_true = rng.normal(size=(n_items, k))
R = U_true @ V_true.T
mask = rng.uniform(size=R.shape) < 0.5                     # only half the ratings are observed

U = rng.normal(0, 0.1, (n_users, k)); V = rng.normal(0, 0.1, (n_items, k))
lam, lr = 0.05, 0.05
for _ in range(3000):
    E = (U @ V.T - R) * mask                               # error only on observed entries
    gU = E @ V + lam * U
    gV = E.T @ U + lam * V
    U -= lr * gU / mask.sum(1, keepdims=True).clip(1)
    V -= lr * gV / mask.sum(0, keepdims=True).T.clip(1)

observed_rmse = np.sqrt((((U @ V.T - R) ** 2) * mask).sum() / mask.sum())
held_out_rmse = np.sqrt((((U @ V.T - R) ** 2) * ~mask).sum() / (~mask).sum())
baseline = np.sqrt(((R[~mask] - R[mask].mean()) ** 2).mean())
assert observed_rmse < 0.3
assert held_out_rmse < 0.6 * baseline                      # predicts unseen entries far better than the global mean

• Two-tower neural models: one network embeds the user (and context), another embeds the item; the score is a dot product. Because item embeddings can be precomputed, this scales to approximate nearest-neighbour search over millions of items.

Hybrid

Combine collaborative signals with content features, which fixes cold start and improves quality. In industry, hybrids dominate.

3. The multi-stage funnel

A catalogue of millions cannot be scored by a heavy model per request. So the system narrows in stages.

Multiple retrieval sources are merged. The ranker predicts several outcomes (multi-task) and combines them into a score, for example $\text{score}=p(\text{click}{)}^a\cdot p(\text{purchase}{)}^b\cdot \text{value}$.

4. Features for ranking

• User: history, recency, frequency, long and short-term interests, demographics, device, location.
• Item: category, price, quality, age, popularity, text and image embeddings.
• Context: time of day, page, previous action, session.
• Cross features: user-category affinity, past interactions with this item or seller.

Compute them consistently for training and serving (see the feature chapter), using point-in-time values.

5. Training data and bias

• Labels from logs: a click on an item that was shown. Unshown items have no label.
• Position bias: items at the top get clicked more because of position, not quality. Mitigate by including position as a training-only feature, using inverse-propensity weighting, or randomised exploration slices.
• Selection bias and feedback loops: the model only learns from what it showed, then shows what it learned. Popular items get more popular. Break the loop with exploration (epsilon-greedy, Thompson sampling, small random slots).
• Negative sampling: for implicit feedback, sample items the user did not interact with as negatives; use in-batch negatives for efficiency and correct for popularity.

6. Learning to rank

The pairwise logistic loss for items $i$ (preferred) and $j$ is $\log (1+e^{-(s_i-s_j)})$. LambdaMART remains a strong, hard-to-beat production approach for search ranking.

7. Cold start

• New user: ask for preferences at onboarding; use popularity, context (location, device, referral), then adapt fast as signals arrive.
• New item: use content features, creator or category priors, and exploration to gather its first impressions.
• New system: start with rules and popularity, collect data, and add models as data accumulates.

8. Evaluation

Offline: hold out the most recent interactions per user (time-based split), then compute ranking metrics (recall@k for retrieval; NDCG@k, MAP, MRR for ranking), plus coverage and diversity. Offline numbers are measured on logged data from the old policy, so they under-reward genuinely new recommendations.

Online: A/B test with the business metric (engagement, conversions, revenue, retention) and guardrails (latency, diversity, complaints, long-term satisfaction). Beware short-term optimisation (clickbait raises clicks and hurts retention).

import numpy as np

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & set(relevant)) / len(relevant)

def mrr(ranked, relevant):
    for i, item in enumerate(ranked, start=1):
        if item in relevant:
            return 1.0 / i
    return 0.0

ranked = ["a", "b", "c", "d", "e"]
relevant = {"c", "e", "z"}
assert abs(recall_at_k(ranked, relevant, 3) - 1 / 3) < 1e-12
assert mrr(ranked, relevant) == 1 / 3

9. Beyond accuracy

• Diversity and novelty: avoid ten near-identical items. Maximal Marginal Relevance trades relevance against similarity to items already picked.
• Serendipity and exploration: surface things the user would not have searched for.
• Fairness and creator health: do small creators get exposure?
• Safety and policy: filter harmful or ineligible content before ranking.
• Explainability: "because you watched...".
• Privacy: data minimisation and consent.

10. A design walk-through (template)

For "design a recommendation system for an e-commerce home page":

1. Goal and metric: purchases and revenue per session, with return rate and latency as guardrails.
2. Data: views, carts, purchases, search queries, item catalogue.
3. Candidate generation: item-item co-purchase, two-tower retrieval, recently viewed, trending in category.
4. Ranking: a boosted-tree or deep model predicting purchase probability, using user, item, context and cross features.
5. Re-ranking: diversity across categories, stock availability, margin and policy rules.
6. Cold start, exploration, and position-bias handling.
7. Evaluation: offline recall and NDCG on a time split, then an A/B test.
8. Serving: precomputed embeddings and an ANN index, a feature store, caching, a latency budget (for example under 100 ms end to end).
9. Monitoring: drift, coverage, freshness, and retraining cadence.

11. Common mistakes

• Treating a missing interaction as a dislike.
• Random train/test splits instead of time-based splits, leaking the future.
• Ignoring position bias and feedback loops.
• Optimising clicks with no view of long-term value.
• Scoring the whole catalogue with a heavy model instead of a funnel.
• No cold-start plan.
• Skipping the popularity baseline.

12. Practice questions

1. Compare content-based and collaborative filtering. When does each fail?
2. Explain matrix factorisation for implicit feedback.
3. Why use a multi-stage architecture? What does each stage optimise?
4. How do you handle position bias in training data?
5. How do you evaluate a recommender offline, and why can offline and online results differ?
6. Describe a two-tower model and how it is served at scale.
7. How would you solve cold start for new items and new users?
8. What is the difference between pointwise, pairwise and listwise ranking?
9. How would you add diversity to a ranked list?