Career roadmap
Recommender Systems Engineer
Decide what a hundred million people see next, at low latency, and prove it made things better.
Before you start RecSys
- Machine learning fundamentals
- Strong Python and SQL
- Backend and distributed systems basics
Recommendation fundamentals
The classical methods still form the baseline every new model is measured against.
The foundation, and still competitive when implemented well.
- User-based and item-based approaches
- Matrix factorisation and ALS
- Implicit versus explicit feedback
- Cold start problems
Essential for cold start and for explaining recommendations.
- Item feature engineering
- Embedding-based similarity
- Hybrid approaches
- Metadata quality dependence
Offline metrics mislead constantly, and knowing how is a senior signal.
- Precision, recall and NDCG at k
- Temporal splits versus random splits
- Popularity bias in evaluation
- Why offline gains often vanish online
Logged behaviour is biased by the system that produced it.
- Implicit signals and their noise
- Position and presentation bias
- Feedback loops and rich-get-richer
- Logging for future training
Optimising clicks alone reliably degrades the product.
- Engagement versus satisfaction
- Multi-objective trade-offs
- Long-term value modelling
- Guardrail metrics
BuildBuild collaborative filtering and content-based recommenders and compare them offline.
Modern architectures
The two-stage retrieval and ranking architecture used by nearly every large system.
The architecture that makes recommendation tractable at scale.
- Candidate generation strategies
- Two-tower retrieval models
- Ranking with rich features
- Reranking for diversity and business rules
Vector retrieval at scale, with latency budgets measured in milliseconds.
- Learning item and user embeddings
- Approximate nearest neighbour indexes
- Index refresh and freshness
- Recall versus latency tuning
Session and sequence models capture intent that static profiles miss.
- Session-based recommendation
- Transformer models for sequences
- Short-term versus long-term interest
- Real-time sequence features
Training/serving skew is the most common production bug in this field.
- User, item and context features
- Point-in-time correctness
- Feature stores and consistency
- Real-time feature computation
Without exploration the system only ever learns about what it already shows.
- Multi-armed and contextual bandits
- Exploration versus exploitation trade-off
- Off-policy evaluation
- Cold start via exploration
BuildImplement a two-tower retrieval model and a ranking model, and measure the pipeline end to end.
Serving at scale
A recommender is a low-latency distributed system that happens to contain models.
Tail latency directly affects revenue in recommendation surfaces.
- Budget allocation across pipeline stages
- Caching strategies and invalidation
- Precomputation versus real-time
- p99 latency and timeouts
Serving two models per request, tens of thousands of times per second.
- Inference optimisation
- Batching and hardware choice
- Model versioning and rollout
- Graceful degradation to fallbacks
New items and changing interests need to reach the system quickly.
- Incremental index updates
- Near-real-time feature updates
- Model retraining cadence
- Handling item churn
Recommendation infrastructure is often among the largest compute costs.
- Sharding and partitioning strategies
- Cost per thousand recommendations
- Capacity planning for peaks
- Efficiency versus quality trade-offs
Silent degradation is common: the system still returns results, just worse ones.
- Model and feature drift detection
- Coverage and catalogue utilisation
- Latency and error dashboards
- Alerting on business metrics
BuildServe recommendations under a 100ms budget with caching, fallbacks and measured tail latency.
Measurement and product impact
Recommenders are judged by online experiments, not offline metrics.
The only measurement that counts, and the one interviews focus on.
- Experiment design for ranking changes
- Interference and network effects
- Long-term holdouts
- Novelty effects
Optimising one metric almost always degrades another.
- Engagement, retention and revenue
- Diversity, novelty and serendipity
- Creator and supply-side health
- Guardrails against harmful optimisation
Recommenders shape what people see, which brings scrutiny and regulation.
- Popularity bias and long-tail exposure
- Filter bubbles and diversity interventions
- Fairness across user and item groups
- Regulatory transparency requirements
Real systems must honour inventory, contracts and editorial decisions.
- Hard constraints in reranking
- Promotion and sponsored placement
- Blocklists and compliance filtering
- Balancing rules against model output
Why did this user see this item is a question you will answer constantly.
- Explainability tooling for recommendations
- Tracing a recommendation end to end
- Diagnosing quality complaints
- Distinguishing model from data issues
BuildRun an A/B test on a recommendation change, with guardrails and a written decision memo.
Interview preparation
Interviews are system design heavy, with modelling and experimentation rounds.
Design the feed. The archetypal recommender interview question.
Ch — Scalable APIs- Two-stage architecture from scratch
- Latency budget allocation
- Feature and data pipeline design
- Cold start handling
Model choice, loss functions and negative sampling get asked precisely.
- Negative sampling strategies
- Loss functions for ranking
- Handling implicit feedback
- Embedding dimensionality choices
How would you know this recommender is better.
- Metric selection and guardrails
- Experiment design pitfalls
- Offline to online correlation
- Deciding to ship or roll back
Practical implementation and data manipulation at scale.
- Implementing NDCG or MRR
- Building a training dataset with correct time splits
- Efficient candidate generation
- PySpark or SQL over interaction logs
A working recommender with honest evaluation and a critique of its own metrics.
- End-to-end project with serving
- Offline evaluation with temporal splits
- Written critique of metric limitations
- A bandit or exploration experiment
BuildA public recommender project with offline metrics and a written evaluation critique.
RecSys tools on your CV
- PyTorch
- Spark
- Feature stores
- FAISS / ScaNN
- Kafka
- Redis
- Experimentation platforms
What RecSys employers ask to see
- An end-to-end recommender with a serving layer
- Offline evaluation using temporal splits, with limitations stated
- An A/B test analysis with guardrail metrics
- A latency optimisation case study
Every marketplace, streaming service, social platform and large retailer runs recommendations. The revenue link is direct, which makes the role well funded and heavily measured.
Content last reviewed 2026-08-31. Guidance only — no institute or paid placement is endorsed anywhere in this book.