Gain insights into ML system design, state-of-the-art techniques, and best practices for scalable production. Learn from top researchers and stand out in your next ML interview.

Machine Learning System Design is an important component of any ML interview. The ability to address problems, identify requirements, and discuss tradeoffs helps you stand out among hundreds of other candidates. Readers of this course able to get offers from Snapchat, Facebook, Coupang, Stitchfix and LinkedIn. 

This course will help you understand the state of the practice on model techniques along with best practices in applying ML models in production at scale. Once you finished the course you can learn more use-cases at: http://mlengineer.io/

Once you're done with the course, you will be able to apply and leverage knowledge from top researchers at tech companies. You will have up to date knowledge in model techniques from hundreds of the latest research and industry papers. There is even a chance that the interviewers will be surprised at the depth of your knowledge.

I really found the quizzes very helpful for testing my ML understanding. Also, the resources shared helped me a lot for revising concepts for my interview preparation. This course will definitely help engineers crack Machine Learning Engineering and Data Science interviews

Senior Data Scientist at Amazon

I have been using your github repo to prep for my interviews and got an offer with NVIDIA with their data science team. Thanks again for your help!

Data Scientist at NVIDIA

I really like what you've built, it'll help a lot of engineers.

MLE at Facebook

It's well organized and the illustrations are well done. Being able to visualize and walk through the steps in order is really helpful in system design. The hints for quizzes is a nice addition, a hint is given if you get stuck in a real interview, mimicking how a real interview 

DS at Fortune 500

Andrew

I just heard back from the recruiter that I passed the Google L5 HC. Thank you very much for sharing the resources on GitHub and for the course on educative.io!

Google Machine Learning Engineer, L5

I got the offer from Intuit. Thanks so much, it would not have been possible without your help.

Senior Machine Learning Engineer, Intuit

I got Google, Facebook, Apple, Tesla, Cruise offer for Senior ML engineer. I thought the course is super helpful. 

Senior Machine Learning Engineer at Cruise

Machine Learning System Design

# Airbnb rental search ranking
 
# 1. Problem statement
Airbnb users search for available homes at a particular location. The system should sort stays from multiple homes in the search result so that the most frequently booked homes appear on top. 




* The naive approach would be to craft a custom score ranking function. For example, a score based on text similarity given a query. This wouldn’t work well because similarity doesn’t guarantee a booking. 

* The better approach would be to sort results based on the likelihood of booking. We can build a supervised ML model to predict booking likelihood. This is a binary classification model, i.e., classify  booking and not-booking. 
  

# 2. Metrics design and requirements
## Metrics
### Offline metrics
* Discounted Cumulative Gain
$DCG_{p} = \sum_{i=1}^p {rel_{i} \over log_{2}(i+1)}$

  * where $rel_{i}$ stands for relevance of result at position $i$. 

- Normalized discounted Cumulative Gain:


$$ nDCG_{p} = {DCG_{p} \over IDCG_{p}}$$

- IDCG is ideal discounted cumulative gain:
$$ IDCG_{p} = \sum_{i=1}^{|REL_{p}|} {2^{rel_{i}} - 1 \over log_{2}(i+1)}$$



### Online metrics
* Conversion rate and revenue lift: This measures the number of bookings per number of search results in a user session.

  $conversion\_rate = \frac{number\_of\_bookings}{ number\_of\_search\_results}$


## Requirements
### Training 

* Imbalanced data and clear-cut session: An average user might do extensive research before deciding on a booking. As a result, the number of non-booking labels has a higher magnitude than booking labels.

* Train/validation data split: Split data by time to mimic production traffic, for example, we can select one specific date to split training and validation data. We then select a few weeks of data before that date as training data and a few days of data after that date as validation data.
 

### Inference 
* Serving: Low latency (50ms - 100ms) for search ranking

* Under-predicting for new listings: Brand new listings might not have enough data for the model to estimate likelihood. As a result, the model might end up under-predicting for new listings. 


## Summary

|**Type**|**Desired goals**|
|:-: |:-: |
|Metrics| Achieve high normalized discounted Cumulative Gain metric |
|Training|Ability to handle imbalance data|
||Split training data and validation data by time|
|Inference|Latency from 50ms to 100ms|
||Ability to avoid under-predicting for new listings|

Let’s dive into the problem statement and metrics required for the Airbnb rental search ranking application.

Machine Learning Primer

Video Recommendation

Feed Ranking

Ad Click Prediction

Rental Search Ranking

Estimate Food Delivery Time

Conclusion

Machine Learning Knowledge

Machine Learning Model Diagnosis

Problem Statement and Metrics

Airbnb rental search ranking

1. Problem statement

2. Metrics design and requirements

Metrics

Offline metrics