Gain insights into ML system design, state-of-the-art techniques, and best practices for scalable production. Learn from top researchers and stand out in your next ML interview.

Machine Learning System Design is an important component of any ML interview. The ability to address problems, identify requirements, and discuss tradeoffs helps you stand out among hundreds of other candidates. Readers of this course able to get offers from Snapchat, Facebook, Coupang, Stitchfix and LinkedIn. 

This course will help you understand the state of the practice on model techniques along with best practices in applying ML models in production at scale. Once you finished the course you can learn more use-cases at: http://mlengineer.io/

Once you're done with the course, you will be able to apply and leverage knowledge from top researchers at tech companies. You will have up to date knowledge in model techniques from hundreds of the latest research and industry papers. There is even a chance that the interviewers will be surprised at the depth of your knowledge.

I really found the quizzes very helpful for testing my ML understanding. Also, the resources shared helped me a lot for revising concepts for my interview preparation. This course will definitely help engineers crack Machine Learning Engineering and Data Science interviews

Senior Data Scientist at Amazon

I have been using your github repo to prep for my interviews and got an offer with NVIDIA with their data science team. Thanks again for your help!

Data Scientist at NVIDIA

I really like what you've built, it'll help a lot of engineers.

MLE at Facebook

It's well organized and the illustrations are well done. Being able to visualize and walk through the steps in order is really helpful in system design. The hints for quizzes is a nice addition, a hint is given if you get stuck in a real interview, mimicking how a real interview 

DS at Fortune 500

Andrew

I just heard back from the recruiter that I passed the Google L5 HC. Thank you very much for sharing the resources on GitHub and for the course on educative.io!

Google Machine Learning Engineer, L5

I got the offer from Intuit. Thanks so much, it would not have been possible without your help.

Senior Machine Learning Engineer, Intuit

I got Google, Facebook, Apple, Tesla, Cruise offer for Senior ML engineer. I thought the course is super helpful. 

Senior Machine Learning Engineer at Cruise

Machine Learning System Design

# Ad click prediction

# 1. Problem statement
Build a machine learning model to predict if an ad will be clicked. 

> For the sake of simplicity, we will not focus on the cascade of classifiers that is commonly used in AdTech.





- Let’s understand the ad serving background before moving forward. The ad request goes through a waterfall model where publishers try to sell its inventory through direct sales with high CPM (Cost Per Million). If it is unable to do so, the publishers pass the impression to other networks until it is sold.




# 2. Metrics design and requirements

## Metrics

During the training phase, we can focus on machine learning metrics instead of revenue metrics or CTR metrics. Below are the two metrics:

### Offline metrics


* Normalized Cross-Entropy (NCE): NCE is the predictive `logloss` divided by the `cross-entropy` of the background CTR. This way NCE is insensitive to background CTR. This is the NCE formula: 

<center>

   $NCE = \frac{-\frac{1}{N} \sum_{i=1}^n (\frac{1+y_i}{2} log(p_i)) + \frac{1-y_i}{2}log(1-p_i))} {-(p*log(p) +(1-p)*log(1-p))}$

</center>


### Online metrics
* Revenue Lift: Percentage of revenue changes over a period of time. Upon deployment, a new model is deployed on a small percentage of traffic. The key decision is to balance between percentage traffic and the duration of the A/B testing phase. 


## Requirements
### Training
* Imbalance data: The Click Through Rate (CTR) is very small in practice (1%-2%), which makes supervised training difficult. We need a way to train the model that can handle highly imbalanced data.

* Retraining frequency: The ability to retrain models many times within one day to capture the data distribution shift in the production environment. 

* Train/validation data split: To simulate a production system, the training data and validation data is partitioned by time.


### Inference 
* Serving: Low latency (50ms - 100ms) for ad prediction. 

* Latency: Ad requests go through a waterfall model, therefore, recommendation latency for ML model needs to be fast. 
* Overspent: If the ad serving model repeatedly serves the same ads, it might end up over-spending the campaign budget and publishers lose money.



## Summary
|**Type**|**Desired goals**|
|:-: |:-: |
|Metrics|Reasonable normalized cross-entropy and click through rate
|Training|Ability to handle imbalance data|
||High throughput with the ability to retrain many times per day|
|Inference|Latency from 50 to 100ms|
||Ability to control or avoid overspent campaign budget while serving ads|

Learn about the problem statement and metrics for building an Ad click prediction machine learning system.

Machine Learning Primer

Video Recommendation

Feed Ranking

Ad Click Prediction

Rental Search Ranking

Estimate Food Delivery Time

Conclusion

Machine Learning Knowledge

Machine Learning Model Diagnosis

Problem Statement and Metrics

Ad click prediction

1. Problem statement