Gain insights into ML system design, state-of-the-art techniques, and best practices for scalable production. Learn from top researchers and stand out in your next ML interview.

Machine Learning System Design is an important component of any ML interview. The ability to address problems, identify requirements, and discuss tradeoffs helps you stand out among hundreds of other candidates. Readers of this course able to get offers from Snapchat, Facebook, Coupang, Stitchfix and LinkedIn. 

This course will help you understand the state of the practice on model techniques along with best practices in applying ML models in production at scale. Once you finished the course you can learn more use-cases at: http://mlengineer.io/

Once you're done with the course, you will be able to apply and leverage knowledge from top researchers at tech companies. You will have up to date knowledge in model techniques from hundreds of the latest research and industry papers. There is even a chance that the interviewers will be surprised at the depth of your knowledge.

I really found the quizzes very helpful for testing my ML understanding. Also, the resources shared helped me a lot for revising concepts for my interview preparation. This course will definitely help engineers crack Machine Learning Engineering and Data Science interviews

Senior Data Scientist at Amazon

I have been using your github repo to prep for my interviews and got an offer with NVIDIA with their data science team. Thanks again for your help!

Data Scientist at NVIDIA

I really like what you've built, it'll help a lot of engineers.

MLE at Facebook

It's well organized and the illustrations are well done. Being able to visualize and walk through the steps in order is really helpful in system design. The hints for quizzes is a nice addition, a hint is given if you get stuck in a real interview, mimicking how a real interview 

DS at Fortune 500

Andrew

I just heard back from the recruiter that I passed the Google L5 HC. Thank you very much for sharing the resources on GitHub and for the course on educative.io!

Google Machine Learning Engineer, L5

I got the offer from Intuit. Thanks so much, it would not have been possible without your help.

Senior Machine Learning Engineer, Intuit

I got Google, Facebook, Apple, Tesla, Cruise offer for Senior ML engineer. I thought the course is super helpful. 

Senior Machine Learning Engineer at Cruise

Machine Learning System Design

# Estimate Delivery Time
## 1. Problem statement
Build a model to estimate the total delivery time given order details, market conditions, and traffic status.

> To keep it simple, we do not consider batching (group multiple orders at restaurants) in this exercise. 

<center>

$Delivery Time = Pickup Time + Point\_to\_Point Time + Drop\_off\_Time$

</center>

## 2. Metrics design and requirements

### Metrics
* Offline metrics: Use Root Mean Squared Error (RMSE)



   $\sqrt{\sum_{k=1}^n\frac{(predict-y)^2}{n}}$



  where, 

  $n$ is the total number of samples, 
  
  $predict$ is Estimated wait time, 

  $y$ is the actual wait time.


- Online metrics: Use A/B testing and monitor #key# RMSE: Root Mean Squared Error #key#, customer engagement, customer retention, etc. 

### Requirements
#### Training
- During training, we need to handle a large amount of data. For this, the training pipeline should have a high throughput. To achieve this purpose, data can be organized in [Parquet files](https://www.educative.io/courses/machine-learning-system-design/training-pipeline)

- The model should undergo retraining every few hours. Delivery operations are under a dynamic environment with a lot of external factors: traffic, weather conditions, etc. So, it is important for the model to learn and adapt to the new environment. For example, on game day, traffic conditions can get worse in certain areas. Without a retraining model, the current model will consistently underestimate delivery time. Schedulers are responsible for retraining models many times throughout the day.

- Balance between overestimation and under-estimation. To help with this, retrain multiple times per day to adapt to market dynamic and traffic conditions.


#### Inference
- For every delivery, the system needs to make real-time estimations as frequently as possible. For simplicity, we can assume we need to make 30 predictions per delivery. 
- Near real-time update, any changes on status need to go through model scoring as fast as possible, i.e., the restaurant starts preparing meals, the driver starts driving to customers. 
- Whenever there are changes in delivery, the model runs a new estimate and sends an update to the customer. 
-  Capture near real-time aggregated statistics, i.e., feature pipeline aggregates data from multiple sources (Kafka, database) to reduce latency.
- Latency from 100ms to 200ms


### Summary

|**Type**|**Desired goals**|
|:-: |:-: |
|Metrics|Optimized for low RMSE. Estimation should be less than 10-15 minutes. If we overestimate, customers are less likely to make orders. Underestimation can cause customers upset.|
|Training|High throughput with the ability to retrain many times per day|
|Inference|Latency from 100ms to 200ms|
















Let’s dive deeper into the problem statement and metrics required for the food delivery system.

Machine Learning Primer

Video Recommendation

Feed Ranking

Ad Click Prediction

Rental Search Ranking

Estimate Food Delivery Time

Conclusion

Machine Learning Knowledge

Machine Learning Model Diagnosis

Problem Statement and Metrics

Estimate Delivery Time

1. Problem statement