Learn about ML pipeline development, delve into best practices, discover advanced Python concepts, and explore testing methodologies to elevate your software engineering skills and career prospects.

Building a Machine Learning Pipeline from Scratch.png

docker_setup.tar.gz

omegaconf-code-widget

omegaconf-spa-widget

omegaconf-spa-widget-copy

omegaconf-code-widget-copy

omegaconf_example

dataset_model_examples

logging_example

docstring_example

pipeline_iris_classification

pipeline_autompg_regression

reporting

pipeline_packaging

inference

unit_testing

pipeline_unit_testing

experiment_tracking

argparse_example

Machine learning (ML) has matured into a mainstream development activity, and data scientists are expected to be able to write production-grade training pipelines. This course will provide you with a foundation in ML pipeline development guided by best practices in software engineering.

You’ll start by learning about code organization, style, and conceptual ideas, such as topological sorting of directed acyclic graphs. You’ll dive into the hands-on development of an ML pipeline. You’ll learn some advanced Python concepts and useful libraries. Finally, you’ll learn about testing methodologies and how to package your code into a distributable library.

By the end of this course, you’ll understand what makes a great ML pipeline work, better appreciate software engineering processes, improve your Python knowledge, and learn how to build ML training pipelines. This knowledge will make you more attractive to potential employers and improve your chances of thriving as a new data scientist.

Building a Machine Learning Pipeline from Scratch

Each statement in the code as a separate unit.

In unit testing, what do you test?

**(True or False)** In test-driven development, you write a function, followed by the corresponding unit test.

(True or False) In test-driven development, you write a function, followed by the corresponding unit test.

What is the minimum number of test cases you should write for the following function?

```python
def foo(x: int) -> int:
    if x == 0:
        return 0
    elif x == 1:
        return "The value is 1."
    else:
        return "The value is > 1."
```

What is the minimum number of test cases you should write for the following function?
<pre><code class="language-python">def foo(x: int) -&gt; int:
 if x == 0:
 return 0
 elif x == 1:
 return &quot;The value is 1.&quot;
 else:
 return &quot;The value is &gt; 1.&quot;
</code></pre>

**(Select all that apply.)** You ran unit tests for the code below and got a code coverage of 67%. What is a possible reason for not achieving 100% coverage?

```python
def foo(x: int) -> int:
    if x == 0:
        return 0
    elif x == 1:
        return "The value is 1."
    else:
        return "The value is > 1."
```


(Select all that apply.) You ran unit tests for the code below and got a code coverage of 67%. What is a possible reason for not achieving 100% coverage?
<pre><code class="language-python">def foo(x: int) -&gt; int:
 if x == 0:
 return 0
 elif x == 1:
 return &quot;The value is 1.&quot;
 else:
 return &quot;The value is &gt; 1.&quot;
</code></pre>

In functional testing, you test the functionality of the code. In performance testing, you make sure that your model performs with adequate accuracy.

In functional testing, you test the functionality of the code. In performance testing, you test aspects of the system such as the time taken to complete a certain task or the scalability of the system to larger datasets.

In functional testing, you perform unit testing. In performance testing, you test aspects of the system such as the time taken to complete a certain task or the scalability of the system to larger datasets. 

In functional testing, you test functions. In performance testing, you test the accuracy of the model.

What is the difference between functional testing and performance testing?

What is the difference between functional testing and performance testing?

Test yourself on the concepts learned in this chapter.

Introduction

Getting Started

Structuring the ML Pipeline

Directed Acyclic Graphs (DAGs)

The ML Library

Create Your First Data Pipeline with a Dashboard

The Pipeline Core

Extending the Pipeline

Build a News ETL Data Pipeline Using Python and SQLite

Testing

Deployment

Other Considerations

Wrapping Up

Appendix

Final Assessment

Quiz: Testing