Explore machine learning's essentials for software engineers, delve into supervised learning, neural networks, and deep learning, and gain skills to tackle real-world data challenges effectively.

pml_final.tar.gz

just_run

chapter06

SPA_07

SPA_gdb_frontEnd

J12-1

J12-2

SPA-server_image

Handson-10

Machine learning is the future for the next generation of software professionals.

This course serves as a guide to machine learning for software engineers. You’ll be introduced to three of the most relevant components of the AI/ML discipline; supervised learning, neural networks, and deep learning. You’ll grasp the differences between traditional programming and machine learning by hands-on development in supervised learning before building out complex distributed applications with neural networks. You’ll go even further by layering networks to create deep learning systems. You’ll work with complex real-world datasets to explore machine behavior from scratch at each phase.

By the end of this course, you’ll have a working knowledge of modern machine learning techniques. Using software engineering, you’ll be prepared to build complex neural networks and wrangle real-world data challenges.

Fundamentals of Machine Learning for Software Engineers

Soon enough, we’ll get to tune our neural network and make it as accurate as possible. Before we do that, however,  we need a reliable test to measure that accuracy. As it turns out, ML testing comes with a subtly counterintuitive hurdle that can easily complicate things .

> **Side note:** This short chapter tells us where that testing trapdoor is, and how to step around it.
# Overfitting

Since the first part of this course, we’ve been using two separate sets of examples: one for training our algorithms, and one for testing them. Let’s refresh our memory: _why do we not use the same examples for both training and testing?_

Let’s use a metaphor to answer that question. Imagine we teach basic math to a class of young kids. We’ve already prepared 60 multiple-answer multiplication quizzes. We plan to assign most of those quizzes as homework. We also plan to select 10 quizzes for a final test to check how well the kids are learning.

We want to split the quizzes into two groups:”**homework**”  and “**test**”. Now, we have two options: either we assign all 60 quizzes to ”homework,” and then reuse 10 of them in ”test,” or we split the quizzes, assign 50 of them to ”homework”, and the remaining 10 to ”test.” Which option would we pick?

Most probably, we would go for the second option for one reason: we do not want the kids to memorize multiplications.We want them to understand the rules of multiplications.Therefore, we would rather test their knowledge on quizzes that they have not seen before. Otherwise, they might get the right answers because they remember those answers from their exercises.

The same reasoning applies when we train a supervised learning system. If we train a network to recognize photos of beagles, we want it to recognize any beagle picture, not just the specific ones it encountered in training. That’s easier said than done. Supervised learning systems, like people, have a tendency to memorize their training examples instead of generalizing from them. Back in [Training vs. Testing](https://www.educative.io/courses/fundamentals-of-machine-learning-for-software-engineers/data-comes-first#Training-vs.-testing), we called this **problem overfitting**.

To counter overfitting, we introduced the idea of a test set. Just like the teacher in our story, we train our neural networks on one set of examples and test them on a different set of examples. That’s how we get a realistic estimate of a network’s performance in production, where it will be faced with data that it’s never seen before.

Because of overfitting, the network tends to be more accurate on familiar training data and less accurate on unfamiliar test data. Conversely, the network’s loss tends to be lower on the training data and higher on the test data.

## Tweak our neural network code
Let’s put those expectations to the test. We build a version of our neural network that tracks the loss and accuracy on both the training and the test set, iteration by iteration. Then we run this hacked network for 10,000 iterations, with batch GD (that is, putting all the examples in one large batch), `n_hidden_nodes=200`, and `lr=0.01:`:


Learn how overfitting can ruin the model.

How Machine Learning Works

Our First Learning Program

Walking the Gradient

Hyperspace

A Discern Machine

Get Real

The Final Challenge

The Perceptron

Designing the Network

Building the Network

Training the Network

How Classifiers Work

Batchin’ Up

The Zen of Testing

Let’s Do Development

A Deeper Kind of Network

Diabetes Prediction Using Keras

Defeating Overfitting

Taming Deep Networks

Beyond Vanilla Networks

Into the Deep

Recognize Handwritten Digits Using a Deep Neural Network

The Threat of Overfitting

Overfitting