Gain insights into using Python for scientific and engineering applications. Delve into real-world scenarios, learning NumPy, Matplotlib, audio processing, and more through practical, hands-on projects.

Python for Scientists and Engineers_468 x 60 copy 2.png

jupter.tar.gz

notebook

obesity_plot_gender

print_gender_pandas

create_sine_wave

notebook_exercise

range

print_gender_pandas_remove_NaN

print_gender_year

print_data_age

print_data_age_with_year

print_data_age_remove_NaN

obesity_plot_age_simple

obesity_plot_age_clear

print_data_age_under_16

print_data_age_25_34

obesity_plot_children_vs_adults

print_movie_top_rated

print_movies_oldie_teen

print_movies_by_gender

print_movies_for_women

print_movies_by_gender_loc

print_movies_diff_gender

movies_diff_gender_plot

print_u_data

print_movies_oldie_index

print_movies_oldies

print_movies_teens

display_image_nouse

grey_image

blur_image

edge_detection_low

edge_detection_high

plot_sin_wave

create_wave_file

display_again

print_frequencies

count_objects

detect_faces

detect_faces_little_mix

count_exact_lines

webcam_detection

single_thread

find_correlation

webcam_face_detection

motion_detection

motion_detection_live

In this course, you will learn to use Python for real-world scientific/engineering applications. For each topic, there will be a real case scenario where you will build a quick solution in Python to solve the problem.

More specifically, you will cover topics such as creating a word counter, NumPy and Matplotlib, audio processing, and a lot more. Throughout each chapter, you will get hands-on experience in building solutions for complex problems that engineers and scientists face every day.

The Practical Guide to Python for Scientists and Engineers

We’ve generated some random data for a few movie ratings. Let’s have a look at it.

{'Terminator': {'Tom': 4.0,
  'Jack': 1.5,
  'Lisa': 3.0,
  'Sally': 2.0},
  
 'Terminator 2': {'Tom': 5.0,
  'Jack' : 1.0,
  'Lisa': 3.5,
  'Sally': 2.0},
  
 'It happened one night': {'Tom': 3.5,
  'Jack': 3.5,
  'Tiger': 4.0,
  'Lisa': 5.0,
  'Michele': 3.0,
  'Sally': 4.0,},
  
 '27 Dresses': {'Tom': 3.0,
  'Jack': 3.5,
  'Tiger': 3.0,
  'Lisa': 5.0,
  'Michele': 4.0,
  'Sally': 4.0},

  'Poirot': {'Tom': 4.0,
  'Jack': 3.0,
  'Tiger': 5.0,
  'Lisa': 4.0,
  'Michele': 3.5,
  'Sally': 3.0,
  },

 'Sherlock Holmes': {'Tom': 4.0,
  'Jack': 3.0,
  'Tiger': 3.5,
  'Lisa': 3.5,
  'Sally': 2.0,
  }}

The movie data is stored as a **dictionary**. Each dictionary has its sub-dictionary. Let’s look at the first movie:

```
'Terminator': {'Tom': 4.0,
  'Jack': 1.5,
  'Lisa': 3.0,
  'Sally': 2.0},
```

The movie *Terminator* has been rated by four people: Tom has given it a score of 4.0, while Jack has given it 1.5, and so on. These numbers are random.

We will notice that not everyone has rated every movie. This is something we will need to take into account when we are calculating the correlation.

Let’s see how we can calculate the correlation for the following:

```
if len(sys.argv) < 2:
    print("Usage: python calc_correlation.py <data file.py>")
    exit(1)
```

We want to give the script a data file to calculate the correlation on. If the file is not provided, we’ll print the usage and exit.

```
with open(sys.argv[1], 'r') as f:
    temp = f.read()
    movies_list = ast.literal_eval(temp)
    print(movies_list)

```
Usually, when we open a file, we have to close it to deal with any errors. The `with` function does all that for us. It will open the file, close it at the end, and handle any errors that may arise.

We’re looking at the code line by line now.
```
with open(sys.argv[1], 'r') as f:
```
We’ll open the file passed, which is the first argument, as read-only.
```
temp = f.read()
movies_list = ast.literal_eval(temp)
```

The first thing we’ll do is read the file into a temp variable. The following line may require some explanation. A nifty feature in Python is the eval() function. It allows us to generate Python code in real-time and run it. This is very dangerous, as we won’t know what we are reading. Someone could insert code to delete all of our files. For this reason, `eval()` is rarely used in practice.

The `literal_eval()` function in the ast library works around the risks of eval. `literal_eval` will only allow Python data structures like lists, dictionaries, or other Python data structures to be read. If we try to read anything else, it will throw an error. But why do we need it anyway? The file we are reading, `ml_data1.py` contains a Python dictionary. However, when we read the file, it is read as a text file. We need to convert it to a Python dictionary. That is what `literal_eval` will do. It takes the data it has read and converts it to a usable Python dictionary. If we try to pass in something dangerous, the function will throw an error. There are other ways to do this, such as the **json** module, which we’ll review later on. Anyway, we have now read the data into a file.

```
correlated_dict = {}
for movie in movies_list:
    correlated_dict[movie] =  find_correlation(movies_list, movie)
```

The `movies_list` text is what we read in from the file. We’ll loop over that and find the correlation for each movie. Let’s look at how the function `find_correlation()` does that.

```
def find_correlation(movie_list, movie_for_correlation):
    '''
    Input:
    movie_list - List of movies
    movie_for_correlation: The movie to calculate the correlation for
    Return:
    Dictionary of correlation for movie_for_correlation
    '''
    correlate_dict = {}

```

The function takes two parameters:

1) A list of movies.

2) The movie to find the correlation for.

It returns a dictionary of correlated values.

```
correlate_dict = {}
for movie in movie_list:

```

We’ll declare `correlate_dict` the final dictionary that we will return. We’ll then loop over the movie list.

```
if movie != movie_for_correlation:
    movie_for_correlation_list = []
    movie_to_compare_list = []

```

When doing the calculations, we don’t want to compare a movie to itself as that would always show a perfect correlation. So we’ll check for that and then declare a few empty arrays.

Using the data file we showed at the start of this lesson, we should see that we have the movie and the people who reviewed it. Now, we’ll loop over the reviewers.

```
for reviewer_name in movie_list[movie_for_correlation]:
    # Check that the reviewer has reviewed the current movie.
    # If so, calculate the correlation coefficient.
    # If they have not reviewed the movie, then it makes no sense to do a correlation.
    if reviewer_name in movie_list[movie]:
```

It’s important to check that the reviewer in the last line that is being looped over reviewed the current movie. If they didn’t we’ll move on to the next reviewer in the next line.

```
if reviewer_name in movie_list[movie]:
        movie_for_correlation_list.append(movie_list[movie_for_correlation][reviewer\
_name])
        movie_to_compare_list.append(movie_list[movie][reviewer_name])

```

We are still in the `for` loop. If the reviewer reviewed the movie, we’ll save their score for the current movie, as well as their score for the movie under consideration.

To make it clear, let’s look at the whole `for` loop:

```
# Loop through the people who reviewed the movie
for reviewer_name in movie_list[movie_for_correlation]:
    # Check that the reviewer has reviewed the current movie.
    # If so, calculate the correlation coefficient.
    # If they have not reviewed the movie, then it makes no sense to do a correlation.
    if reviewer_name in movie_list[movie]:
        movie_for_correlation_list.append(movie_list[movie_for_correlation][reviewer\
_name])
        movie_to_compare_list.append(movie_list[movie][reviewer_name])

```

This `for` loop will loop over the reviewers, and if they reviewed both the movies, will store their scores in two arrays. The complicated code compensates for the fact that not all people reviewed all movies.

```
correlate_dict[movie] = np.corrcoef(movie_for_correlation_list,movie_to_compare_list\
)[1][0]
return correlate_dict
```

Finally, we’ll calculate the correlation using the np.corrcoef() function we studied in the lesson [What is Recommendation Engine](https://www.educative.io/courses/practical-guide-python-scientists-engineers/what-is-a-recommendation-engine). The function creates a dictionary with correlation scores for the current movie. For example, if the movie is *Terminator*, the dictionary would store what correlation score *Terminator* has compared to *Poirot*, *Sherlock Holmes*, etc.

Each movie will have its list. So *Poirot* will have its dictionary, which will store its relation to *Terminator* and other movies.

We’ll return the dictionary back in the main loop:

```
for movie in movies_list:
    correlated_dict[movie] =  find_correlation(movies_list, movie)
print(correlated_dict)
```

As we said, we’re creating a correlation dictionary for each movie. The goal is to create a correlation index that we can use later to make recommendations.

```
json.dump(correlated_dict, open("corr_dict.py",'w'))
```

Remember when we said we’d show another way to write Python dictionaries to file? This is the easier way, using the json module. It opens the file and dumps our dictionary in one go. Let’s open our file `corr_dict.py` and look at it.

Let’s look at how to find correlations between different datasets.

Finding Correlation Between the Movie Ratings

Introduction

Create a Word Counter in Python

An Introduction to NumPy and Matplotlib

Python pandas

Audio Processing

Image and Video Processing

MultiThreading vs. Multiprocessing in Python

Machine Learning with an Amazon-like Recommendation Engine

Conclusion

Finding Correlation Between the Movie Ratings