Gain insights into Pandas' powerful tools for data analysis. Learn about Series, DataFrames, and optimize performance for data operations, filtering, and file uploads in data science and machine learning.

Pandas is a very popular Python library that provides powerful, flexible and high-performance tools to process data.

If you’re looking to get into data science, machine learning, or you just want to brush up on your analytics skills, then look no further. In this course, you’ll learn how to process and analyze data with Pandas. 

You’ll start out by covering the basic data structures in Pandas: Series and dataframes. In the latter half of the course, you’ll learn how to perform operations on different data structures, and how to optimize performance for file uploads, data filtering, and more.

Data Analysis & Processing with Pandas

pandas provides extensive methods for data selection. You can find the corresponding lesson in [Select data from DataFrame](https://www.educative.io/courses/data-analysis-processing-with-pandas/select-data-from-dataframe). Below are some alternative methods you could choose,

- Pass condition expression to `[]` operation.
- Use the `query()` function.
- Use the `eval()` function.

# Performance comparison

When the data size is small, it doesn’t matter which one you choose, but the `[]` operation may be faster.  If the data size is large or the filter condition is complicated, you should choose it carefully. Below we will show the performance comparison of these methods over a relatively large data set.

Because the performance comparison varies greatly in different environments, the result below is just a reference, which is the result of my PC.

>**Performance comparison**
>
>Running time of df[(df["a"] % 2==0) & (df["b"] > 5)] is 0.0235804370s
>
>Running time of df.query("a % 2 == 0 & b > 5") is 0.0111912069s
>
>Running time of df[df.eval('a % 2 == 0 & b > 5')] is 0.0037667719s

From these results, you may notice that `eval` is the fastest, and `[]`operation is the slowest.

Here is the complete code so you can have a try.

In this lesson, we show how to choose the method to select data.

Introduction

Basic Operations on DataFrames

Other topics

Improve performance

Optimized Data Filtering

Performance comparison