Why Do We Need Filtering?

One of the most commonly performed operations on a DataFrame is filtering. It essentially involves excluding some values to meet a condition or set of conditions.

When to filter a DataFrame?

We may need to filter for several reasons:

• We filter when we need to remove redundant or unnecessary data for some tasks. Suppose we’re working on a machine-learning task to predict customer churn. A customer’s bank account number is redundant in this case and needs to be filtered out.

• Consider a data analyst working at a bank. The bank is planning to announce a promotion for customers that meet certain criteria in terms of account balance, monthly spending, and products (possession of a credit card or checking account). The data analyst needs to filter the data to find customers eligible for the promotion.

• Filtering is also necessary for data cleaning and manipulation. We may want to filter out observations (rows in a DataFrame) that have missing values in some columns.

• Filtering is frequently used in data analysis as well. Suppose we’re grouping customers according to the spending amount. In this case, we filter customers based on the amount spent and then group them.

We can filter in terms of observations (rows) or features (columns). Thankfully, Pandas is quite flexible and efficient for such operations. In this chapter, we’ll cover different methods that can be used to filter both rows and columns in a DataFrame.