Text Preprocessing with Python/

...

/

Types of Irrelevant Text Data

Types of Irrelevant Text Data

Learn about different types of irrelevant text data.

We'll cover the following...

Introduction

Irrelevant text data refers to words, phrases, or sentences in the larger text context that are unimportant during analysis. This makes dealing with irrelevant text data an essential step in text preprocessing. It can improve the accuracy and efficiency of NLP tasks, such as sentiment analysis, topic modeling, and document classification. In the following sections, we’ll look at some examples of irrelevant text data and how to remove them using various NLP libraries.

Stopwords

In the introductory chapter of this course, we defined what stopwords are. These are common words that don’t carry much meaning or contribute to understanding the text. Let’s practice removing them using the NLTK library.

Press + to interact
Python 3.8
Files
review_id,review_text,rating
1,"Great product! I highly recommend it.",5
2,"The quality of this item is excellent.",4
3,"Not satisfied with the purchase. The product arrived damaged.",2
4,"Amazing service! Prompt delivery and great customer support.",5
5,"This product is a complete waste of money.",1
6,"Barack Obama was an American President.",5
7,"I met John Doe yesterday. He was very friendly.",4
8,"Jane Smith lives in New York.",3
9,"Hawaii is a beautiful place for vacation.",5
10,"United States is my dream destination.",4

Let’s review the code line by line:

  • Lines 1–5: We import the nltk library for NLP, the pandas library, and stopwords from the corpus module in the NLTK library. We download the stopwords corpus and set quiet=True so we don’t get an installation message in the output. Later, we create a set of English stopwords.

  • Line 7: We read the CSV file named reviews.csv and store its content in a DataFrame called df. ...