Term-Based and Document-Based Indexing

Introduction

A few common indexing techniques used in text preprocessing include term-based indexing, document-based indexing, inverted indexing, and positional indexing. These techniques use different approaches to index text data based on keywords, phrases, or other relevant metadata and enable efficient searching, classification, and analysis of extensive collections of text data.

Term-based indexing

Term-based indexing involves indexing documents based on individual terms or words that appear in the documents. By associating each term with a list of document identifiers where it occurs, we efficiently retrieve documents based on specific query terms. One of the advantages of term-based indexing is its fast and efficient retrieval of relevant documents containing the query terms. However, this type of indexing can be memory-intensive as it also requires the application of text preprocessing steps like tokenization, normalization, and stemming to handle variations in term spellings or word forms. In the example below, let’s see how to apply term-based indexing using Python. We’ll read reviews from a CSV file, tokenize them into words, remove common English stopwords, and then create an inverted index that maps each term to the list of review IDs in which it appears.