Search⌘ K
AI Features
Home
Courses
Natural Language Processing with TensorFlow

CNNs for Sentence Classification: Building a Tokenizer

Explore how to build a tokenizer that converts text into numerical sequences for sentence classification with CNNs. Understand how to handle variable sentence lengths using padding and truncation, and prepare input data effectively for TensorFlow models.

We'll cover the following...

Implementation: Building a tokenizer

Now it’s time to build a tokenizer that can map words to numerical IDs: 

from tensorflow.keras.preprocessing.text import Tokenizer
# Define a tokenizer and fit on train data
tokenizer = Tokenizer()
tokenizer.fit_on_texts(train_df["question"].tolist())

Here, we simply create a Tokenizer object and use the fit_on_texts() function to train it on the training corpus. In this process, the tokenizer will map words in the vocabulary to IDs. We’ll convert all of the train, validation, and test inputs to sequences of word IDs. ...