Introducing Tokenization

Tokenization is the first step in a text processing pipeline. It is always the first operation because all the other operations require the tokens.

Tokenization means splitting the sentence into its tokens. A token is a unit of semantics. You can think of a token as the smallest meaningful part of a piece of text. Tokens can be words, numbers, punctuation, currency symbols, and any other meaningful symbols that are the building blocks of a sentence. The following are examples of tokens: