Merging and Splitting Tokens
Let’s see how we can tokenize multiword expressions and multiword entities.
We'll cover the following...
Overview
We extracted the name entities in the previous section, but what if we want to unite or split multiword named entities? And what if the tokenizer performed this not so well on some exotic tokens, and we want to split them by hand? In this lesson, we'll cover a very practical remedy for our multiword expressions, multiword named entities, and typos.
doc.retokenize is the correct tool for merging and splitting the spans. Let's see an example of retokenization by merging a multiword named entity, as follows:
Access this course and 1400+ top-rated courses and projects.