The Continuous Bag-of-Words Algorithm

How the CBOW model works

The continuous bag-of-words (CBOW) model works in a similar way to the skip-gram algorithm, with one significant change in the problem formulation. In the skip-gram model, we predict the context words from the target word. However, in the CBOW model, we predict the target word from contextual words. Let’s compare what data looks like for the skip-gram algorithm and the CBOW model by taking the example sentence:

The dog barked at the mailman.

For the skip-gram algorithm, the data tuples—(input word, output word)—might look like this:

(dog, the), (dog, barked), (barked, dog), and so on.

For CBOW, the data tuples would look like the following:

([the, barked], dog), ([dog, at], barked), and so on.

Consequently, the input of the CBOW has a dimensionality of $2 × m × D,$ where $m$ is the context window size, and $D$ is the dimensionality of the embeddings. The conceptual model of CBOW is shown in the figure below: