SpanBERT: Exploring Architecture

In SpanBERT, we mask a contiguous span of tokens in the sentence. Let $x_s$ and $x_e$ be the start and end position of the masked tokens, respectively. We feed the tokens to SpanBERT and it returns the representation of all the tokens. The representation of token $i$ is represented as $R_i$. The representation of the tokens in the span boundary is denoted as $R_{s-1}$ and $R_{e+1}$ ...