Scaling Distilled Sentence Embedding Models with Linear Attention Beyond 512 Tokens on GLUE STS-B

Recent state-of-the-art natural language understanding models, such as BERT and XLNet, score a pair of sentences (A and B) using multiple cross-attention operations - a process in which each word in sentence A attends to all words in sentence B and vice versa. As a result, computing the similarity between a query sentence and a set of candidate sentences, requires the propagation of all query-candidate sentence-pairs throughout a stack of cross-attention layers. This exhaustive process becomes computationally prohibitive when the number of candidate sentences is large. In contrast, sentence em

Research goal: How do distilled sentence embedding models with linear attention layers scale in terms of training convergence speed and final Pearson correlation on the GLUE STS-B task when increasing sequence length beyond 512 tokens?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.5/10.