Comparative Analysis of RWKV Linear and Softmax Attention for Zero-Shot Cross-Domain Semantic Textual Similarity

This paper investigates the efficacy of RWKV, a novel language model architecture known for its linear attention mechanism, for generating sentence embeddings in a zero-shot setting. I conduct a layer-wise analysis to evaluate the semantic similarity captured by embeddings from different hidden layers of a pre-trained RWKV model. The performance is assessed on the Microsoft Research Paraphrase Corpus (MRPC) dataset using Spearman correlation and compared against a GloVe-based baseline. My results indicate that while RWKV embeddings capture some semantic relatedness, they underperform compared

Research goal: How does RWKV's linear attention mechanism compare to softmax attention in zero-shot cross-domain semantic textual similarity accuracy on the STS Benchmark and SICK-R datasets?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 8.2/10.