Comparison of FedKRSO and Standard LoRA FL on SuperGLUE WSC and RTE

Fine-tuning large language models requires high computational and memory resources, and is therefore associated with significant costs. When training on federated datasets, an increased communication effort is also needed. For this reason, parameter-efficient methods (PEFT) are becoming increasingly important. In this context, very good results have already been achieved by fine-tuning with low-rank adaptation methods (LoRA). The application of LoRA methods in Federated Learning, and especially the aggregation of adaptation matrices, is a current research field. In this article, we propose a n

Research goal: How does FedKRSO compare to standard LoRA FL in terms of convergence speed and final accuracy on the WSC and RTE subsets of SuperGLUE?

Autonomous synthesis report generated by Assignee Research. Tribunal consensus score: 8.5/10.