Empirical evaluation of low-rank adaptation for efficient fine-tuning of large language models

This study evaluates Low-Rank Adaptation (LoRA) as a parameter-eficient method for fine-tuning large language models (LLMs), with a focus on the Qwen2.5-3B-Instruct model and the S1.1 dataset. Traditional full fine-tuning of LLMs demands substantial memory and computation, whereas LoRA introduces trainable low-rank matrices into selected layers, significantly reducing the number of trainable parameters while keeping the base model weights fixed.

A series of experiments was conducted on the Baskerville HPC cluster to analyse LoRA’s ezects on memory usage, training time, and model performance under various configurations and GPU settings. Results indicate that LoRA reduces trainable parameters by over 99% and lowers memory usage by up to 34% on a single GPU, with further reductions observed in multi-GPU environments. Although residual memory usage increases due to LoRA’s adapter layers, the overall memory footprint remains considerably lower than that of full fine-tuning.

Performance comparisons reveal that carefully chosen LoRA configurations can approach baseline accuracy, especially when higher rank values and optimised learning rates are used. However, slower convergence and early loss plateaus were observed in some settings. These findings highlight LoRA as an ezective solution for enabling scalable fine-tuning of LLMs in resource-constrained environments.