A Review on Sample Efficiency Improvement Methods in Reinforcement Learning

Abstract

Reinforcement Learning (RL) faces a significant challenge in sample efficiency due to the need for costly interactions with the environment in real-world applications. This review, by examining 21 main papers and 11 supporting papers, argues that sample efficiency is not merely about reducing the number of interactions, but rather extracting the maximum useful information from each real interaction. In this paper, the reviewed methods are categorized into five levels: structural, policy-temporal, task-preference, data-driven, and semantic. This categorization is a proposal by the authors. The integrated conceptual framework of "Multi-Scale Information Density," along with four levers and a gradual trust-opening schedule, provides a framework for selecting and combining methods. In addition to the categorization, the paper presents a comparative table of 21 studies, three illustrative application scenarios, and a conceptual proposal for quantitatively measuring sample efficiency. This framework has not yet been tested in a single experimental setting.