Variance Reduction Techniques in Deep Reinforcement Learning with Noisy Environments

Deep Reinforcement Learning (DRL) has demonstrated remarkable success in various domains. However, its performance often degrades significantly in environments characterized by high levels of stochasticity or noisy observations. This paper investigates the impact of variance in gradient estimates on the stability and convergence of DRL algorithms. We explore and compare several variance reduction techniques, including Generalized Advantage Estimation (GAE), Proximal Policy Optimization (PPO) clipping, and a novel approach incorporating a learned noise model within the critic network. The efficacy of these methods is evaluated in simulated environments with varying levels of noise, highlighting their strengths and limitations in mitigating the adverse effects of high variance.