Multi-Agent Reinforcement Learning for Cooperative Coded Caching via Homotopy Optimization

Introducing cooperative coded caching into small

cell networks is a promising approach to reducing traffic loads.

By encoding content via maximum distance separable (MDS)

codes, coded fragments can be collectively cached at small-cell

base stations (SBSs) to enhance caching efficiency. However, content

popularity is usually time-varying and unknown in practice.

As a result, cached content is anticipated to be intelligently

updated by taking into account limited caching storage and

interactive impacts among SBSs. In response to these challenges,

we propose a multi-agent deep reinforcement learning (DRL)

framework to intelligently update cached content in dynamic

environments. With the goal of minimizing long-term expected

fronthaul traffic loads, we first model dynamic coded caching as

a cooperative multi-agent Markov decision process. Owing to the

use of MDS coding, the resulting decision-making falls into a class

of constrained reinforcement learning problems with continuous

decision variables. To deal with this difficulty, we custom-build a

novel DRL algorithm by embedding homotopy optimization into

a deep deterministic policy gradient formalism. Next, to empower

the caching framework with an effective trade-off between complexity

and performance, we propose centralized, and partially

and fully decentralized caching controls by applying the derived

DRL approach. Simulation results demonstrate the superior

performance of the proposed multi-agent framework.