Fast Q-learning for Improved Finite Length Performance of Irregular Repetition Slotted ALOHA

In this paper, we study the problem of designing
adaptive Medium Access Control (MAC) solutions for wireless
sensor networks (WSNs) under the Irregular Repetition Slotted
ALOHA (IRSA) protocol. In particular, we optimize the degree
distribution employed by IRSA for finite frame sizes. Motivated
by characteristics of WSNs, such as the restricted computational
resources and partial observability, we model the design of IRSA
as a Decentralized Partially Observable Markov Decision Process
(Dec-POMDP). We have theoretically analyzed our solution in
terms of optimality of the learned IRSA design and derived
guarantees for finding near-optimal policies. These guarantees
are generic and can be applied in resource allocation problems
that exhibit the waterfall effect, which in our setting manifests
itself as a severe degradation in the overall throughput of
the network above a particular channel load. Furthermore, we
combat the inherent non-stationarity of the learning environment
in WSNs by advancing classical Q-learning through the use of
virtual experience (VE), a technique that enables the update
of multiple state-action pairs per learning iteration and, thus,
accelerates convergence. Our simulations confirm the superiority
of our learning-based MAC solution compared to traditional
IRSA and provide insights into the effect of WSN characteristics
on the quality of learned policies.