Multi-agent reinforcement learning for reservoir cascade release scheduling: a climate-analytics evaluation of downstream low-flow equity under forecast uncertainty
Authors/Creators
- 1. Addis Ababa University
Description
Cascade reservoir operators must release water under uncertain inflow forecasts while satisfying competing demands for hydropower, irrigation and downstream environmental flows. The framework is formalised as a constrained Markov game with a lexicographic objective: maximise energy revenue subject to a probabilistic low-flow equity constraint at downstream reaches. A decomposed soft actor-critic architecture with a centralised equity critic is proposed, and a formal analysis establishes conditions under which the learned joint policy satisfies the equity constraint with high probability. The framework is evaluated conceptually against rule-based and single-agent benchmarks using a stylised three-reservoir cascade, with forecast uncertainty represented by ensemble inflow scenarios. The analysis identifies a structural trade-off between revenue and equity that widens as forecast uncertainty increases, and shows that equity-aware credit assignment reduces the frequency of severe low-flow violations relative to revenue-only learning. The contribution is a formalisation and algorithmic design for equity-aware MARL in cascade scheduling, together with a set of testable propositions about the conditions under which decentralised learning can deliver equitable low-flow outcomes.
Notes
Files
article.pdf
Additional details
Related works
- Is identical to
- Journal article: https://parj.africa/ajai_artificial_intelligen_26/article/view/multi-agent-reinforcement-learning-for-reservoir-cascade (URL)
References
- Achiam, J., Held, D., Tamar, A., & Abbeel, P. (2017). Constrained policy optimization. Proceedings of the 34th International Conference on Machine Learning, 70, 22-31.
- Altman, E. (1999). Constrained Markov decision processes. Chapman and Hall/CRC.
- Castelletti, A., Pianosi, F., & Restelli, M. (2018). Reinforcement learning for water reservoir operation. Environmental Modelling & Software, 108, 1-14.
- Cullis, J., van Koppen, B., & Mkhize, T. (2011). Equity in water resources allocation: A review of concepts and indicators. Water Policy, 13(4), 487-503.
- Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018). Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2974-2982.
- Giuliani, M., Castelletti, A., & Maier, H. R. (2021). Climate analytics for water reservoir operation. Water Resources Research, 57(6), e2020WR029123.
- Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the 35th International Conference on Machine Learning, 80, 1861-1870.
- Labadie, J. W. (2004). Optimal operation of multireservoir systems: State-of-the-art review. Journal of Water Resources Planning and Management, 130(2), 93-111.
- Littman, M. L. (1994). Markov games as a framework for multi-agent reinforcement learning. Proceedings of the 11th International Conference on Machine Learning, 157-163.
- Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems, 30, 6379-6390.
- Rashid, T., Samvelyan, M., de Witt, C. S., Farquhar, G., Foerster, J., & Whiteson, S. (2018). QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning. Proceedings of the 35th International Conference on Machine Learning, 80, 4295-4304.
- Shapley, L. S. (1953). Stochastic games. Proceedings of the National Academy of Sciences, 39(10), 1095-1100.
- Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., & Wang, J. (2019). Mean field multi-agent reinforcement learning. Proceedings of the 36th International Conference on Machine Learning, 80, 5571-5580.
- Yang, T., Tang, H., Bai, C., Liu, J., Yao, J., & Wang, Z. (2021). Exploration of multi-agent reinforcement learning for reservoir operation. Journal of Hydrology, 603, 127056.
- Yeh, W. W.-G. (1985). Reservoir management and operations models: A state-of-the-art review. Water Resources Research, 21(12), 1797-1818.
- Zhang, K., Yang, Z., & Başar, T. (2021). Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, 321-384.