Strategic Insights: Evaluating Large Language Models' Decision-Making in Multi-Player Game-Theoretic Environments
Description
⚠️This research is conducted primarily by AI, where human author(s) serve as advisor(s). There are significant flaws found in this AI-generated research, and footnoted with "Human author note:". The purpose of this study is to identify and understand the opportunities and limitations of AI in scientific discovery.
⚠️Use with caution.
Abstract
Large Language Models (LLMs) excel in language tasks but their strategic decision-making in interactive, multi-agent scenarios—critical for applications like negotiation systems or social simulations—remains understudied. This paper examines twelve anonymized LLMs in six multi-player game theory scenarios, encompassing cooperative, betraying, and sequential categories, with ten agents per instance across repeated rounds and multiple runs. We propose the Strategic Rationality Score (SRS), a novel composite metric normalizing deviations from Nash equilibria across games, enabling quantitative benchmarking of LLM rationality. Our findings reveal inconsistent equilibrium-seeking behavior, weak correlations with architectural features like parameter size, and minimal adaptation over interactions, suggesting inherent limitations in opponent modeling and long-term reasoning. These results contrast with expectations from scaling laws and highlight biases toward short-term gains. Contributions include SRS for cross-game evaluation, large-scale multi-player simulations (360 instances), and linkages to LLM traits, advancing AI behavioral analysis for safer multi-agent deployments. Code and data are available at https://github.com/jkli898/AI_Generated_Research_LLM_Game_Theory for review.