Do Multi-agent Systems Make Better Software? An Empirical Evaluation of LLM-Based Frameworks for Software Development

Large language model (LLM)-based multi-agent frameworks (MAFs) for software development are currently on the rise. Among these, several LLM-based MAFs are particularly gaining traction as they cover multiple software development phases, not just the coding phase. However, the proper benchmarking of these systems is missing in the literature: they are mainly benchmarked on games, they are not properly compared among each other, and the impact of backbone LLMs is not known. Therefore, in this paper, we compare four prominent LLM-based MAFs: MetaGPT, ChatDev, AgileCoder and HyperAgent with 14 different open-weight backbone LLMs on two use cases of different complexities and types (game vs. non-game-related). We perform comparison using different quantitative and qualitative evaluation metrics, on the overall framework level as well as on agent- and phase-level. Our results reveal that performances remarkably differ across use cases, although a promising framework (and LLM combination) can be identified.