MooBench: A micro-benchmark for performance overhead measurement of observability tools

EventType

Research Software Publication

Abstract

Understanding the runtime behavior of software is inherently difficult due to the unpredictability of the software's behavior itself and the non-determinism of underlying layers, such as Just-In-Time (JIT) compilation in virtual machines, operating system scheduling, and CPU frequency scaling. Observability tools aim to answer questions regarding runtime behavior of software, such as "How much time did this request take?" or "How often did method A call method B?". These questions are answered using telemetry data, i.e., measurement data that is obtained from the code execution. To collect telemetry data, additional code needs to be executed, which introduces overhead. This overhead affects both system performance and the accuracy of the measurements themselves. The MooBench microbenchmark measures this overhead and contains factorial experiments that facilitate breaking down this overhead into its root causes.

The MooBench benchmark was originally developed to examine the performance overhead of the Kieker observability framework in Java and was extended as a general overhead measurement microbenchmark for various observability tools, currently within the Java and Python ecosystem. In this paper, we describe why it is needed, how it is structured, and how it is used in research.

Prerequisites

MooBench requires knowledge of performance benchmarking, in particular, the Java language and its virtual machine to understand the analysis of performance details produced by the MooBench framework. MooBench focuses on measuring the monitoring overhead of observability tools, where an observability agent is woven into a runtime of a micro-benchmark application for its observability task – monitoring method call traces.

Outcomes

MooBench provides a ready-made infrastructure to add an observability agent, and compare its runtime performance with other agents added to the framework. For research software engineers in academia and industry, the analysis results provide the basis for improving new tools, and the MooBench framework itself exposes the core performance criteria for software runtimes.

Acknowledgements

We acknowledge contributions over the years from André van Hoorn, Jan Waller, Reiner Jung, Christian Wulf, Florian Fittkau, Ferit Altin, Serafim Simonov, and Nils Christian Ehmke. This research is funded by the Deutsche Forschungsgemeinschaft (DFG – German Research Foundation), grant no. 528713834, and UK Research and Innovation (UKRI) through the UKRI Metascience Research Grants program (Reference S26368).