Easy, collaborative benchmarking with the extended benchmarking automation tool xbat

Research software development increasingly requires balancing scientific objectives with performance optimization, reproducibility, and resource efficiency. Yet many RSEs face barriers to accessing the hardware expertise and tools necessary for performance analysis especially in HPC environments where understanding low-level metrics is essential for sustainable software development. There are many benchmarking and profiling tools available including LIKWID, perf, NVIDIA Nsight Systems, HPCToolkit, TAU and VTune. Some of these tools require complex setups, specialised hardware, or have a demanding initial learning phase. Moreover, accessing and interpreting performance metrics can be challenging and error prone, particularly for RSEs without in-depth hardware expertise. To address these challenges, we present xbat (extended benchmarking automation tool) as an easy-to-use, universally applicable, powerful tool to automate (continuous) benchmarking and simplify performance analysis. xbat unifies the benchmarking lifecycle from job configuration through analysis in a single, accessible platform. Developed by MEGWARE Computer Vertrieb und Service GmbH, xbat has been fully open-source since April 2025 and remains under active development, with extensive online documentation, guides, and a public demo instance.

This talk outlines the architecture, capabilities and features of xbat,
demonstrating its ability to simplify (continuous) benchmarking and assist RSEs in improving performance of their research software. The entire benchmarking workflow, which encompasses job configuration (with support for parameter studies), submission, data collection, and analysis, can be executed seamlessly through a user-friendly web interface. For users preferring command-line workflows or automation, xbat also supports job submission via the Slurm CLI and provides a comprehensive API for programmatic integration. A containerised deployment model and integration with user management tools allow for a smooth integration of xbat into HPC infrastructures. In addition, xbat supports collaboration by allowing RSEs to share software and job configurations as well as benchmarking results. Including CPU, memory, I/O, GPU, FPGA and energy usage, low- and high-level performance metrics are collected, totaling up to 140 depending on hardware availability. With LIKWID as the primary data source, xbat performs measurements at the thread and core level, then dynamically aggregates this data across NUMA domains, sockets, nodes, and multi-node jobs. This multi-level aggregation enables performance analysis at various scales ranging from individual components to entire systems providing insights specifically tailored for multiple use cases. Moreover, xbat performs micro-benchmarks to measure the bandwidth of caches and DRAM as well as floating-point operations per second (FLOPS). By comparing these measurements with the corresponding results of the research software, one can assess, via roofline plots, whether the application is memory- or compute-bound. The xbat web interface provides RSEs with a central hub for benchmarking, as well as performance data presentation through interactive, customisable graphs and detailed statistics. Results can be easily integrated into reports or presentations, with graphs exportable as images and their underlying data available in CSV or JSON format. This combination of visual and statistical insights helps RSEs to identify inefficiencies and evaluate the influence of various configurations and parameter settings.
Future enhancements include extended metric collection, increased focus on analysis and statistics, and automated bottleneck and anomaly detection.