Replication Kit - Roseau: Fast, Accurate, Source-based Breaking Change Analysis in Java

Contains all artefacts for our publication: Roseau: Fast, Accurate, Source-based Breaking Change Analysis in Java @ ICSME 2025

Organization

This repository is organized as follows. Subdirectories contain specific instructions for running the different tools and scripts.

• roseau: This directory contains the source code of Roseau as well as intructions to compile it and execute its test suite. First, compile and install Roseau locally with mvn clean install.
• accuracy-dataset: This directory contains our refined and extend version of Jezek and Dietrich's benchmark
• libraries-dataset: This directory contains the scripts we used to collect popular Java libraries from Maven Central in Section V-B
• results: This directory contains the pre-computed accuracy, performance, and longitudinal results together with Jupyter notebooks presenting the numbers and plots we use in the paper
• benchmark: This directory contains the analysis pipeline that we use in Section V, including:
  • Running Roseau, Japicmp, and Revapi on the improved accuracy dataset and collecting their accuracy (Section V-A)
  • Running the JMH benchmarks to measure their runtime performance (Section V-B)
  • Running the preliminary experiments and the full historical analysis of Guava, h2database, and commons-lang (Section V-C)

Paper abstract

Understanding API evolution and the introduction of breaking changes (BCs) in software libraries is essential for library maintainers to manage backward compatibility and for researchers to conduct empirical studies on software library evolution. In Java, tools such as JAPICMP and REVAPI are commonly used to detect BCs between library releases, but their reliance on binary JARs limits their applicability. This restriction hinders large-scale longitudinal studies of API evolution and fine-grained analyses such as commit-level BCs detection.

In this paper, we introduce ROSEAU, a novel static analysis tool that constructs technology-agnostic API models from library code equipped with rich semantic analyses. API models can be analyzed to study API evolution and compared to identify BCs between any two versions of a library (releases, commits, branches, etc.). Unlike traditional approaches, ROSEAU can build API models from either source code or bytecode and is optimized for large-scale longitudinal analyses of library histories.

We assess the accuracy, performance, and suitability of ROSEAU for longitudinal studies of API evolution, using JAPICMP and REVAPI as baselines. We extend and refine an established benchmark of BCs and show that ROSEAU achieves higher accuracy (F1 = 0.99) than JAPICMP (F1 = 0.86) and REVAPI (F1 = 0.91). We analyze 60 popular libraries from Maven Central and find that ROSEAU delivers excellent performance, detecting BCs between versions in under two seconds, including in libraries with hundreds of thousands of lines of code. We further illustrate the inadequacy of JAPICMP and REVAPI for longitudinal studies and the novel analysis capabilities offered by ROSEAU by tracking the evolution of Google’s Guava API and the introduction of BCs over 14 years and 6,839 commits, reducing analysis times from a few days to a few minutes.