Can Agents reconstruct microservice architecture?

Minor Correction (June 2026): Restored a missing return in one helper function in evaluation.py (calculate_metrics) and removed a stale internal version label from the docstring of scripts/plot_topology.py (comment-only level change). Reported results are unaffected. They are reproducible from the precomputed CSVs in results/analysis/RQ1/ and results/analysis/RQ2/, and from the included .jmp files.

This replication package contains all artifacts required to reproduce the empirical study on LLM-based reconstruction of microservice architectures from source code.

Package Contents

• Agent implementation (src/agent/)
  A tool-calling agent that explores software repositories using three tools (list_directory, read_file, and search_text) and generates a structured JSON representation of the recovered architecture, including components, connections, and endpoints.

• Evaluation pipeline (src/evaluation/)
  Scripts for computing precision, recall, and F1-score against the ground-truth architectures, including fuzzy matching for components and endpoints.

• Subject systems (data/applications/)
  Seventeen open-source Spring Boot microservice applications used in the empirical study.

• Ground-truth architectures (data/groundtruth_textual/)
  Manually validated reference architectures for all subject systems.

• Raw evaluation results (results/analysis/analysis_20runs_raw.csv)
  Per-run evaluation scores from 20 runs × 2 models × 17 applications, corresponding to the data used in the published analysis.

• Statistical analysis inputs and JMP templates (results/analysis/RQ1/ and results/analysis/RQ2/)
  Input datasets and JMP analysis templates for the Anderson–Darling, Friedman, and Wilcoxon statistical tests.

• Visualization script (scripts/plot_topology.py)
  Script used to regenerate the architectural element type-effect boxplot reported in the study.

Setup (Python 3.11+)

python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp keys.env.example keys.env   # add API credentials to run the agent end-to-end

Reproducing the Published Results

Reproducing the analysis without re-running the LLM

The package includes all per-run evaluation results required to reproduce the published statistical analysis and figures:

python scripts/build_analysis_inputs.py   # regenerate JMP input CSVs
python scripts/plot_topology.py           # regenerate the topology-effect boxplot

Full reproduction from scratch

Re-running the full experiment requires valid LLM API credentials:

python scripts/batch_run.py --runs 20 --providers gpt-4o llama4_16x17b --temperature 0.3

See README.md for complete documentation, repository structure, and detailed script options.

License

MIT License.

CC BY 4.0 license.