Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation"
Authors/Creators
- 1. University of the Basque Country (UPV/EHU) - Faculty of Computer Science
- 2. LKS Next
Description
Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation"
- Journal: Science of Computer Programming
- Year: 2026
- Authors: Eneko Pizarro, Maider Azanza, and Beatriz Perez
This repository contains the replication package for the paper "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation".
Repository structure
1) Test suite (resources under test)
These are the classes and functions used for the analysis and metric collection in the paper. The table below lists each tested item, its origin, and the evaluation focus.
| Function/Method | Function Type | Evaluation Focus | Source |
| assemble | Rental Assembler | Complex object interactions, database mocking, multi-layered architecture |
https://github.com/BSPQ18-19/BSPQ19-S1.git |
| isPrime | Prime Checker | Algorithmic logic, exception handling, high cyclomatic complexity, multiple error paths | Software Engineering II (course), EHU/UPV |
| addUser | User Management | Input validation, null handling, database mocking, CRUD operations | https://github.com/jononekin/Forum-assertJ.git |
| getBonus | Bonus Calculator | Nested conditional business rules, external dependencies | https://github.com/jononekin/Forum-assertJ.git |
| isIPV4Valid | IPV4 Validator | Utility function integration, helper function interaction | https://www.geeksforgeeks.org/program-to-validate-an-ip-address/ |
| isStrobogrammic | Strobogrammatic Number | Data structure manipulation (map/dictionary), low leakage risk | https://www.w3resource.com/java-exercises/basic/java-basic-exercise-186.php |
| palindrome | Palindrome Detector | Multi-function integration, coordinated test generation | https://code-exercises.com/programming/hard/4/longest-palindrome-in-word |
| isLeapYear | Leap Year Checker | Boundary conditions, multiple branching paths | Software Engineering II (course), EHU/UPV |
2) Aggregated results
Folder: Aggregated_Results_by_Function_Oct.2025
Comprehensive results for all methods and LLMs tested. Data were collected between March 2024 and October 2025. The results include the metrics described in the evaluation framework section of the paper.
Page index:
- Weights: Metric descriptions, calculation and evaluation methods, and aggregation weights.
- Journal-Summary Published: Aggregated results as reported in the paper.
- Comparison: GitHub test vs Custom Agent: Aggregated results comparing test generation methods (GitHub Copilot `/test` vs Custom Agent prompt).
3) Prompt artifacts
- Prompt_for_Integration_Testing-2025: Final prompt used to generate the test cases.
- Custom_Agent_Prompt_for_Integration_Testing.md: Custom Agent personalization structured prompt used in the measurement method.
4) Sources
- sources.7z: Source code classes and functions from which the test classes shown in the paper were generated.
5) Expert test cases
Folder: Expert_Test_Cases
Curated, human-authored test suites used as reference baselines for the evaluation.
6) Evaluation
Folder: Evaluation
Evaluation artifacts including:
- Evaluation_Consent_Form.pdf: Participant consent form used in the evaluation study.
- Evaluation_Information_Sheet.pdf`: Information sheet provided to evaluation participants.
7) Results and measurements
Folder: Results_and_Measurements
Raw and processed outputs from all measurement campaigns, organized by collection window and target function.
8) Related project
Tetrics PoC: https://github.com/lksnext-ai-lab/tetrics.git
Tetrics is a research prototype implementing a domain-independent, continuous evaluation framework for LLM-powered development tools.
Files
Aggregated_Results_by_Function_Oct.2025.pdf
Files
(95.6 MB)
| Name | Size | |
|---|---|---|
|
md5:fdbc28803a97e703e86a67a39484b112
|
103.5 kB | Preview Download |
|
md5:7e1c3ccbbbbc8f670bcac49ca5fc3f8e
|
9.4 kB | Preview Download |
|
md5:4cd1e91b4553874868a7f8dfc39e7a55
|
190.0 kB | Preview Download |
|
md5:ef934de112680e37319ad19dc2067a7d
|
2.4 MB | Preview Download |
|
md5:d59177d666165b08c5e7304081b4d679
|
77.7 kB | Preview Download |
|
md5:efa6d93bdb285df4fed0c0f9a8c43d46
|
3.7 kB | Preview Download |
|
md5:091b18c5056e583cb9b72314a1668a10
|
82.3 MB | Preview Download |
|
md5:1feae2ae8acd617acd7677f50c3322c6
|
10.5 MB | Download |
Additional details
Software
- Repository URL
- https://github.com/lksnext-ai-lab/tetrics.git