Published February 13, 2026 | Version v1

Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation"

  • 1. University of the Basque Country (UPV/EHU) - Faculty of Computer Science
  • 2. LKS Next

Contributors

  • 1. ROR icon University of the Basque Country

Description

Replication Package: "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation"

  • Journal: Science of Computer Programming
  • Year: 2026
  • Authors: Eneko Pizarro, Maider Azanza, and Beatriz Perez

This repository contains the replication package for the paper "Beyond the Hype: Enabling Informed LLM Adoption in Industry Through Systematic Evaluation".

Repository structure

1) Test suite (resources under test)

These are the classes and functions used for the analysis and metric collection in the paper. The table below lists each tested item, its origin, and the evaluation focus.

Function/Method Function Type Evaluation Focus Source
assemble Rental Assembler Complex object interactions, database mocking, multi-layered architecture

https://github.com/BSPQ18-19/BSPQ19-S1.git

isPrime Prime Checker Algorithmic logic, exception handling, high cyclomatic complexity, multiple error paths Software Engineering II (course), EHU/UPV
addUser User Management Input validation, null handling, database mocking, CRUD operations https://github.com/jononekin/Forum-assertJ.git
getBonus Bonus Calculator Nested conditional business rules, external dependencies https://github.com/jononekin/Forum-assertJ.git
isIPV4Valid IPV4 Validator Utility function integration, helper function interaction https://www.geeksforgeeks.org/program-to-validate-an-ip-address/
isStrobogrammic Strobogrammatic Number Data structure manipulation (map/dictionary), low leakage risk https://www.w3resource.com/java-exercises/basic/java-basic-exercise-186.php
palindrome Palindrome Detector Multi-function integration, coordinated test generation https://code-exercises.com/programming/hard/4/longest-palindrome-in-word
isLeapYear Leap Year Checker Boundary conditions, multiple branching paths Software Engineering II (course), EHU/UPV

 

2) Aggregated results

Folder: Aggregated_Results_by_Function_Oct.2025

Comprehensive results for all methods and LLMs tested. Data were collected between March 2024 and October 2025. The results include the metrics described in the evaluation framework section of the paper.

Page index:

  • Weights: Metric descriptions, calculation and evaluation methods, and aggregation weights.
  • Journal-Summary Published: Aggregated results as reported in the paper.
  • Comparison: GitHub test vs Custom Agent: Aggregated results comparing test generation methods (GitHub Copilot `/test` vs Custom Agent prompt).

3) Prompt artifacts

  • Prompt_for_Integration_Testing-2025: Final prompt used to generate the test cases.
  • Custom_Agent_Prompt_for_Integration_Testing.md: Custom Agent personalization structured prompt used in the measurement method.

4) Sources

  • sources.7z: Source code classes and functions from which the test classes shown in the paper were generated.

5) Expert test cases

Folder: Expert_Test_Cases

Curated, human-authored test suites used as reference baselines for the evaluation.

6) Evaluation

Folder: Evaluation

Evaluation artifacts including:

  • Evaluation_Consent_Form.pdf: Participant consent form used in the evaluation study.
  • Evaluation_Information_Sheet.pdf`: Information sheet provided to evaluation participants.

7) Results and measurements

Folder: Results_and_Measurements

Raw and processed outputs from all measurement campaigns, organized by collection window and target function.

8) Related project

Tetrics PoC: https://github.com/lksnext-ai-lab/tetrics.git

Tetrics is a research prototype implementing a domain-independent, continuous evaluation framework for LLM-powered development tools.

Files

Aggregated_Results_by_Function_Oct.2025.pdf

Files (95.6 MB)

Name Size
md5:fdbc28803a97e703e86a67a39484b112
103.5 kB Preview Download
md5:7e1c3ccbbbbc8f670bcac49ca5fc3f8e
9.4 kB Preview Download
md5:4cd1e91b4553874868a7f8dfc39e7a55
190.0 kB Preview Download
md5:ef934de112680e37319ad19dc2067a7d
2.4 MB Preview Download
md5:d59177d666165b08c5e7304081b4d679
77.7 kB Preview Download
md5:efa6d93bdb285df4fed0c0f9a8c43d46
3.7 kB Preview Download
md5:091b18c5056e583cb9b72314a1668a10
82.3 MB Preview Download
md5:1feae2ae8acd617acd7677f50c3322c6
10.5 MB Download

Additional details