Artifact of Large Language Model Based Mutations in Genetic Improvement (Journal Version)
Creators
-
1.
University of Stirling
-
2.
University College London
-
3.
King's College London
-
4.
Johannes Gutenberg University Mainz
Description
Context
This is the artifact of the Large Language Model Based Mutations in Genetic Improvement (Journal Version) paper, following our preliminary work at SSBSE 2023 paper with additional data and functionality.
In our preliminary work, we explored the feasibility of combining the Gin Java GI toolkit with OpenAI LLMs in order to generate an edit for the JCodec tool. In the Journal version, we extend this investigation involving three LLMs, three types of prompts, and five real-world software projects. We sample the edits at random, as well as using local search. We share here the code, the data and all the logs collected during our experiments.
Files
- The code of gin with support for LLMs queries: gin-llm-llm.zip.
- A script to restart the Ollama service, following our investigation of why some prompts failed in Section 5.3: restart-ollama.py.
- The logs recording the output of the edits during ```gin``` run with each of the projects: gin_executions_allstderrs.zip.
- We further investigated this log manually to pinpoint possible reasons for some prompts failing to suggest a good patch, in Section 5.3.
Note: The code, LLMs prompt and experimental infrastructure, data from the evaluation, and results are available as open source here. The code is also under the ‘llm’ branch of github.com/gintool/gin (commit f2f6e10; branched from master commit 2359f57 pending full integration with Gin).
Experimental Setups
We evaluated our idea on the following targets:
| Project | URL | Branch |
| JCodec | github.com/jcodec/jcodec | master (7e52834) |
| JUnit4 | github.com/junit-team/junit4 | r4.13.2 |
| GSON | github.com/google/gson | gson-parent-2.10.1 |
| commons-net | github.com/apache/commons-net | rel/commons-net-3.10.0 |
| karate | github.com/karatelabs/karate | v1.4.1 |
Tested on the following machines/Spec.: 17.0.9 & 3.9.6 & 0.1.27
| Project, Search (LLM) | Machine Specification | Java Version | Maven Version | Ollama Version |
| JCodec, RS (OpenAI, Mistral) | AMD Threadripper 3990x, 64C/128T, 128GB, Titan RTX | 17.0.10 | 3.9.6 | 0.1.28 |
| JCodec, RS (tinydolphin) | Intel Xeon W-2245, 8C/16T, 128GB, RTX 2080 TI | 17.0.8 | 3.9.6 | 0.1.27 |
| JCodec, LS (*) | Intel Xeon 2620v3, 12C/24T, 32GB, Titan X | 17.0.9 | 3.9.0 | 0.1.24 |
| GSON, RS (OpenAI, Mistral) | AMD Threadripper 3990x, 64C/128T, 128GB, Titan RTX | 17.0.10 | 3.9.6 | 0.1.28 |
| GSON, RS (tinydolphin) | Intel Xeon W-2245, 8C/16T, 128GB, RTX 2080 TI | 17.0.8 | 3.9.6 | 0.1.27 |
| GSON, LS (*) | Intel Xeon W-2245, 8C/16T, 128GB, RTX 2080 TI | 17.0.8 | 3.9.6 | 0.1.27 |
| JUnit4, RS & LS (*) | Intel Xeon 2620v3, 12C/24T, 32GB, Titan X | 17.0.9 | 3.9.0 | 0.1.24 |
| commons-net, RS & LS (*) | Intel Xeon 2620v4, 16C/32T, 32GB | 17.0.9 | 3.9.6 | 0.1.27 |
| karate, RS & LS (*) | Intel Xeon 2620v3, 12C/24T, 32GB, Titan X | 17.0.9 | 3.9.0 | 0.1.24 |
In the table: we specified each machine's specifications for each experiment. The target project describes an experiment, the search type (local search (LS) or random search (RS)) and the LLM model used. An asterisk means the specification refers to all LLM models used.
We employed Ollama in its CPU mode to prompt from TinyDolphin (0f9dd11f824c, 637MB) and Mistral (61e88e884507, 4.1GB). We used ChatGPT 3.5.