Artifact of Large Language Model Based Mutations in Genetic Improvement (Journal Version)

Context

This is the artifact of the Large Language Model Based Mutations in Genetic Improvement (Journal Version) paper, following our preliminary work at SSBSE 2023 paper with additional data and functionality.

In our preliminary work, we explored the feasibility of combining the Gin Java GI toolkit with OpenAI LLMs in order to generate an edit for the JCodec tool. In the Journal version, we extend this investigation involving three LLMs, three types of prompts, and five real-world software projects. We sample the edits at random, as well as using local search. We share here the code, the data and all the logs collected during our experiments.

Files

• The code of gin with support for LLMs queries: gin-llm-llm.zip.
• A script to restart the Ollama service, following our investigation of why some prompts failed in Section 5.3: restart-ollama.py.
• The logs recording the output of the edits during ```gin``` run with each of the projects: AggregratedResultFiles-20240815.zip, and random-sampler-raw-data.zip. These files include the stderr and logs files with .txt files containing the prompt and response, and indication of where was the compilation error (if occurred). The time information is here: RunTimes.csv.
  • We further investigated this log manually to pinpoint possible reasons for some prompts failing to suggest a good patch, in Section 5.3.
• To select the LLMs for our experiments, we did a set of small experiments to see if we got sensible answers to our prompts. Some results were odd, and hence, we did not use some of the models we tried initially. See some of our logs here: Local-LLM-Results-20240805T142525Z-001.zip.

Note: The code, LLMs prompt and experimental infrastructure, data from the evaluation, and results are available as open source here. The code is also under the ‘llm’ branch of github.com/gintool/gin (commit f2f6e10; branched from master commit 2359f57 pending full integration with Gin).

Experimental Setups

We evaluated our idea on the following targets:

Tested on the following machines/Spec.: 17.0.9 & 3.9.6 & 0.1.27

In the table: we specified each machine's specifications for each experiment. The target project describes an experiment, the search type (local search (LS) or random search (RS)) and the LLM model used. An asterisk means the specification refers to all LLM models used.

We employed Ollama in its CPU mode to prompt from TinyDolphin (0f9dd11f824c, 637MB) and Mistral (61e88e884507, 4.1GB). We used ChatGPT 3.5.