Replication package for "Fully Autonomous Programming with Large Language Models"

This repository contains the replication package for the paper "Fully Autonomous Programming with Large Language Models", Vadim Liventsev, Anastasiia Grishina, Aki Härmä, and Leon Moonen, accepted for the 2023 ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO'23). The paper is deposited on arXiv, will be available at the publisher's site, and a copy is included in this repository.

The replication package is archived on Zenodo with DOI: 10.5281/zenodo.7837282. The source code is distributed under the MIT license, the data is distributed under the CC BY 4.0 license.

Organization

The repository is organized as follows:

• Archived source code in the src folder, with a dedicated README.

• Analysis of the results in the analysis folder, with a dedicated README.

• Archive of SEIDR-generated solutions for PSB2 as a git bundle: psb2-solutions.bundle. 

  • These solutions can be inspected by cloning the bundle using git clone psb2-solutions.bundle which will create a folder psb2-solutions with a dedicated README in the master branch.
  • The other branches contain the code that was generated in specific experiments/configurations. List which experiments are available using git branch -r, select one using git checkout <branch>, and look at the iteratively synthesized solution for a problem using git log -p -- {problem}.{cpp/py}. Alternatively, these can be inspected on GitHub.

Citation

If you build on this data or code, please cite this work by referring to the paper:

@inproceedings{liventsev2023:fully,
   title = {Fully Autonomous Programming with Large Language Models},
   author = {Vadim Liventsev and Anastasiia Grishina and Aki Härmä and Leon Moonen},
   booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'23)},
   year = {2023},
   publisher = {ACM}
   doi = {https://doi.org/10.1145/3583131.3590481},
}

External References

• The SEIDR source code is maintained on GitHub.
• The SEIDR-generated solutions for PSB2 are versioned on GitHub as well.