SymboleoLLM

This repository contains all the files used in the experiments described in the paper on generating Symboleo specifications from natural language contracts using various LLMs. The structure is as follows:

Prompts/
Contains the 38 text files (test_case_<number>.txt) representing the prompts used in the experiments.
Each prompt corresponds to a specific configuration tested.


Specifications/
Includes the Symboleo specifications generated by the tested LLMs.
Subdirectories are organized by LLM :
* ChatGPT 4.o/
* Claude 3.5 Haiku/
* Gemini 1.5 Pro 002/
* Llama 3.2/
* Mistral_7B/

Each subdirectory contains files (Test_<number>.symboleo) where each is the Symboleo specification generated from the corresponding individual test case (prompt).


generate_prompts.py
The Python script used to create the 38 prompts.
The script automates the generation of prompt templates based on predefined configurations.


metrics_results.xlsx
An Excel file containing the evaluation results of all generated Symboleo specifications.
Includes metrics such as error types, severity, and cross-LLM comparisons.

prompt_grid.csv
Outlines the configurations and attributes of prompts used in the experiments for generating Symboleo specifications.


Usage
Prompts: Use the files in prompts/ as input for LLMs to replicate or modify the experiments.
Specifications: Analyze the generated Symboleo specifications in Specifications/ for each test case and LLM.
Script: Run or adapt the Python script in generate_prompts.py to generate or customize prompts.
Metrics: Review metrics_results.xlsx for detailed performance metrics of the generated specifications.

Contact
{mzito042, aanda, srajp029, damyot, jmylopou}@uottawa.ca
For further details, refer to the paper or contact the authors.