Replication Package for the Paper: “Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis”

This replication package includes the raw data, questionnaire answers, and a Python notebook needed for reproducing the results detailed in the paper titled “Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis.”

Repository Structure

1. Scenarios: Contains an Excel file encompassing all 141 scenarios collected (in Italian).
2. Training and Validation Messages: Includes the jsonl files necessary for fine-tuning the model.
3. Testing Messages and Ground Truth: Contains the messages utilized for testing the models.
4. Results: Contains Excel files with the responses from the 2 human experts and the 5 model as well as the review of the 3 human reviewer.
5. Tables: Contains the full Wilcoxon Test Results for H01 and H02 as well as the raw RQs results.

Replication Process

To replicate the results of our study, open the provided Python Notebook in Google Colab and follow the instructions to seamlessly reproduce the results.

Instructions for Use

To utilize this replicability package, refer to the steps outlined in the notebook file.

Remarks

If you encounter any issues or have any questions, please reach out to the authors of the paper. We will be glad to assist you!