There is a newer version of the record available.

Published July 13, 2026 | Version v3
Dataset Open

A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

Authors/Creators

Description

Replication Package for MODELS 2026 Research Track titled - A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

This repository provides a benchmarking pipeline for evaluating Large Language Models (LLMs) on Object Constraint Language (OCL) generation from:

  • UML class diagrams (PlantUML format)
  • Natural language specifications

It includes:

  • Original UML/OCL datasets
  • Natural language specification for each UML model
  • 3 Systematic UML transformations - identifier renaming, attribute refication, and association refication
  • Prompting framework for LLM-based OCL generation

The directory structure is as follows:

├── dataset
│   ├── UML
│   │   ├── Airport
│   │   │   ├── Airport.puml
│   │   │   ├── Airport.use
│   │   │   └── Airport.ocl
│   │   └── EmploymentAgency
│   │
│   ├── Transformed_UML
│   ├── specification.json
│   └── transformed_specification.json      
│
└── transformations
│   ├── __init__.py
│   ├── utils.py
│   ├── uml_parser.py
│   ├── rename_transformation.py
│   ├── attribute_transformation.py
│   ├── association_transformation.py
│   └── runner.py
├── evaluation
│   ├── llm_runner.py
│   ├── prompts.py
│   ├── fine_tuning_LLM_for_OCL_script.ipynb
│   └── config.py
│
└── run.py 

specification.json, and transformed_specification.json contain the natural language specification for each UML model

Fine-Tuning

fine_tuning_LLM_for_OCL_script.ipynb fine-tunes a causal LM on OCL generation using QLoRA (4-bit quantization + LoRA adapters). 

  • Dataset: fpan/text-to-ocl-from-ecore(https://huggingface.co/datasets/fpan/text-to-ocl-from-ecore), split 80/10/10 into train/val/test.
  • Requirements: HuggingFace account with an API token, and a GPU runtime (e.g. Google Colab). To run, open the notebook, replace "HUGGINGFACE_API_TOKEN" with your token, and execute all cells.

Installation


1. Download the artifact archive from: https://doi.org/10.5281/zenodo.20454636
2. Extract the archive and navigate to the extracted directory.
3. Install dependencies: pip install -r requirements.txt
4. Update evaluation/config.py with the OpenRouter API key.  

Evaluation Procedure


The generated OCL constraints are evaluated using the USE (UML-based Specification Environment) tool.

Prerequisites


Install the USE tool from the official repository:

- USE GitHub Repository: https://github.com/useocl/use

Additional installation instructions and documentation are available in the repository.

Step 1: Generate OCL Constraints

To just apply the transformation on the orignal UML models:

Run python run.py --mode {transformation} --task transform

The following choices for the transformations could be applied: rename, attribute, association, rename_attribute, rename_association, attribute_association, full

Example: python run.py --mode rename --task transform

To generate OCL constraints using the benchmarking pipeline:

python run.py --mode {transformation} --task llm

Example:

python run.py --mode rename --task llm

To generate OCL expression using transformed dataset:

Run python run.py --mode full --task both

This produces OCL constraints generated by the selected LLM for the corresponding UML model and natural language specification.

Step 2: Open the UML Model in USE

For each benchmark instance:

1. Launch USE.
2. Load the corresponding .use model file located in:
dataset/UML/<ModelName>/<ModelName>.use

The .use file contains:

- The UML class model.
- The reference OCL constraints associated with the model.

Step 3: Load Test Instances

The repository includes object diagrams (test instances) in `.soil` format.

Load a test instance into USE using either:

1. The graphical interface; or
2. The USE command line.

For example:

open <test_instance>.soil

These instances represent valid and invalid system states against which OCL constraints can be evaluated.

Step 4: Evaluate Generated OCL Constraints

Replace or add the generated OCL constraint obtained from Step 1 to the loaded model.

Using the loaded object instances, USE automatically evaluates whether the constraint is satisfied or violated.

The evaluator should compare the behavior of the generated constraint against the expected behavior on the provided test instances.

A generated OCL constraint is considered correct if it classifies the provided test instances consistently with the reference constraint.

Step 5: Repeat for Transformed Models

To evaluate robustness under model transformations:

1. Generate transformed UML models:
python run.py --mode {transformation} --task transform
2. Generate OCL constraints for the transformed models:
python run.py --mode {transformation} --task llm
3. Repeat Steps 2–4 using the transformed UML models and generated OCL constraints.

Additional Documentation


Detailed instructions for loading models, executing commands, and evaluating OCL constraints can be found in the USE documentation:

https://github.com/useocl/use

Please refer to the USE user manual for environment-specific instructions and advanced usage.

Reproducing the Experimental Results


To reproduce the robustness experiments reported:

1. Generate transformed datasets:
   python run.py --mode full --task transform

2. Generate OCL constraints:
   python run.py --mode full --task llm

3. Evaluate generated constraints using USE.

4. Compare the generated constraints against the reference constraints on the provided test instances.

The resulting pass/fail outcomes can be used to compute the robustness metrics reported in the paper. The robustness score reported in the paper is computed as the percentage of generated OCL constraints that remain semantically equivalent to the reference constraint after transformation. Semantic equivalence is determined by executing both constraints against the provided test instances in USE and comparing their classification results.

Example


1. Generate OCL for the Airport model:
python run.py --mode rename --task llm

Expected output:

dataset/Transformed_renamed_UML/Airport/transformed_renamed_Airport.ocl

2. Open in USE tool:

dataset/Transformed_renamed_UML/Airport/transformed_renamed_Airport.use

3. Load:

dataset/Transformed_renamed_UML/Airport/Test_instances/*.soil

Expected outcome:

The generated OCL constraint should accept all valid test instances and reject all invalid test instances provided for the Airport benchmark.

Supplementary Material

Supplementary_Material.pdf provides real-world evidence grounding the three benchmark transformations

Files

Artifact.zip

Files (813.8 kB)

Name Size Download all
md5:13860346d8ec3aa8f8ac1cf66da6e412
813.8 kB Preview Download

Additional details

Dates

Submitted
2026-03-27