Published January 31, 2025 | Version v1.0.0

Replication Package for "Requirements Traceability Link Recovery via Retrieval-Augmented Generation"

  • 1. ROR icon Karlsruhe Institute of Technology

Description

This is the replication package for our paper "Requirements Traceability Link Recovery via Retrieval-Augmented Generation".
This package contains the source code, the dataset used in the evaluation, and the results of the evaluation.

Requirements

  • Java JDK 21 + Maven 3

  • OpenAI subscription: API Key

  • Ollama Instance

Structure of this Repository

  • / : Contains the source code of the approach
    (Note: The most recent version of the tool can be found at ArDoCo/LiSSA-RATLR)

  • /cache : Contains the cache for the evaluation.

  • /configs : Contains the configurations used in the evaluation.

  • /datasets : Contains the datasets used in the evaluation and the results of the evaluation.

  • /results-r2r : Contains the results of the evaluation.

  • /experiments : Contains the aggregated results of our experiments, as well as the scripts and results of the statistical tests.

Installation

  1. You can use the provided JAR.

  2. Alternatively, build the project using Maven: mvn package

    The JAR will be created in the target folder (e.g., target/ratlr-*-jar-with-dependencies.jar).

Usage

  1. Create a .env file based on the provided template.

  2. Run the JAR from the top-level directory:

    java -jar target/ratlr-*-jar-with-dependencies.jar eval -c ./configs/req2req/<<DesiredConfig>>

Results of Evaluation

All results are available in the results-r2r folder. Results are stored as Markdown (MD) files in the respective projects. Each file includes the configuration and the results of the evaluation. Example configuration:

{
  "cache_dir" : "./cache/CCHIT",
  "gold_standard_configuration" : {
    "hasHeader" : true,
    "path" : "./datasets/req2req/CCHIT/answer.csv"
  },
  "source_artifact_provider" : {
    "name" : "text",
    "args" : {
      "artifact_type" : "requirement",
      "path" : "./datasets/req2req/CCHIT/high"
    }
  },
  "target_artifact_provider" : {
    "name" : "text",
    "args" : {
      "artifact_type" : "requirement",
      "path" : "./datasets/req2req/CCHIT/low"
    }
  },
  "source_preprocessor" : {
    "name" : "artifact",
    "args" : {}
  },
  "target_preprocessor" : {
    "name" : "artifact",
    "args" : {}
  },
  "embedding_creator" : {
    "name" : "openai",
    "args" : {
      "model" : "text-embedding-3-large"
    }
  },
  "source_store" : {
    "name" : "custom",
    "args" : {}
  },
  "target_store" : {
    "name" : "custom",
    "args" : {
      "max_results" : "4"
    }
  },
  "classifier" : {
    "name" : "mock",
    "args" : {}
  },
  "result_aggregator" : {
    "name" : "any_connection",
    "args" : {
      "source_granularity" : "0",
      "target_granularity" : "0"
    }
  },
  "tracelinkid_postprocessor" : {
    "name" : "identity",
    "args" : {
      "reverse" : "false"
    }
  }
}

Statistics

  • # TraceLinks (GS): 587

  • # Source Artifacts: 116

  • # Target Artifacts: 1064

Results

  • True Positives: 92

  • False Positives: 372

  • False Negatives: 495

  • Precision: 0.1983

  • Recall: 0.1567

  • F1 Score: 0.1751

Files

ArDoCo/ReplicationPackage-REFSQ25_Requirements-TLR-via-RAG-v1.0.0.zip

Files (29.5 MB)

Additional details