Benchmark Overview

Supported Tasks

We construct a benchmark dataset based on student-built ontologies to enable the experimental investigation of four knowledge engineering sub-tasks:

• Detection of modeling issues
• Classification of modeling issues
• Explanation of modeling issues
• Generation of alternative modeling solutions as possible corrections

Contents

The benchmark contains axioms that include the following OWL property restrictions:

• Existential restrictions (owl:someValuesFrom)
• Universal restrictions (owl:allValuesFrom)
• Cardinality restrictions 

  • owl:minQualifiedCardinality

  • owl:maxQualifiedCardinality

  • owl:qualifiedCardinality

Creation and Further Details

The complete methodology for creating the benchmark is described in the following publication:

Tsaneva, S., Herwanto, G. B., Llugiqi, M., and Sabou, M. Knowledge Engineering with Large Language Models: A Capability Assessment in Evaluating Ontology Property Restrictions. Submitted to Semantic Web Journal (Under Review) View Publication

Sample.csv

To facilitate the piloting of the expert annotation of the benchmark, 14 axioms (15%) were annotated by two experts each in an initial evaluation round. These 14 axioms were excluded from the final benchmark and can be used to pilot experimental investigations.

Benchmark Dataset 

Benchmark-NoEval.csv

The complete set of axioms is provided without the expert annotation to prevent potential data leaks into the training data of large language models.

The complete dataset including expert annotations can be provided upon request. 

LLM Dataset 

Additional datasets produced during the experimental investigations are also available. These include outputs of 4 LLMs:

• GPT 4o (gpt-4o-2024-08-06)
• Claude Sonnet 3.7 (claude-3-7-sonnet-20250219)
• Llama 3.3 (Llama-3.3-70B-Instruct-Turbo)
• DeepSeek V3 (version from 2024/12/26)

on the 4 knowledge engineering subtasks (detection, classification, explanation and generation).

The complete additional datasets including expert annotations can be provided upon request.