Knowledge Graph Triple Validation by LLMs and Human-in-the-Loop

Suplementary material for the sumbitted article to the IPM Special issue on Large Language Models and Data Quality for Knowledge Graphs. 

The dataset is an extension of [1] and includes the following columns:

• subj the subject/head of the triple
• rel the predicate of the triple
• obj the object/tail of the triple
• support-level indicating the reliability of the triple
• gpt-4o-1 [1: valid, 0: invalid], response from 1st GPT prompt
• gpt-4o-2 [1: valid, 0: invalid], response from 2nd GPT prompt
• gpt-4o-3 [1: valid, 0: invalid], response from 3rd GPT prompt
• gpt-4o-majority [1: valid, 0: invalid], GPT annotation, computed as majority vote of gpt-4o-1,gpt-4o-2,gpt-4o-3
• ann-random[1: valid, 0: invalid], randomly selected annotation from the expert annotations avaialble in [1] 
• ann-new [1: valid, 0: invalid], junior expert annotation 

[1] https://github.com/danilo-dessi/SKG-pipeline/tree/main/eval