Knowledge Graph Triple Validation by LLMs and Human-in-the-Loop

Suplementary material for the sumbitted article to the IPM Special issue on Large Language Models and Data Quality for Knowledge Graphs. 

The dataset is an extension of [1] and includes the following columns:

• subj the subject/head of the triple
• rel the predicate of the triple
• obj the object/tail of the triple
• support-level indicating the reliability of the triple
• ann-random[1: valid, 0: invalid], randomly selected annotation from the expert annotations avaialble in [1] 
• ann-new [1: valid, 0: invalid], junior expert annotation 
• gpt-4o-1 [1: valid, 0: invalid], response from 1st GPT prompt
• gpt-4o-2 [1: valid, 0: invalid], response from 2nd GPT prompt
• gpt-4o-3 [1: valid, 0: invalid], response from 3rd GPT prompt
• gpt-4o-majority [1: valid, 0: invalid], GPT annotation, computed as majority vote of gpt-4o-1,gpt-4o-2,gpt-4o-3
• claude-1 [1: valid, 0: invalid], response from 1st claude prompt
• claude-2 [1: valid, 0: invalid], response from 2nd claude prompt
• claude-3 [1: valid, 0: invalid], response from 3rd claude prompt
• claude-majority [1: valid, 0: invalid], claude annotation, computed as majority vote of claude-1,claude-2,claude-3
• llama-1 [1: valid, 0: invalid], response from 1st llama prompt
• llama-2 [1: valid, 0: invalid], response from 2nd llama prompt
• llama-3 [1: valid, 0: invalid], response from 3rd llama prompt
• llama-majority [1: valid, 0: invalid], llama annotation, computed as majority vote of llama-1,llama-2,llama-3

[1] https://github.com/danilo-dessi/SKG-pipeline/tree/main/eval