Published September 28, 2022 | Version 0.1
Dataset Open

MUHAI Benchmark : Task 2 (Credibility of knowledge-based generated gossip stories)

  • 1. Vrije Universiteit Amsterdam

Contributors

Contact person:

  • 1. Vrije Universiteit Amsterdam

Description

Meaning and Understanding in Human-Centric AI (MUHAI) Benchmark
Task 2 (Credibility of knowledge-based generated gossip stories)

 

This dataset aims at investigating whether the use of Knowledge Graphs has an impact on the credibililty of automatically-generated stories.


The submission includes the following data:

  1. Generated stories (.txt)
  2. Story generation template 
  3. A tsv file with entities and triples (to be used for generating stories)
  4. Evaluation description : Questions and metrics submitted to the users

The "gossip stories" are generated with the T5 languge model fine-tuned on the WebNLG challenge. The model takes the triples (file 3) as input and generates one sentence each. A link prediction algorithm based on Jaccard's similarity learns the likelihood of two entities to be related (3). Then, the narrative continues with automatically generated celebrity background descriptions.

The credibility of the story is evaluated using a questionnaire based on Gaziano et. al. The questionnaire was filled in by the test subjects after reading each generated article. One for a KG-generated text where links were predicted using the link prediction and one for text that was generated using triples of random entities (celebrities). 

Full code available at : https://github.com/kmitd/muhai-credibility-KR

Files

evaluation questions and measures.txt

Files (16.7 kB)

Name Size Download all
md5:de1365d58086df28e237fdd4bd095db9
5.7 kB Download
md5:019ab7397097a7d0e03c95446cead364
619 Bytes Preview Download
md5:10f3bdd8cbb3a78451f150d8df57e7b9
817 Bytes Preview Download
md5:d652b690377f92a03771e60279a249e9
829 Bytes Preview Download
md5:f8fa21688060f0ea9cf908a331e864f6
1.1 kB Preview Download
md5:e810c63f7bc09b8ae207c02fed1c0d6f
957 Bytes Preview Download
md5:9d1f5ae688ec3b15f5da8b1d0306f3fa
715 Bytes Preview Download
md5:ca535b8c2bf8caa7d6cbfa78a68044d0
950 Bytes Preview Download
md5:2477b4f5d898f7f41a797d8f9a26da7b
941 Bytes Preview Download
md5:ac7ce13f33f136f320ae71d4cda6463e
904 Bytes Preview Download
md5:4b6954917d3d0de0d411f8ad8fc4d5ff
1.4 kB Preview Download
md5:f880205ca41b57631cd5997f36f983d3
966 Bytes Preview Download
md5:b819ffff67de00b707e0fca348958206
752 Bytes Preview Download

Additional details

Related works

Continues
10.5281/zenodo.7081523 (DOI)

Funding

MUHAI – Meaning and Understanding in Human-centric AI 951846
European Commission