There is a newer version of the record available.

Published September 15, 2022 | Version 0.1
Dataset Open

MUHAI Benchmark : Task 1 (Short story generation with Knowledge Graphs)

  • 1. Vrije Universiteit Amsterdam

Description

Meaning and Understanding in Human-Centric AI (MUHAI) Benchmark
Task 1 (Short story generation with Knowledge Graphs and Language Models)
 

The dataset can be used to test understandability of text generated through the combination of knowledge graphs and language models without using knowledge graph embeddings.

The task here is to generate 5-sentence stories from a set of subject-predicate-object triples that are extracted from a knowledge graph. Two steps need to be performed:

1. Language model fine-tuning (SVO triple extraction + model fine-tuning)
2. Story generation (knowledge enrichment + text generation) 

The submission includes the following data:

  1. Original ROC stories corpus (100 stories)
  2. ROC stories encoded with relevant triples (extracted through SpaCy, 2 versions, with and without coreference resolution)
  3. Stories generated by the pre-trained model (GPT2-simple)
  4. Stories generated by the fine-tuned model (DICE + ConceptNet + DBpedia )
  5. Stories generated by the fine-tuned model (DICE + ConceptNet + DBpedia + WordNet )
  6. Stories generated by the GPT-2-keyword-generation (an open-source software that uses GPT-2 to generate text pertaining to the specified keywords)
  7. Model results
  8. Evaluation metrics description
  9. User-evaluation questionnaire 

Code : https://github.com/kmitd/muhai-dice_story

Files

dice_dbpedia_100.txt

Files (76.2 MB)

Name Size Download all
md5:c9258815afccb9f50b08cfb5a2fbc77e
24.0 kB Preview Download
md5:b7e8e0ccd5ed83bf27626676f095db08
57.2 MB Preview Download
md5:f340fa0da072ed4494a4cac29e3b67ae
18.9 MB Preview Download
md5:78641546fb87ed23f387f282dedc25d2
474 Bytes Preview Download
md5:49cf0f66dfbd89183feee86bf870d03f
468 Bytes Preview Download
md5:154b1f19a5bb5e5eeb26d59dc5fac0ad
24.2 kB Preview Download
md5:58a7e45bab5a8faffd661621444ff8dc
17.5 kB Preview Download
md5:590ba54926e51947494c4f49e93e105a
19.3 kB Preview Download
md5:debd38cb5aafbeb6acae0c540aff048e
16.0 kB Preview Download

Additional details

Funding

MUHAI – Meaning and Understanding in Human-centric AI 951846
European Commission

References

  • Yang, X., & Tiddi, I. (2020, January). Creative storytelling with language models and knowledge graphs. In CIKM (Workshops).
  • Blin, I. (2022). Building Narrative Structures from Knowledge Graphs. In European Semantic Web Conference (pp. 234-251). Springer, Cham.