There is a newer version of the record available.

Published September 15, 2022 | Version 0.1
Dataset Open

MUHAI Benchmark : Task 1 (Short story generation with Knowledge Graphs)

  • 1. Vrije Universiteit Amsterdam


Meaning and Understanding in Human-Centric AI (MUHAI) Benchmark
Task 1 (Short story generation with Knowledge Graphs and Language Models)

The dataset can be used to test understandability of text generated through the combination of knowledge graphs and language models without using knowledge graph embeddings.

The task here is to generate 5-sentence stories from a set of subject-predicate-object triples that are extracted from a knowledge graph. Two steps need to be performed:

1. Language model fine-tuning (SVO triple extraction + model fine-tuning)
2. Story generation (knowledge enrichment + text generation) 

The submission includes the following data:

  1. Original ROC stories corpus (100 stories)
  2. ROC stories encoded with relevant triples (extracted through SpaCy, 2 versions, with and without coreference resolution)
  3. Stories generated by the pre-trained model (GPT2-simple)
  4. Stories generated by the fine-tuned model (DICE + ConceptNet + DBpedia )
  5. Stories generated by the fine-tuned model (DICE + ConceptNet + DBpedia + WordNet )
  6. Stories generated by the GPT-2-keyword-generation (an open-source software that uses GPT-2 to generate text pertaining to the specified keywords)
  7. Model results
  8. Evaluation metrics description
  9. User-evaluation questionnaire 

Code :



Files (76.2 MB)

Name Size Download all
24.0 kB Preview Download
57.2 MB Preview Download
18.9 MB Preview Download
474 Bytes Preview Download
468 Bytes Preview Download
24.2 kB Preview Download
17.5 kB Preview Download
19.3 kB Preview Download
16.0 kB Preview Download

Additional details


MUHAI – Meaning and Understanding in Human-centric AI 951846
European Commission


  • Yang, X., & Tiddi, I. (2020, January). Creative storytelling with language models and knowledge graphs. In CIKM (Workshops).
  • Blin, I. (2022). Building Narrative Structures from Knowledge Graphs. In European Semantic Web Conference (pp. 234-251). Springer, Cham.