Are Large Language Models Reliable Argument Quality Annotators?

This is a dataset of 320 arguments, each annotated with 15 different argument quality dimensions. The annotations were performed by 2 groups of human annotators: expert and novice, as well as large language models with different prompt variations.  

Please find more details in the corresponding publication: https://webis.de/publications.html#mirzakhmedova_2024b

Code for experiments can be found at: https://github.com/webis-de/RATIO-24

Please use the following citation key: 

@InProceedings{mirzakhmedova:2024b,
  author =                   {Nailia Mirzakhmedova and Marcel Gohsen and Chia Hao Chang and Benno Stein},
  booktitle =                {1st International Conference on Recent Advances in Robust Argumentation Machines {(RATIO-24)}},
  doi =                      {10.1007/978-3-031-63536-6_8},
  editor =                   {Philipp Cimiano and Anette Frank and Michael Kohlhase and Benno Stein},
  month =                    jun,
  pages =                    {129--146},
  publisher =                {Springer},
  site =                     {Bielefeld, Germany},
  title =                    {{Are Large Language Models Reliable Argument Quality Annotators?}},
  volume =                   14638,
  year =                     2024
}