Vocabulary evaluation of LLMs

This repository contains the results for the different vocabulary tests run on LLM tools/models presented in the paper: "The continued usefulness of vocabulary tests for evaluating large language models" currently published  in PLOS ONE: https://doi.org/10.1371/journal.pone.0308259

The name of the files correspond to the vocabulary tests for which results are presented in Tables 3-6 in the paper (note that questions for the TOEFL test are not public).  

In each file, the first column has the question posed to the LLM tool/model, followed by the correct answer. The rest of the columns correspond to the answers of the different LLM tools/models evaluated.  The results and percentages are summarized at the bottom of the file after the last test items.

The models evaluated are:

To cite our work:

@article{10.1371/journal.pone.0308259,
    doi = {10.1371/journal.pone.0308259},
    author = {Martínez, Gonzalo AND Conde, Javier AND Merino-Gómez, Elena AND Bermúdez-Margaretto, Beatriz AND Hernández, José Alberto AND Reviriego, Pedro AND Brysbaert, Marc},
    journal = {PLOS ONE},
    publisher = {Public Library of Science},
    title = {Establishing vocabulary tests as a benchmark for evaluating large language models},
    year = {2024},
    month = {12},
    volume = {19},
    url = {https://doi.org/10.1371/journal.pone.0308259},
    pages = {1-17},
    number = {12},

}