ProfNER corpus: gold standard annotations for profession detection in Spanish COVID-19 tweets

Gold Standard annotations for SMM4H-Spanish shared task and unannotated test and background files. SMM4H 2021 accepted at NAACL (scheduled in Mexico City in June) https://2021.naacl.org/.

Please, cite:

Miranda-Escalada, A., Farré-Maduell, E., Lima-López, S., Gascó, L., Briva-Iglesias, V., Agüero-Torales, M., & Krallinger, M. (2021, June). The profner shared task on automatic recognition of occupation mentions in social media: systems, evaluation, guidelines, embeddings and corpora. In Proceedings of the Sixth Social Media Mining for Health (# SMM4H) Workshop and Shared Task (pp. 13-20).

@inproceedings{miranda2021profner,
  title={The profner shared task on automatic recognition of occupation mentions in social media: systems, evaluation, guidelines, embeddings and corpora},
  author={Miranda-Escalada, Antonio and Farr{\'e}-Maduell, Eul{\`a}lia and Lima-L{\'o}pez, Salvador and Gasc{\'o}, Luis and Briva-Iglesias, Vicent and Ag{\"u}ero-Torales, Marvin and Krallinger, Martin},
  booktitle={Proceedings of the Sixth Social Media Mining for Health (\# SMM4H) Workshop and Shared Task},
  pages={13--20},
  year={2021}
}

Introduction:
The entire corpus contains 10,000 annotated tweets. It has been split into training, validation and test (60-20-20). The current version contains the training and development set of the shared task with Gold Standard annotations.
In addition, it contains the unannotated test and background sets will be released. Participants must submit predictions for the files under the directory "test-background-txt-files"

For subtask-1 (classification), annotations are distributed in a tab-separated file (TSV). The TSV format follows the format employed in SMM4H 2019 Task 2:
tweet_id    class
 
For subtask-2 (Named Entity Recognition, profession detection), annotations are distributed in 2 formats: Brat standoff and TSV. See the Brat webpage for more information about the Brat standoff format (https://brat.nlplab.org/standoff.html). The TSV format follows the format employed in SMM4H 2019 Task 2:
tweet_id    begin    end    type    extraction

In addition, we provide a tokenized version of the dataset, for participant's convenience. It follows the BIO format (similar to CONLL). The files were generated with the brat_to_conll.py script (included), which employs the es_core_news_sm-2.3.1 Spacy model for tokenization.

Zip structure:

subtask-1: files of tweet classification subtask. Content:

• One TSV file per corpus split (train and valid).
• train-valid-txt-files: folder with training and validation text files. One text file per tweet. One sub-directory per corpus split (train and valid).
• train-valid-txt-files-english: folder with training and validation text files Machine Translated to English.
• test-background-txt-files: folder with the test and background text files. You must make your predictions for these files and upload them to CodaLab.

subtask-2: files of Named Entity Recognition subtask. Content:

• brat: folder with annotations in Brat format. One sub-directory per corpus split (train and valid)
• TSV: folder with annotations in TSV. One file per corpus split (train and valid)
• BIO: folder with corpus in BIO tagging. One file per corpus split (train and valid)
• train-valid-txt-files: folder with training and validation text files. One text file per tweet. One sub-directory per corpus split (train and valid)
• train-valid-txt-files-english: folder with training and validation text files Machine Translated to English.
• test-background-txt-files: folder with the test and background text files. You must make your predictions for these files and upload them to CodaLab.

Annotation quality:

We have performed a consistency analysis of the corpus. 10% of the documents have been annotated by an internal annotator as well as by the linguist experts following the same annotation guideliens.

The preliminary Inter-Annotator Agreement (pairwise agreement) is 0.919.

For further information, please visit https://temu.bsc.es/smm4h-spanish/ or email us at encargo-pln-life@bsc.es
 

Do not share the data with other individuals/teams without permission from the task organizer. Tweets IDs are the primary source of information. Tweet texts are provided as support material. By downloading this resource, you agree to the Twitter Terms of Service, Privacy Policy, Developer Agreement, and Developer Policy.

Resources:

• Web
• Annotation guidelines (in Spanish)
• Annotation guidelines (in English)
• FastText COVID-19 Twitter embeddings
• Occupations gazetteer
• Conference Proceedings