AI for mapping multi-lingual academic papers to the United Nations' Sustainable Development Goals (SDGs)

PLEASE GO TO LATEST VERSION

In this report we demonstrate how we made the multi-lingual text classifier to match research papers to the Sustainable Development Goals (SDGs) of the United Nations.

We trained the BERT multi-language model to classify the 169 individual SDG Targets, based on the English abstracts in the corpus of 1.4 million research papers.  We gathered that data using Scopus with the Aurora SDG Query model v5, which has an evaluated average precision of 70% and recall of 14%.

This is a follow-up project of the query based Aurora SDG classification model v5. The purpose of this project is to try and tackle several issues: 1. to label also research output to an SDG that is written in a non-english language, 2. to include papers that use other terms than the exact terms used in keyword searches, 3. to have a classification model that works independent from any other database specific query language.

In this report we show how we decided to use the abstracts only and the mBERT model to train the classifier. Also we show why we trained 169 individual models, instead of 1 multi-label model, including the evaluation for prediction. We will show how to prepare the data for training, and how to run the code to train the models on multiple GPU cores. Next we show how to prepare the data for prediction and how to use the code to predict English and non-English texts. And finally we evaluate the model by reviewing a sample of non-English research papers, and provide some tips to increase the reliability of the predicted outcomes.

This collection will contain:

1. Report / technical documentation describing the method and evaluating the models.
   • AI_for_mapping_... SDGs_.pdf
   • (NEED TO FINISH SECTIONS EVALUATION AND CONCLUSION)
2. Text classification models: a table containing the download urls for each of the mBERT models for each SDG-Target  and SDG-Goal in .h5 format.
   • SDGs_many_BERTs_models_download_urls.csv
   • (.csv format, semicolon separated)
   • for only SDG-Goals models: https://doi.org/10.5281/zenodo.5835849
3. Training data sample in .csv format. Containing abstract and columns of SDG-Targets with 1 or 0.
   • train_data_sample_aurora_sdg_v5_worldwide_set_doi_abstracts_sdg_targets_2009-2020-in-columns.csv
   • (.csv format, comma separated)
   • get full data here (1.4 million doi's labeled to SDG Targets):  https://doi.org/10.5281/zenodo.5205672 
   • due to licencing agreements with Scopus we are only allowed to share a limited amount of abstracts. Support https://i4oa.org to get more free-to-use abstracts in Crossref.
4. Training code in python. Explaining what parameters we used to train the models on GPU hardware.
   • train.py
   • get / fork code here: https://github.com/Aurora-Network-Global/sdgs_many_berts
5. Test Statistics on accuracy of each of the trained models
   • SDGs_many_BERTs_models_test_statistics.csv
   • (.csv format, tab separated)
6. Prediction data sample. Text file (UTF8) containing abstracts of papers in different languages on each row.
   • predict_data_sample_multiple_language_abstracts.csv
   • (.csv format, list, languages row 2-31: EN, row 32-152: NL )
7. Prediction code in python. Setup to run the models to classify a text fragment.
   • predict.py
   • get / fork code here: https://github.com/Aurora-Network-Global/sdgs_many_berts