OSDG Community Dataset (OSDG-CD)

Dataset Information

In support of the global effort to achieve the Sustainable Development Goals (SDGs) , OSDG is realising a series of SDG-labelled text datasets. Being the first one in the series, the OSDG Community Dataset (OSDG-CD) is the direct result of the work of more than a thousand volunteers from over 100 countries who have contributed to our understanding of SDGs via the OSDG Community Platform (OSDG-CP). The dataset contains tens of thousands of text excerpts (henceforth: texts) which were validated by the Community volunteers with respect to SDGs. The data can be used to derive insights into the nature of SDGs using either ontology-based or machine learning approaches.

The file contains 32,120 text excerpts and a total of 172,769 assigned labels.

Source Data

The dataset consists of text excerpts of paragraph length derived from publicly available documents, including reports, policy documents and publication abstracts. A significant number of documents (more than 3 thousand) originate from UN-related sources such as SDG-Pathfinder and SDG Library. These sources often contain documents that already have SDG labels associated with them. Each text is comprised of 3 to 6 sentences and is about 90 words on average.

Methodology

All texts are evaluated by volunteers on the OSDG-CP. The Platform is an ambitious attempt to bring together researchers, subject matter experts and SDG advocates from all around the world to create a large and accurate source of textual information on SDGs. The Community volunteers use the Platform to participate in labelling exercises where they validate each text's relevance to SDGs based on their background knowledge. In each exercise, the volunteer is shown a text together with an SDG label associated with it - this usually comes from the source - and asked to either accept or reject the suggested label.

There are three types of exercises: 

1. All volunteers start with the mandatory introductory exercise that consists of 10 pre-selected texts. Each volunteer must fully complete this exercise before they can access two other exercise types. Upon completion, the volunteer reviews the exercise by comparing their answers with the answers of the rest of the Community using aggregated statistics we provide, i.e., the share of those who accepted and rejected the suggested SDG label for each of the 10 texts. This helps the volunteer to get a feel of the platform. 

2. SDG-specific exercises where the volunteer validates texts with respect to a single SDG, e.g., SDG 1 No Poverty. 

3. All SDGs exercise where the volunteer validates a random sequence of texts where each text can have any SDG as its associated label. 

After finishing the introductory exercise, the volunteer is free to select either SDG-specific or All SDGs exercises. Each exercise, regardless of its type, consists of 100 texts. Once the exercise is finished, the volunteer can either label more or exit the platform. Of course, the volunteer can finish the exercise early. All progress is saved and recorded still. To ensure quality, each text is validated by up to 9 different volunteers and all texts included in the public release of the data have been validated by at least 3 different volunteers.

It is worth keeping in mind that all exercises present the volunteers with a binary decision problem, i.e., either accept or reject a suggested label. The volunteers are never asked to select one or more SDGs a certain text might relate to. The rationale behind this set-up is that asking a volunteer to select from 17 SDGs is extremely inefficient. Currently, all texts are validated against only one associated SDG label. 

Column Description

• doi - Digital Object Identifier of the original document
• text_id - unique text identifier
• text - text excerpt from the document
• sdg - the SDG the text is validated against
• labels_negative - the number of volunteers who rejected the suggested SDG label
• labels_positive - the number of volunteers who accepted the suggested SDG label
• agreement - agreement score based on the formula \(agreement = \frac{|labels_{positive} - labels_{negative}|}{labels_{positive} + labels_{negative}}\)

Further Information

To learn more about the project, visit OSDG's website and its official GitHub page. Queries can be directed to community@osdg.ai.