Gold Standard and Annotation Dataset for CO2 Emissions Annotation

This repository contains the results of a research project which provides a benchmark dataset for extracting greenhouse gas emissions from corporate annual and sustainability reports. 

The zipped datasets file contains two datasets, gold_standard and annotation_dataset(inside the outer zip file there is a password-protected zip file containing the two datasets. To unpack, use the password is provided in the outer zip file).

Data collection

1. A Large Language Model (LLM) based pipeline was used to extract the greenhouse gas emissions from the reports (see columns prefixed with llm_ in annotation_dataset). The extracted emissions follow the categories Scope 1, 2 (market-based) and 2 (location-based) and 3, as defined in the GHGP protocol (see variables scope).
2. Annotation of the pipeline output was done in 3 phases: first by non-experts (see columns prefixed with non_expert_ in annotation_dataset), then by expert groups (columns prefixed with exp_group_ in annotation_dataset) in case of disagreement of non-experts and finally in a discussion of all experts (columns prefixed with exp__disc in annotation_dataset) in case of disagreement between expert groups. The annotation guidelines for the non-experts and experts are also included in this repository.
3. The annotation results from all three phases are combined to form the final benchmark dataset: gold_standard. Codebooks detailing each variable of each of the two datasets are also provided. More details about the annotation template or the data wrangling scripts can be found in the GitHub repository. 

Merging of datasets

Users can match the two datasets (gold_standard and annotation_dataset). Due to mismatches between the company_name in the report_name and the actual company_name, manual correction has been carried out. The gold_standard now contains both the old (report_name_old, company_name_old) and the new (report_name, company_name) variables. To join the gold_standard and annotation_dataset, please add the suffix _old to the columns (report_name, company_name) in annotation_dataset and use the old variables (report_name_old, company_name_old) from the gold_standard and additionally report_year and merge_id (index column). The merge_id already includes the company name and report year implicitly, but to avoid column duplication in the join operation, it should be included as join variables. For example this is useful when comparing LLM extractions to gold standard data.

# Merge datasets

# Add suffix _old to columns report_name and company_name in annotation dataset 
annotations_join <- annotations |> rename(report_name_old = report_name, company_name_old = company_name)

# Join annotations to gold standard
df <- gold_standard |> left_join(annotations_join, 
                          join_by(report_name_old, company_name_old, report_year, merge_id))