Crowd4SDG - Crowdsourced image classification and damage assessment

This data set contains crowdsourced classification and damage assessment of images of an earthquake extracted from social media.  

A data set of 907 images posted on Twitter related to the 2019 Albanian Earthquake, that are filtered and pre-classified using an automated technique is cross-validated for accuracy by two different crowds. One, digital humanitarian volunteers using the crowdsourcing platform CROWD4EMS and another, paid micro-taskers of the Amazon Mechanical Turk. In order to compare and evaluate the efficiency and accuracy of the volunteers and the paid micro taskers, ground truth is established with the help of a team of experts, who validated the same set of data. 

Parameters considered for volunteer contributions: The dataset was imported to the Crowd4EMS platform for Crowd contribution. In the forum, each volunteer will see the image to be validated along with the tweet text and the link to the original tweet. The user has to validate whether the given image is relevant or irrelevant to the disaster. In case of doubt, the user can refer to the tutorial explaining the relevance or skip the task. Once the image's relevance is validated, the user will be asked to label the severity of the impact, as seen in the image.

The Automated algorithm has pre-classified the images as severe and minimal damage. The Crowd4EMS platform lets the volunteer label them as 'severe damage,' moderate damage',' minimal damage', and' no damage'. Each task has to be answered at least three times, and the final consensus is taken as per the inter-rater agreement. 

Parameters considered for micro-taskers contribution: The dataset was imported to the Amazon Mechanical Turk platform for Crowd contribution. In the platform, each worker will see only the image that is to be categorised as follows: The user has to validate whether the given image depicts severe damage, moderate damage, minimal damage, no damage or irrelevant to the disaster. Each task has to be answered at least ten times, and the final consensus is taken as per the inter-rater agreement. 

Acknowledgements: We want to thank Muhammad Imran of Qatar Computing Research Institute for sharing their pre-filtered social media imagery dataset on the Albanian earthquake from the Artificial Intelligence for Disaster Response (AIDR) Platform. We would also like to extend our gratitude to the volunteers for their contribution on the Crowd4EMS Platform.