Published July 9, 2021 | Version Published
Conference paper Restricted

It's About Time: A View of Crowdsourced Data Before and During the Pandemic

  • 1. CYENS – Centre of Excellence Nicosia, Cyprus
  • 2. Open University of Cyprus & CYENS – Centre of Excellence Nicosia, Cyprus


Data attained through crowdsourcing have an essential role in the development of computer vision algorithms. Crowdsourced data might include reporting biases, since crowdworkers usually describe what is “worth saying” in addition to images’ content. We explore how the unprecedented events of 2020, including the unrest surrounding racial discrimination, and the COVID-19 pandemic, might be reflected in responses to an open-ended annotation task on people images, originally executed in 2018 and replicated in 2020. Analyzing themes of Identity and Health conveyed in workers’ tags, we find evidence that supports the potential for temporal sensitivity in crowdsourced data. The 2020 data exhibit more race-marking of images depicting non-Whites, as well as an increase in tags describing Weight. We relate our findings to the emerging research on crowdworkers’ moods. Furthermore, we discuss the implications of (and suggestions for) designing tasks on proprietary platforms, having demonstrated the possibility for additional, unexpected variation in crowdsourced data due to significant events.


This work has been partly supported by the project that has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 739578 (RISE – Call: H2020-WIDESPREAD-01-2016-2017-TeamingPhase2) and the Republic of Cyprus through the Deputy Ministry of Research, Innovation and Digital Policy.



The record is publicly accessible, but files are restricted to users with access.

Additional details


CyCAT – Cyprus Center for Algorithmic Transparency 810105
European Commission
RISE – Research Center on Interactive Media, Smart System and Emerging Technologies 739578
European Commission