Going beyond "shut-up and calculate" performance metrics: a low-cost story on pollution sensors

Air pollution and climate change are fully entangled and a large proportion of the world's population is unaware of the planet's greatest environmental threats. In order to tackle them, we need to know in as much detail what and how much is emitted, what secondary pollution formation mechanisms are dominant, as well as the spatial distribution of pollutants. Low-cost sensors (LCS) could help in this challenge by offering information on spatial and temporal scales never seen before.

When a potential user is given the task of finding an instrument capable of answering questions such as “Are people in this area exposed to safe levels of X?”; “What is the maximum concentration in site Y?”; “Did intervention "Z" work?), they often find themselves with many available alternatives and a flood of information that will not always be as transparent as we’d like it to be. In an ideal world each instrument would have been thoroughly assessed in a relevant environment, and typical performance metrics -R2, RMSE, MAE- would be available to describe which instrument would fit our purpose. However, LCS suffer from hardware and software related issues, but also single metrics have limited validity as they synthesise lots of information into one single number. In this study, we evaluate the performance of 13 different commercial LCS devices in UK roadside and urban background locations (in London, Manchester & York), under a range of conditions (4 seasons, ~3 years), using visual methods such as Bland-Altman and Relative Expanded Uncertainty plots, for a more complete panorama of the LCS capabilities in real life applications. The tools generated for the analysis are open-source (R and Python code, via GitHub), hoping to contribute to the information interpretation by interested parties, beyond what a dimensionless numerical index can offer.