Software From: A Retrospective Assessment of COVID-19 Model Performance in the US

All weekly mortality forecast/observation data were collected from the COVID-19 Forecast Hub's publicly available data repository (https://doi.org/10.5281/zenodo.6301718). State population data from the United States Census Bureau is also required to generate Figure 1 (https://www2.census.gov/programs-surveys/popest/datasets/2020-2021/state/totals/NST-EST2021-alldata.csv). Instructions for downloading the required data are provided in 'Read_Process_Data_070222.R'.

This data was read and processed using the included code ('Read_Process_Data_070222.R'), which also produced Figure 1 in our manuscript. The following COVID-19 forecasting models were included in our analysis:

• BPagano-RtDriven
  • https://bobpagano.com/covid-19-modeling/
• CovidAnalytics-DELPHI
  • https://www.covidanalytics.io/DELPHI_documentation_pdf
• COVIDhub-baseline
  • https://covid19forecasthub.org/
• COVIDhub_CDC-ensemble
  • https://www.cdc.gov/coronavirus/2019-ncov/science/forecasting/mathematical-modeling.html
• CU-nochange
  • https://doi.org/10.1101/2020.03.21.20040303
• CU-scenario_low
  • https://doi.org/10.1101/2020.03.21.20040303
• CU-scenario_mid
  • https://doi.org/10.1101/2020.03.21.20040303
• CU-select
  • https://doi.org/10.1101/2020.03.21.20040303
  • https://www.medrxiv.org/content/10.1101/2020.05.04.20090670v2
• DDS-NBDS
  • https://dds-covid19.github.io/
• epiforecasts-ensemble1
  • https://doi.org/10.12688/wellcomeopenres.16006.1
• GT-DeepCOVID
  • https://ojs.aaai.org/index.php/AAAI/article/view/17808
• IHME-CurveFit
  • https://www.medrxiv.org/content/10.1101/2020.03.27.20043752v1
• JHU_CSSE-DECOM
  • https://systems.jhu.edu/research/public-health/predicting-covid-19-risk/
• JHUAPL-Bucky
  • https://github.com/mattkinsey/bucky
• Karlen-pypm
  • https://arxiv.org/abs/2007.07156
• MIT_CritData-GBCF
  • https://github.com/sakethsundar/covid-forecaster
• MOBS-GLEAM_COVID
  • https://uploads-ssl.webflow.com/58e6558acc00ee8e4536c1f5/5e8bab44f5baae4c1c2a75d2_GLEAM_web.pdf
• PSI-DRAFT
  • https://github.com/reichlab/covid19-forecast-hub/tree/master/data-processed/PSI-DRAFT
• RobertWalraven-ESG
  • http://rwalraven.com/COVID19
• SteveMcConnell-CovidComplete
  • https://stevemcconnell.com/covid
• UCSD_NEU-DeepGLEAM
  • https://datascience.ucsd.edu/COVID19/
• UMass-MechBayes
  • https://github.com/dsheldon/covid
• USC-SI_kJalpha
  • https://arxiv.org/abs/2007.05180

We share the list of forecast locations and dates (i.e., questions) that were part of the analysis in the file 'Question_List.csv' (which can be generated by running the code in 'Read_Process_Data_070222.R').

The resulting dataset was then analyzed for predictive and probabilistic performance using the included code 'CM main 070822.R'. Other included functions ('calibrationScore1.R'; 'informationScore1.R'; 'constructDM1.R'; and 'globalWeights_opt1.R') are necessary to run this performance assessment code.

For more information on the performance criteria, forecast data, observation data, and model selection process, please see the main text of our manuscript (to be published soon). For more information on running the code and generating the output files, please see our README file.