A big problem: The COVID-19 models that informed our government’s response pretty much failed at making good predictions. Unfortunately, frequent and big changes to modeling predictions has harmed the public confidence with official COVID-19 health guidance. We all understand that COVID-19 models are updated with new data as time goes on, and that might lead to some “tweaked” predictions. However, it’s quite a big deal when the initial predictions are very wrong. For example, when the number of actual US deaths is much greater than predicted estimates. A spectacular example is the University of Washington’s IHME model. This is the most widely cited COVID-19 model and has repeatedly given wrong predictions of mortality.
When it comes to modeling, there are at least two sources of errors: incomplete data and wrong assumptions. The COVID-19 highlights both problems. COVID-19 data suffered from a lack of testing early in the epidemic, and by differences in how counties and states report COVID-19 cases and deaths. Since we knew very little about how the virus transmits from person to person, that makes it very difficult to build a reliable model based on assumptions. These problems are not unique to COVID-19 and we can expect similar issues to emerge in any future outbreak caused by a new pathogen.
In the early phase of a pandemic, the data will be sparse and potentially misleading. We can expect a ton of news reporting, much of it turning out to be wrong. The effects of human behaviors, like social distancing and wearing masks, will not be easy to anticipate. I’m not confident that public health professionals or infectious disease experts can make good predictions under these conditions.
In recent history, our top disease experts gave failed predictions for bird flu, swine flu, BSE and SARS. I’m not discounting the good research done by infectious disease experts, but research and forecasting are two very different things. And despite the checkered history for predicting the course of outbreaks, we still need accurate high-stakes disease modeling to steer government and community-level actions.
The solution? We should prioritize individual forecasters based on the quality of their past predictions, and not by the number of scientific papers they published. In particular, we should be receptive to successful forecasters, whether they’re infectious disease experts or not. The problem is that forecasters need ample opportunities to practice making “real world predictions”. To be clear, I’m referring to predictions made under fluid (if not chaotic) conditions and with incomplete data. Thankfully, there is a group doing exactly that. For more information on tracking the predictive abilities of individual forecasters, please see the Good Judgement Open project.
The creators of Good Judgement have discovered that certain forecasters will consistently make better predictions than many of their peers. These “superforecasters” use a blend of publicly available data, probability, and intuition. It’s not clear whether governments would make decisions based on non-expert advice from a superforecaster, but if that person has a strong track record of making accurate predictions, then why not?
~ James
