RAPID: Time-Sensitive Human Forest and Model Forecasts for COVID-19 Vaccine and Treatment Trials

Accurate, time-specific predictions are important for planning and decision making during fast-moving pandemics. In particular, whether an effective COVID-19 vaccine will be available in 9, 12, or 18 months is an issue of vital national interest. The main objective of this project is to compare the accuracy of a new method for crowd-based forecasting of time-specific outcomes?such as clinical trial transitions of COVID-19 treatments and vaccines?to that of new machine learning models. The research will examine the relative strengths of crowd and modeling methods and explore combinations of the two in predicting clinical trial results. A forecasting tournament is the project?s main method for human data collection. It starts in 2020 and continues until 2021. People with interest in forecasting and clinical trials are encouraged to sign up for participation, independently of their background. Study participants complete surveys and forecasting training, and will then have the opportunity to make probabilistic forecasts on specific trial events over several months, with regular accuracy feedback. To broaden the impacts of this work, the research team disseminates the aggregate forecasts about clinical trial phase transition of COVID-19 treatments and vaccines through public health information channels. These forecasts, combined with predictive training and accuracy feedback provided to study participants, may aid the coordination of public health and clinical development efforts to overcome the pandemic.

The primary research goal of the project is to improve the predictive performance of crowd-based methods, machine models and ensembles of the two. Psychologists have shown that taking the outside view, by examining a prediction problem in context of historical reference classes, improves accuracy. The crowd-based approach, referred to as human forest, combines reference class forecasting and collective intelligence approaches to produce data-driven estimates from a group of forecasters. The time-specific human forest variant employs a survival analysis approach, enabling forecasters to construct reference classes and obtain unbiased historical estimates in the presence of missing data. On the decision science front, the research goals include testing the effects of interfaces featuring historical estimates; understanding the psychology of reference class selection; examining time-scope sensitivity in judgmental forecasting; and assessing the relative importance of subject matter expertise versus general predictive competence. On the machine-modeling front, the research goals include integrating survival-type models into machine learning and improving their performance using bi-level optimization to choose hyper-parameters. The results are released as soon as they become available.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.