Semiparametric Methods for Data Assimilation and Uncertainty Quantification

There is a growing demand in many scientific disciplines for efficient tools to automatically learn models and make predictions from limited noisy observations. For these predictions to be actionable, they must also have quantifiable uncertainty, and be robust to model misspecification. This is particularly relevant in light of events such as the COVID-19 pandemic, where models have to be constantly adapted to include new phenomena such as unreported and asymptomatic cases and constantly evolving social distancing rules and compliance. Other applications include large complex systems such as weather forecasting and social network dynamics where first-principles models are powerful but have difficulty capturing the full range of phenomena involved. The semiparametric framework will help address the growing problem of un-modeled phenomena by allowing existing models to be automatically merged with model-free methods that leverage data to learn a correction to the model in order to match the observed data. The new tools will allow application to a class of high dimensional problems with spatial structure, such as geosystems problems, social networks, and global disease dynamics. Beyond improving forecasting, the semiparametric approach will include accurate uncertainty quantification, which is critical in these application domains. The investigator will train a graduate student and undergraduate students who will be able to carry this research forward, as well as developing and disseminating this key expertise. These students will learn to apply both state-of-the-art and the newly developed methods which will prepare them for future work in applied and computational mathematics.

The investigator will develop semiparametric modeling techniques that optimally leverage the strengths of parametric (model based) and nonparametric (model-free or data-driven) methods. Specifically, the semiparametric framework allows the flexible nonparametric models to fill in the gaps and correct the low-dimensional model error in a parametric model. The framework employs an ensemble of states in the parametric model to represent the uncertainty in a forecast or state estimate, while a full probability distribution is estimated for the nonparametric model. At each filtering or forecasting step, the ensemble is updated by sampling individual corrections from the model error distribution estimated by the nonparametric model. These sampled corrections will automatically correct biases in the model and inflate the uncertainty when necessary in order to match reality. The evolution of the nonparametric model will typically need to be conditional to the high-dimensional state of the parametric model, which current methods to do not allow. In other words, information must flow in both directions: the nonparametric model corrects the parametric model, but is also informed by the current state of the parametric model. In order to overcome this crucial challenge, supervised dimensionality reduction techniques will be combined with a novel method of learning mappings between non-diffeomorphic spaces. This will allow a Bayesian update of the nonparametric state estimate based on the learned projection of the parametric state. The research includes a novel higher order unscented ensemble forecast that will form the basis for a higher order Kalman filter. These advances will make the best use of available computation resources, since the higher order ensemble forecasting and filtering methods can scale up from small to large ensembles as resources allow. The higher order methods will improve accuracy and uncertainty quantification by estimating higher order moments of the state estimate and the forecast. For the ensemble forecast, a novel multivariate quadrature method will be applied that uses rank-1 tensor decompositions of the higher moments as quadrature nodes. For the Kalman update, higher order equations will be used based on a maximum entropy closure of the moment equations derived from the Kushner equation (which fully describes the true solution). The advances will effectively use data to learn a model-free correction to a parametric model, simultaneously alleviating model error and the curse-of-dimensionality.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.