Hyper-differential sensitivity analysis with respect to model discrepancy: Posterior optimal solution sampling

Joseph Hart; Bart van Bloemen Waanders

doi:10.3934/fods.2024027

Article Contents

2025, Volume 7, Issue 1: 99-133. Doi: 10.3934/fods.2024027

This issue Previous Article A practical existence theorem for reduced order models based on convolutional autoencoders Next Article Stacked networks improve physics-informed training: Applications to neural networks and deep operator networks

Hyper-differential sensitivity analysis with respect to model discrepancy: Posterior optimal solution sampling

Joseph Hart^, and
Bart van Bloemen Waanders

Sandia National Laboratories, USA

^*Corresponding author: Joseph Hart
^*Corresponding author: Joseph Hart

Received: February 2024

Revised: May 2024

Early access: June 2024

Published: March 2025

Abstract / Introduction Full Text(HTML) Figure(10) / Table(5) Related Papers Cited by

Abstract

Optimization constrained by high-fidelity computational models has potential for transformative impact. However, such optimization is frequently unattainable in practice due to the complexity and computational intensity of the model. An alternative is to optimize a low-fidelity model and use limited evaluations of the high-fidelity model to assess the quality of the solution. This article develops a framework to use limited high-fidelity simulations to update the optimization solution computed using the low-fidelity model. Building off a previous article [25], which introduced hyper-differential sensitivity analysis with respect to model discrepancy, this article provides novel extensions of the algorithm to enable uncertainty quantification of the optimal solution update via a Bayesian framework. Specifically, we formulate a Bayesian inverse problem to estimate the model discrepancy and propagate the posterior model discrepancy distribution through the post-optimality sensitivity operator for the low-fidelity optimization problem. We provide a rigorous treatment of the Bayesian formulation, a computationally efficient algorithm to compute posterior samples, a guide to specify and interpret the algorithm hyper-parameters, and a demonstration of the approach on three examples which highlight various types of discrepancy between low and high-fidelity models.

Keywords:

Hyper-differential sensitivity analysis,
post-optimality sensitivity analysis,
PDE-constrained optimization,
model discrepancy,
Bayesian analysis.

Mathematics Subject Classification: Primary: 65K10, 65C-99; Secondary: 65K99.

Citation:

Full Text(HTML)

Figure 1. Low-fidelity optimization solution for the reaction diffusion example. Left: optimal source $ \tilde{{{{\boldsymbol{{z}}}}}} $; right: state target $ T $ alongside the low and high-fidelity state solutions evaluated at $ \tilde{{{{\boldsymbol{{z}}}}}} $, i.e. $ \tilde{S}(\tilde{{{{\boldsymbol{{z}}}}}}) $ and $ S(\tilde{{{{\boldsymbol{{z}}}}}}) $, respectively

Download: Full-size image PowerPoint slide

Figure 2. Left: generalized eigenvalues (21) for the reaction diffusion example; right: the posterior optimal solution mean relative error (in the left axis) and posterior optimal solution variance (in right axis) for various subspace ranks

Download: Full-size image PowerPoint slide

Figure 3. Left: the low and high-fidelity optimal sources $ \tilde{{{{\boldsymbol{{z}}}}}} $ and $ {{{\boldsymbol{{z}}}}}^\star $, respectively, alongside the posterior optimal solution mean $ \overline{{{{\boldsymbol{{z}}}}}} $ with posterior optimal solution samples (in grey); right: the high-fidelity objective function values for 500 optimal source posterior samples, the vertical lines from left to right indicates the value the high-fidelity objective $ J(S({{{\boldsymbol{{z}}}}}), {{{\boldsymbol{{z}}}}}) $ evaluated at the high-fidelity optimal source, the posterior optimal source mean, and the low-fidelity optimal source. The top and bottom rows correspond to a subspace projection ranks of 4 and 11, respectively

Download: Full-size image PowerPoint slide

Figure 4. Diagram depicting the mass-spring system. We consider the coupled system for both blocks 1 and 2 as the high-fidelity model. A low-fidelity model is derived by assuming that block 2 is stationary

Download: Full-size image PowerPoint slide

Figure 5. Low-fidelity optimization solution for the mass-spring system. Left: low-fidelity optimal forcing $ \tilde{{{{\boldsymbol{{z}}}}}} $; right: state trajectory $ T $ alongside the low and high-fidelity block 1 displacement solutions evaluated at $ \tilde{{{{\boldsymbol{{z}}}}}} $, i.e. $ \tilde{S}(\tilde{{{{\boldsymbol{{z}}}}}}) $ and $ S(\tilde{{{{\boldsymbol{{z}}}}}}) $, respectively

Download: Full-size image PowerPoint slide

Figure 6. Left: the low and high-fidelity optimal forcings $ \tilde{{{{\boldsymbol{{z}}}}}} $ and $ {{{\boldsymbol{{z}}}}}^\star $, respectively, alongside the posterior optimal solution mean $ \overline{{{{\boldsymbol{{z}}}}}} $ with posterior optimal solution samples (in grey); right: block 1 trajectory under the forcings from the left panel

Download: Full-size image PowerPoint slide

Figure 7. Left: schematic of the computational domain, source, and target locations (top), and low-fidelity optimal source $ \tilde{{{{\boldsymbol{{z}}}}}} $ (bottom); right: low-fidelity model prediction (top) and and high-fidelity model prediction (bottom) evaluated at $ {{{\boldsymbol{{z}}}}} = \tilde{{{{\boldsymbol{{z}}}}}} $

Download: Full-size image PowerPoint slide

Figure 8. Comparison of the posterior optimal solution using projection subspace ranks $ r = 1 $ (top) and $ r = 2 $ (bottom). Left: posterior optimal solution mean $ \overline{{{{\boldsymbol{{z}}}}}} $; center: posterior optimal solution pointwise standard deviation; right: histogram of the high-fidelity objective function values corresponding to the posterior optimal solution samples with the vertical lines indicating $ J(S(\overline{{{{\boldsymbol{{z}}}}}}), \overline{{{{\boldsymbol{{z}}}}}}) $ (red broken line) and $ J(S(\tilde{{{{\boldsymbol{{z}}}}}}), \tilde{{{{\boldsymbol{{z}}}}}}) $ (black solid line)

Download: Full-size image PowerPoint slide

Figure 9. Left: samples of the prior discrepancy $ {{\delta}}({{{\boldsymbol{{z}}}}}, {{{\boldsymbol{{\theta}}}}}) $ evaluated at $ {{{\boldsymbol{{z}}}}} = \tilde{{{{\boldsymbol{{z}}}}}} $, i.e. a mean zero Gaussian distribution with covariance $ {{{\boldsymbol{{W}}}}}_{{{\boldsymbol{{u}}}}}^{-1} $; right: samples from a mean zero Gaussian distribution with covariance $ {{{\boldsymbol{{W}}}}}_{{{\boldsymbol{{z}}}}}^{-1} $

Download: Full-size image PowerPoint slide

Figure 10. Top: discrepancy data $ {\boldsymbol{{d}}}_\ell = S({{{\boldsymbol{{z}}}}}_\ell)-\tilde{S}({{{\boldsymbol{{z}}}}}_\ell) $, the posterior discrepancy mean evaluated at $ {{{\boldsymbol{{z}}}}}_\ell $, and posterior discrepancy samples $ {{\delta}}({{{\boldsymbol{{z}}}}}, \overline{{{{\boldsymbol{{\theta}}}}}}+\hat{{{{\boldsymbol{{\theta}}}}}}+\breve{{{{\boldsymbol{{\theta}}}}}}) $ evaluated at $ {{{\boldsymbol{{z}}}}} = {{{\boldsymbol{{z}}}}}_\ell $, $ \ell = 1, 2 $. Bottom left: optimization variable data $ {{{\boldsymbol{{z}}}}}_1 $, $ {{{\boldsymbol{{z}}}}}_2 $, and a reference source $ {{{\boldsymbol{{z}}}}}_{ref} $; right: the posterior discrepancy $ {{\delta}}({{{\boldsymbol{{z}}}}}, \overline{{{{\boldsymbol{{\theta}}}}}}+\hat{{{{\boldsymbol{{\theta}}}}}}+\breve{{{{\boldsymbol{{\theta}}}}}}) $ evaluated at $ {{{\boldsymbol{{z}}}}} = {{{\boldsymbol{{z}}}}}_{ref} $

Download: Full-size image PowerPoint slide

Table 1. Hyper-parameters used in the diffusion-reaction example

$ \alpha_{{{\boldsymbol{{u}}}}} $	$ \beta_{{{\boldsymbol{{u}}}}} $	$ \alpha_{{{\boldsymbol{{z}}}}} $	$ \beta_{{{\boldsymbol{{z}}}}} $	$ \alpha_{{\boldsymbol{{d}}}} $
$ 4 $	$ 2 \times 10^{-2} $	$ 10^{-10} $	$ 3 \times 10^{-2} $	$ 10^{-4} $

| Show Table

DownLoad: CSV

Table 2. Hyper-parameters used in the mass-spring system

$ \alpha_{{{\boldsymbol{{u}}}}} $	$ \beta_{{{\boldsymbol{{u}}}}} $	$ \alpha_{{{\boldsymbol{{z}}}}} $	$ \beta_{{{\boldsymbol{{z}}}}} $	$ \alpha_{{\boldsymbol{{d}}}} $
$ 10^4 $	$ 5 \times 10^{-2} $	$ 10^{-10} $	$ 10^{-1} $	$ 10^{-1} $

| Show Table

DownLoad: CSV

Table 3. Hyper-parameters used in the advection diffusion example

$ \alpha_{{{\boldsymbol{{u}}}}} $	$ \beta_{{{\boldsymbol{{u}}}}} $	$ \alpha_{{{\boldsymbol{{z}}}}} $	$ \alpha_{{\boldsymbol{{d}}}} $
$ 4 $	$ 5 \times 10^{-1} $	$ 10^{-8} $	$ 10^{-2} $

| Show Table

DownLoad: CSV

Table 4. Cost values for illustrative computational cost comparison

$ f $	$ \tilde{f} $	$ \tilde{a} $	$ e_{{{\boldsymbol{{u}}}}} $	$ e_{{{\boldsymbol{{z}}}}} $
$ 100 $	$ 15 $	$ 3 $	$ 1 $	$ 1 $

| Show Table

DownLoad: CSV

Table 5. Algorithm parameters for illustrative computational cost comparison

$ N $	$ s $	$ q $	$ r $	$ \ell_E $	$ \ell_H $
$ 2 $	$ 100 $	$ 500 $	$ 50 $	$ 10 $	$ 10 $

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	A. Agarwal and L. T. Biegler, A trust-region framework for constrained optimization using reduced order modeling, Optimization and Engineering, 14 (2013), 3-35. doi: 10.1007/s11081-011-9164-0.
[2]	N. M. Alexandrov, J. E. Dennis, R. M. Lewis and V. Torczon, A trust-region framework for managing the use of approximation models in optimization, Structural Optimization, 15 (1998), 16-23. doi: 10.1007/BF01197433.
[3]	H. Antil, D. P. Kouri, M. Lacasse and D. Ridzal, Frontiers in PDE-Constrained Optimization, IMA Vol. Math. Appl., 163. Springer, New York, 2018. doi: 10.1007/978-1-4939-8636-1.
[4]	P. D. Arendt, D. W. Apley and W. Chen, Quantification of model uncertainty: Calibration, model discrepancy, and identifiability, Journal of Mechanical Design, 134 (2012), 1-12. doi: 10.1115/1.4007390.
[5]	U. M. Ascher and E. Haber, Grid refinement and scaling for distributed parameter estimation problems, Inverse Problems, 17 (2001), 571-590. doi: 10.1088/0266-5611/17/3/314.
[6]	L. Biegler, G. Biros, O. Ghattas, M. Heinkenschloss, D. Keyes, B. Mallick, Y. Marzouk, L. Tenorio, B. van Bloemen Waanders and K. Willcox, Large-Scale Inverse Problems and Quantification of Uncertainty, Wiley Ser. Comput. Stat., John Wiley & Sons, Ltd., Chichester, 2011. doi: 10.1002/9780470685853.
[7]	L. T. Biegler, O. Ghattas, M. Heinkenschloss, D. Keyes and B. van Bloemen Waanders, Real-Time PDE-Constrained Optimization, vol. 3, SIAM Computational Science and Engineering, 2007. doi: 10.1137/1.9780898718935.
[8]	L. T. Biegler, O. Ghattas, M. Heinkenschloss and B. van Bloemen Waanders, Large-scale PDE-constrained optimization: An introduction, Large-scale PDE-constrained optimization (Santa Fe, NM, 2001), Springer-Verlag Lecture Notes in Computational Science and Engineering, 30 (2003), 3-13. doi: 10.1007/978-3-642-55508-4_1.
[9]	G. Biros and O. Ghattas, Parallel Lagrange-Newton-Krylov-Schur methods for PDE-constrained optimization. Part Ⅰ: the Krylov-Schur solver, SIAM Journal on Scientific Computing, 27 (2005), 687-713. doi: 10.1137/S106482750241565X.
[10]	J. F. Bonnans and A. Shapiro, Optimization problems with perturbations: A guided tour, SIAM Review, 40 (1998), 228-264. doi: 10.1137/S0036144596302644.
[11]	A. Borzi, High-order discretization and multigrid solution of elliptic nonlinear constrained optimal control problems, Journal of Computational and Applied Mathematics, 200 (2007), 67-85. doi: 10.1016/j.cam.2005.12.023.
[12]	K. Brandes and R. Griesse, Quantitative stability analysis of optimal solutions in PDE-constrained optimization, Journal of Computational and Applied Mathematics, 206 (2007), 908-926. doi: 10.1016/j.cam.2006.08.038.
[13]	J. Brynjarsdóttir and A. O'Hagan, Learning about physical parameters: The importance of model discrepancy, Inverse Problems, 30 (2014), 1-24. doi: 10.1088/0266-5611/30/11/114007.
[14]	D. E. Bryson and M. P. Rumpfkeil, Multifidelity Quasi-Newton method for design optimization, AIAA Journal, 56 (2018), 4074-4086. doi: 10.2514/1.J056840.
[15]	T. Bui-Thanh, A Gentle Tutorial on Statistical Inversion Using the Bayesian Paradigm, Tech. report, The Institute for Computational Engineering and Sciences, 2012.
[16]	T. Bui-Thanh, O. Ghattas, J. Martin and G. Stadler, A computational framework for infinite-dimensional Bayesian inverse problems. Part Ⅰ: The linearized case, with applications to global seismic inversion, SIAM Journal on Scientific Computing, 35 (2013), A2494-A2523. doi: 10.1137/12089586X.
[17]	Y. Daon and G. Stadler, Mitigating the influence of the boundary on PDE-based covariance operators, Inverse Problems and Imaging, 12 (2018), 1083-1102. doi: 10.3934/ipi.2018045.
[18]	A. V. Fiacco and A. Ghaemi, Sensitivity analysis of a nonlinear structural design problem, Computers and Operations Research, 9 (1982), 29-55. doi: 10.1016/0305-0548(82)90005-3.
[19]	P. Gardner, T. J. Rogers, C. Lord and R. J. Barthorpe, Learning model discrepancy: A Gaussian process and sampling-based approach, Mechanical Systems and Signal Processing, 152 (2021), 107381, 1-20. doi: 10.1016/j.ymssp.2020.107381.
[20]	A. A. Gorodetsky, G. Geraci, M. S. Eldred and J. D. Jakeman, A generalized approximate control variate framework for multifidelity uncertainty quantification, Journal of Computational Physics, 408 (2020), 109257, 29 pp. doi: 10.1016/j.jcp.2020.109257.
[21]	R. Griesse, Parametric sensitivity analysis in optimal control of a reaction-diffusion system – part Ⅱ: Practical methods and examples, Optimization Methods and Software, 19 (2004), 217-242. doi: 10.1080/10556780410001654250.
[22]	R. Griesse, Parametric sensitivity analysis in optimal control of a reaction diffusion system. Ⅰ. Solution differentiability, Numerical Functional Analysis and Optimization, 25 (2004), 93-117. doi: 10.1081/NFA-120034120.
[23]	E. Haber and U. M. Ascher, Preconditioned all-at-once methods for large, sparse parameter estimation problems, Inverse Problems, 17 (2001), 1847-1864. doi: 10.1088/0266-5611/17/6/319.
[24]	J. Hart and B. van Bloemen Waanders, Enabling hyper-differential sensitivity analysis for ill-posed inverse problems, SIAM Journal on Scientific Computing, 45 (2023), A1947-A1970. doi: 10.1137/22M147699X.
[25]	J. Hart and B. van Bloemen Waanders, Hyper-differential sensitivity analysis with respect to model discrepancy: Optimal solution updating, Computer Methods in Applied Mechanics and Engineering, 412 (2023), 116082, 18 pp doi: 10.1016/j.cma.2023.116082.
[26]	J. Hart, B. van Bloemen Waanders and R. Hertzog, Hyper-differential sensitivity analysis of uncertain parameters in PDE-constrained optimization, International Journal for Uncertainty Quantification, 10 (2020), 225-248. doi: 10.1615/Int.J.UncertaintyQuantification.2020032480.
[27]	J. Hart, B. van Bloemen Waanders, L. Hood and J. Parish, Sensitivity driven experimental design to facilitate control of dynamical systems, Journal of Optimization Theory and Applications, 196 (2023), 855-881. doi: 10.1007/s10957-023-02172-w.
[28]	S. B. Hazra and V. Schulz, Simultaneous pseudo-timestepping for aerodynamic shape optimization problems with state constraints, SIAM Journal on Scientific Computing, 28 (2006), 1078-1099. doi: 10.1137/05062442X.
[29]	D. Higdon, J. Gattiker, B. Williams and M. Rightley, Computer model calibration using high-dimensional output, Journal of the American Statistical Association, 103 (2008), 570-583. doi: 10.1198/016214507000000888.
[30]	M. Hintermüller and L. N. Vicente, Space mapping for optimal control of partial differential equations, SIAM J. Opt., 15 (2005), 1002-1025. doi: 10.1137/S105262340342907X.
[31]	M. Hinze, R. Pinnau, M. Ulbrich and S. Ulbrich, Optimization with PDE constraints, Math. Model. Theory Appl., 23. Springer, New York, 2009. doi: 10.1007/978-1-4020-8839-1.
[32]	M. C. Kennedy and A. O'Hagan, Bayesian calibration of computer models, Journal of the Royal Statistical Society, 63 (2001), 425-464. doi: 10.1111/1467-9868.00294.
[33]	U. Khristenko, L. Scarabosio, P. Swierczynski, E. Ullmann and B. Wohlmuth, Analysis of boundary effects on PDE-based sampling of Whittle–Matérn random fields, SIAM/ASA Journal on Uncertainty Quantification, 7 (2019), 948-974. doi: 10.1137/18M1215700.
[34]	C. D. Laird, L. T. Biegler, B. van Bloemen Waanders and R. A. Bartlett, Contamination Source Determination for Water Networks, ASCE J. Water Res. Mgt. Plan., (2005), 125-134. doi: 10.1061/(ASCE)0733-9496(2005)131:2(125).
[35]	F. Lindgren, H. Rue and J. Lindström, An explicit link between Gaussian fields and Gaussian Markov random fields: The stochastic partial differential equation approach, Journal of the Royal Statistical Society Series B, 73 (2011), 423-498. doi: 10.1111/j.1467-9868.2011.00777.x.
[36]	Y. Ling, J. Mullins and S. Mahadevan, Selection of model discrepancy priors in Bayesian calibration, Journal of Computational Physics, 276 (2014), 665-680. doi: 10.1016/j.jcp.2014.08.005.
[37]	A. March and K. Willcox, A provably convergent multifidelity optimization algorithm not requiring high-fidelity derivatives, AIAA Journal, 50 (2012), 1079-1089. doi: 10.2514/6.2010-2912.
[38]	K. A. Maupin and L. P. Swiler, Model discrepancy calibration across experimental settings, Reliability Engineering & System Safety, 200 (2020), 1-10. doi: 10.2172/1833158.
[39]	F. Menhorn, G. Geraci, D. T. Seidl, M. S. Eldred, R. King, H.-J. Bungartz and Y. Marzouk, Higher moment multilevel estimators for optimization under uncertainty applied to wind plant design, AIAA SciTech Forum, (2020), 1-16. doi: 10.2514/6.2020-1952.
[40]	L. W. T. Ng and K. E. Willcox, Multifidelity approaches for optimization under uncertainty, International Journal for Numerical Methods in Engineering, 100 (2014), 746-772. doi: 10.1002/nme.4761.
[41]	J. Oakley and A. O'Hagan, Bayesian inference for the uncertainty distribution of computer model outputs, Biometrika, 89 (2002), 769-784.
[42]	B. Peherstorfer, K. Willcox and M. Gunzburger, Survey of multifidelity methods in uncertainty propagation, inference, and optimization, SIAM Review, 60 (2018), 550-591. doi: 10.1137/16M1082469.
[43]	N. Petra, J. Martin, G. Stadler and O. Ghattas, A computational framework for infinite-dimensional Bayesian inverse problems. Part Ⅱ: Stochastic Newton MCMC with application to ice sheet flow inverse problems, SIAM Journal on Scientific Computing, 36 (2014), A1525-A1555. doi: 10.1137/130934805.
[44]	A. K. Saibaba, J. Hart and B. van Bloemen Waanders, Randomized algorithms for generalized singular value decomposition with application to sensitivity analysis, Numerical Linear Algebra with Applications, 28 (2021), e2364, 27 pp. doi: 10.1002/nla.2364.
[45]	A. M. Stuart, Inverse problems: A Bayesian perspective, Acta Numerica, 19 (2010), 451-559. doi: 10.1017/S0962492910000061.
[46]	I. Sunseri, A. Alexanderian, J. Hart and B. van Bloemen Waanders, Hyper-differential sensitivity analysis for nonlinear Bayesian inverse problems, International Journal for Uncertainty Quantification, 14 (2024), 1-20. doi: 10.1615/Int.J.UncertaintyQuantification.2023045300.
[47]	I. Sunseri, J. Hart, B. van Bloemen Waanders and A. Alexanderian, Hyper-differential sensitivity analysis for inverse problems constrained by partial differential equations, Inverse Problems, 36 (2020), 125001, 23 pp. doi: 10.1088/1361-6420/abaf63.
[48]	C. R. Vogel, Sparse matrix computations arising in distributed parameter identification, SIAM Journal on Matrix Analysis and Applications, 20 (1999), 1027-1037. doi: 10.1137/S0895479897317703.
[49]	C. R. Vogel, Computational Methods for Inverse Problems, Frontiers Appl. Math., 23. SIAM Frontiers in Applied Mathematics Series, 2002. doi: 10.1137/1.9780898717570.
[50]	P. Whittle, On stationary processes in the plane, Biometrika, 41 (1954), 434-449. doi: 10.1093/biomet/41.3-4.434.