Reduced basis approximations of parameterized dynamical partial differential equations via neural networks

Peter Sentz; Kristian Beckwith; Eric C. Cyr; Luke N. Olson; Ravi Patel

doi:10.3934/fods.2024044

Article Contents

2025, Volume 7, Issue 1: 338-362. Doi: 10.3934/fods.2024044

This issue Previous Article Scientific machine learning for closure models in multiscale problems: A review Next Article Neural network approaches for parameterized optimal control

Reduced basis approximations of parameterized dynamical partial differential equations via neural networks

1.
Brown University, USA
2.
Sandia National Laboratories, USA
3.
University of Illinois at Urbana-Champaign, USA

^*Corresponding author: Peter Sentz
^*Corresponding author: Peter Sentz

Received: January 2024

Revised: July 2024

Early access: October 2024

Published: March 2025

Abstract / Introduction Full Text(HTML) Figure(9) / Table(2) Related Papers Cited by

Abstract

Projection-based reduced order models are effective at approximating parameter-dependent differential equations that are parametrically separable. When parametric separability is not satisfied, which occurs in both linear and nonlinear problems, projection-based methods fail to adequately reduce the computational complexity. Devising alternative reduced order models is crucial for obtaining efficient and accurate approximations to expensive high-fidelity models. In this work, we develop a timestepping procedure for dynamical parameter-dependent problems, in which a neural-network is trained to propagate the coefficients of a reduced basis expansion. This results in an online stage with a computational cost independent of the size of the underlying problem. We demonstrate our method on several parabolic partial differential equations, including a problem that is not parametrically separable.

Keywords:

Reduced basis methods,
proper orthogonal decomposition,
neural networks,
parameterized partial differential equations,
finite elements.

Mathematics Subject Classification: Primary: 68T07, 65M99.

Citation:

Full Text(HTML)

Figure 1. One residual layer of a ResNet showing the structure of the residual basic block used. The matrices and bias vectors define the parameters for this layer

Download: Full-size image PowerPoint slide

Figure 2. A two layer ResNet for timestepping (see also Equation (31)). The residual $ R_j $ functions contain the weights and biases – cf. Figure 1. For compactness, we have introduced the notation $ \tilde{t}_k = t_k + \Delta t/2 $ for the initial input

Download: Full-size image PowerPoint slide

Figure 3. 2D advection-diffusion: The mean relative error over the test parameters as a function of time. The measure in Equation (39) is evaluated for the reconstructed solutions using the $ L^2 $-projection (black), the neural network approximation (red), and the coefficients computed using Galerkin POD (blue). The timesteps are marked every 5 steps

Download: Full-size image PowerPoint slide

Figure 4. 2D advection-diffusion: (Left) Mean relative $ L^2 $-error Equation (39) with respect to the full-order solution for a timestep size of $ 10\Delta t $. The shaded region denotes $ \pm $ one standard deviation over the test set. Timesteps are marked every 4 steps. (Right) Mean relative $ L^2 $-error Equation (39) with respect to the full-order solution for a timestep size of $ 20\Delta t $. The shaded region denotes $ \pm $ one standard deviation over the test set. Timesteps are marked every 2 steps

Download: Full-size image PowerPoint slide

Figure 5. 1D advection-diffusion: The mean relative error over the test parameters as a function of time. The measure in Equation (39) is evaluated for the reconstructed solutions using the $ L^2 $-projection (black), the neural network approximation (red), and the coefficients computed using Galerkin POD (blue). The timesteps are marked every 10 steps

Download: Full-size image PowerPoint slide

Figure 6. 1D advection-diffusion: Two solution profiles at different parameter values. The pointwise relative error (in magnitude) with the full-order solution in the neural network approximation (solid) and the POD (dashed) is shown in the background

Download: Full-size image PowerPoint slide

Figure 7. 1D advection-diffusion: Relative $ L^2 $-error Equation (38) as a function of time using the $ L^2 $-projection (black), the neural network approximation (red), and the coefficients computed using Galerkin POD (blue). The timesteps are marked every 10 steps. (Left) Error corresponding to $ Pe = .105 $, the smallest Peclet number in the test set. (Right) Error corresponding to $ Pe = 18.94 $, the largest Peclet number in the test set

Download: Full-size image PowerPoint slide

Figure 8. Non-affine diffusion-reaction: Solution snapshots at various points in time

Download: Full-size image PowerPoint slide

Figure 9. Non-affine diffusion-reaction: The mean relative error over the test parameters as a function of time. The measure in Equation (39) is evaluated for the reconstructed solutions using the $ L^2 $-projection (black), and the neural network approximation (red). The model is trained on the time interval to the left of the dashed vertical line; to the right of the dashed vertical line corresponds to extrapolation

Download: Full-size image PowerPoint slide

Table 1. Model parameters and execution times for the examples in Section 5. All numerical experiments were executed on a MacBook Pro with 2 GHz Quad-Core Intel Core i5 processor

Parameter	2D Adv. Diff. (33)	1D Adv. Diff. (40)	2D non-affine (41)
# of residual blocks	3	2	2
hidden layers per block	2	2	4
width of each block	12	13	18
total # of weights/biases	1,101	900	2,640
rollout length $ m $	3	4	5
activation function	tanh	tanh	tanh
optimizer	L-BFGS	L-BFGS	L-BFGS
training set size	100 $ {\mathit{\boldsymbol{\mu}}} $ values	100 $ {\mathit{\boldsymbol{\mu}}} $ values	225 $ {\mathit{\boldsymbol{\mu}}} $ values
training set size	50 timesteps	100 timesteps	40 timesteps
NN training time	$ \sim $ 1.5 m	$ \sim $ 3 h	$ \sim $ 1.5 h
online NN cost per parameter	$ \sim 1 $ ms	$ \sim 0.4 $ ms	$ \sim 0.4 $ ms
full-order cost per parameter	$ \sim 60 $ ms	$ \sim 10 $ ms	$ \sim 70 $ ms

| Show Table

DownLoad: CSV

Table 2. Notation used throughout

Variable	Definition
$ \Omega \in \mathbb{R}^d $	spatial domain
$ T $	end time of the simulation
$ u({\mathit{\boldsymbol{x}}}, t; {\mathit{\boldsymbol{\mu}}}) $	PDE solution
$ F $	differential operator
$ {\mathit{\boldsymbol{\mu}}} $	vector of $ P $ parameters, $ \mu_1 $, $\dots, \mu_P $
$ V^h $: $ \phi_1 $, $\dots, \phi_{N_h} $	full-order basis on mesh $ h $
$ V^{ \text{rb}} $: $ \psi_1 $, $ \dots, \psi_{N_{ \text{rb}}} $	reduced basis on mesh $ h $
$ t_k $	$ k=1 $, $\dots, N_t $ timestep $ k $ with $ t_{N_{t}} = T $
$ N_t $	number of timesteps to the end time
$ \mathcal{D} $	parameter space
$ N_\text{s} $	number of POD samples
$ \mathcal{D}_{\text{POD}} $	POD sample space
$ S_j $	data matrix for parameter sample $ {{\mathit{\boldsymbol{\mu}}}}_j $
$ U $	data matrix combining all compressed samples
$ N_{ \text{rb}_j} $	first stage basis functions for parameter sample $ j $ (SVD)
$ \hat{n}_i $	# of retained singular values for parameter sample $ i $
$ \epsilon_{{\mathit{\boldsymbol{\mu}}}} $	stopping criteria, first stage
$ \epsilon_{t} $	stopping criteria, second stage
$ {\mathit{\boldsymbol{\alpha}}}(t;{\mathit{\boldsymbol{\mu}}}) $	vector of full-order basis coefficients
$ {\mathit{\boldsymbol{c}}}(t;{\mathit{\boldsymbol{\mu}}}) $	vector of reduced basis coefficients
$ u^{rb} $	reduced order PDE solution
$ \mathcal{N}_R $	mapping of a parameter to RB coefficient
$ L_T $	least squares single-step loss
$ L_T^m $	least squares multi-step loss
$ \mathcal{T} $	a training set specifying time nodes and parameters
$ \mathcal{B} $	a mini-batch defined as a subset of $ \mathcal{T} $
$ \mathcal{N}_T $	the neural net and mapping
$ \mathcal{D}_{\text{train}} $	training set of parameters
$ N_{\text{train}} $	number of training samples
$ \mathcal{F} $	training data features
$ \mathcal{T} $	training data targets
$ L $	number of layers in a ResNet
$ R_i $	ResNet block
$ I $	identity
$ A^{(j)} $	dense NN layer
$ W^{(j)} $	weight matrix
$ b^{(j)} $	bias vector
$ \sigma $	activation function

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	S. E. Ahmed, O. San, A. Rasheed and T. Iliescu, Nonlinear proper orthogonal decomposition for convection-dominated flows, Phys. Fluids, 33 (2021). doi: 10.1063/5.0074310.
[2]	J. Baker, E. Cherkaev, A. Narayan and B. Wang, Learning proper orthogonal decomposition of complex dynamics using heavy-ball neural ODEs, J. Sci. Comput., 95 (2023), Paper No. 54, 27 pp. doi: 10.1007/s10915-023-02176-8.
[3]	P. Benner, S. Gugercin and K. Willcox, A survey of projection-based model reduction methods for parametric dynamical systems, SIAM Rev., 57 (2015), 483-531. doi: 10.1137/130932715.
[4]	R. Bollapragada, J. Nocedal, D. Mudigere, H. J. Shi and P. T. P. Tang, A progressive batching L-BFGS method for machine learning, Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, (2018), 620-629.
[5]	K. Carlberg, M. Barone and H. Antil, Galerkin v. least-squares Petrov–Galerkin projection in nonlinear model reduction, J. Comput. Phys., 330 (2017), 693-734. doi: 10.1016/j.jcp.2016.10.033.
[6]	F. Casenave, A. Ern and T. Lelievre, A nonintrusive reduced basis method applied to aeroacoustic simulations, Adv. Comput. Math., 41 (2015), 961-986. doi: 10.1007/s10444-014-9365-0.
[7]	S. Chaturantabut and D. C. Sorensen, Nonlinear model reduction via discrete empirical interpolation, SIAM J. Sci. Comput., 32 (2010), 2737-2764. doi: 10.1137/090766498.
[8]	J. H. Chaudhry, L. N. Olson and P. Sentz, A least-squares finite element reduced basis method, SIAM J. Sci. Comput., 43 (2021), A1081-A1107. doi: 10.1137/090766498.
[9]	R. T. Q. Chen, Y. Rubanova, J. Bettencourt and D. K. Duvenaud, Neural ordinary differential equations, Advances in Neural Information Processing Systems, Montréal, Canada, (2018).
[10]	M. Drohmann, B. Haasdonk and M. Ohlberger, Reduced basis approximation for nonlinear parametrized evolution equations based on empirical operator interpolation, SIAM J. Sci. Comput., 34 (2012), A937-A969. doi: 10.1137/10081157X.
[11]	B. A. Freno and K. T. Carlberg, Machine-learning error models for approximate solutions to parameterized systems of nonlinear equations, Comput. Methods Appl. Mech. Engrg., 348 (2019), 250-296. doi: 10.1016/j.cma.2019.01.024.
[12]	S. Fresca, L. Dede´ and A. Manzoni, A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs, J. Sci. Comput., 87 (2021), Paper No. 61, 36 pp. doi: 10.1007/s10915-021-01462-7.
[13]	S. Fresca and A. Manzoni, POD-DL-ROM: enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition, Comput. Methods Appl. Mech. Engrg., 388 (2022), 114181, 27 pp. doi: 10.1016/j.cma.2021.114181.
[14]	S. Giere, T. Iliescu, V. John and D. Wells, SUPG reduced order models for convection-dominated convection–diffusion–reaction equations, Comput. Methods Appl. Mech. Engrg., 289 (2015), 454-474. doi: 10.1016/j.cma.2015.01.020.
[15]	M. A. Grepl, Certified reduced basis methods for nonaffine linear time-varying and nonlinear parabolic partial differential equations, Math. Models Methods Appl. Sci., 22 (2012), 1150015. doi: 10.1142/S0218202511500151.
[16]	M. A. Grepl, Y. Maday, N. C. Nguyen and A. T. Patera, Efficient reduced-basis treatment of nonaffine and nonlinear partial differential equations, ESAIM Math. Model. Numer. Anal., 41 (2007), 575-605. doi: 10.1051/m2an:2007031.
[17]	E. Haber and L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2017), 014004. doi: 10.1088/1361-6420/aa9a90.
[18]	K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770-778. doi: 10.1109/CVPR.2016.90.
[19]	J. S. Hesthaven and S. Ubbiali, Non-intrusive reduced order modeling of nonlinear problems using neural networks, J. Comput. Phys., 363 (2018), 55-78. doi: 10.1016/j.jcp.2018.02.037.
[20]	R. Lam, A. Sanchez-Gonzalez, M. Willson, P. Wirnsberger, M. Fortunato, F. Alet, S. Ravuri, T. Ewalds, Z. Eaton-Rosen, W. Hu, A. Merose, S. Hoyer, G. Holland, O. Vinyals, J. Stott, A. Pritzel, S. Mohamed and P. Battaglia, Learning skillful medium-range global weather forecasting, Science, 382 (2023), 1416-1421. doi: 10.1126/science.adi2336.
[21]	D. C. Liu and J. Nocedal, On the limited memory BFGS method for large scale optimization, Math. Program., 45 (1989), 503-528. doi: 10.1007/BF01589116.
[22]	R. Maulik, B. Lusch and P. Balaprakash, Reduced-order modeling of advection-dominated systems with recurrent neural networks and convolutional autoencoders, Phys. Fluids, 33 (2021). doi: 10.1063/5.0039986.
[23]	J. Nocedal, Updating quasi-Newton matrices with limited storage, Math. Comp., 35 (1980), 773-782. doi: 10.1090/S0025-5718-1980-0572855-7.
[24]	M. Ohlberger and S. Rave, Reduced basis methods: Success, limitations and future challenges, preprint, (2015), arXiv: 1511.02021.
[25]	P. Pacciarini, P. Gervasio and A. Quarteroni, Spectral based discontinuous Galerkin reduced basis element method for parametrized Stokes problems, Comput. Math. Appl., 72 (2016), 1977-1987. doi: 10.1016/j.camwa.2016.01.030.
[26]	E. J. Parish and K. T. Carlberg, Time-series machine-learning error models for approximate solutions to parameterized dynamical systems, Comput. Methods Appl. Mech. Engrg., 365 (2020), 112990, 44 pp. doi: 10.1016/j.cma.2020.112990.
[27]	R. G. Patel, N. A. Trask, M. A. Wood and E. C. Cyr, A physics-informed operator regression framework for extracting data-driven continuum models, Comput. Methods Appl. Mech. Engrg., 373 (2021), 113500, 23 pp. doi: 10.1016/j.cma.2020.113500.
[28]	J. Pathak, S. Subramanian, P. Harrington, S. Raja, A. Chattopadhyay, M. Mardani, T. Kurth, D. Hall, Z. Li, K. Azizzadenesheli, P. Hassanzadeh, K. Kashinath and A. Anandkumar, Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators, preprint, (2022), arXiv: 2202.11214.
[29]	A. Quarteroni, A. Manzoni and F. Negri, Reduced Basis Methods for Partial Differential Equations: An Introduction, Springer International Publishing, Switzerland, 2016. doi: 10.1007/978-3-319-15431-2.
[30]	F. Rathgeber, D. A. Ham, L. Mitchell, M. Lange, F. Luporini, A. T. T. McRae, G. T. Bercea, G. R. Markall and P. H. J. Kelly, Firedrake: Automating the finite element method by composing abstractions, ACM Trans. Math. Software, 43 (2016), Art. 24, 27 pp. doi: 10.1145/2998441.
[31]	C. J. G. Rojas, A. Dengel and M. D. Ribeiro, Reduced-order model for fluid flows via neural ordinary differential equations, preprint, (2021), arXiv: 2102.02248.
[32]	G. Rozza, D. B.P Huynh and A. T. Patera, Reduced basis approximation and a posteriori error estimation for affinely parametrized elliptic coercive partial differential equations, Arch. Comput. Methods Eng., 15 (2008), 229-275. doi: 10.1007/s11831-008-9019-9.
[33]	S. Rubino, A streamline derivative POD-ROM for advection-diffusion-reaction equations, ESAIM Proc. Surveys, 64 (2018), 121-136. doi: 10.1051/proc/201864121.
[34]	Q. Wang, J. S. Hesthaven and D. Ray, Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem, J. Comput. Phys., 384 (2019), 289-307. doi: 10.1016/j.jcp.2019.01.031.
[35]	E Weinan, A proposal on machine learning via dynamical systems, Commun. Math. Stat., 5 (2017), 1-11. doi: 10.1007/s40304-017-0103-z.
[36]	K. Wu and D. Xiu, Data-driven deep learning of partial differential equations in modal space, J. Comput. Phys., 408 (2020), 109307. doi: 10.1016/j.jcp.2020.109307.
[37]	J. Xu and K. Duraisamy, Multi-level convolutional autoencoder networks for parametric prediction of spatio-temporal dynamics, Comput. Methods Appl. Mech. Engrg., 372 (2020), 113379, 36 pp. doi: 10.1016/j.cma.2020.113379.