Stacked networks improve physics-informed training: Applications to neural networks and deep operator networks

Amanda A. Howard; Sarah H. Murphy; Shady E. Ahmed; Panos Stinis

doi:10.3934/fods.2024029

Article Contents

2025, Volume 7, Issue 1: 134-162. Doi: 10.3934/fods.2024029

This issue Previous Article Hyper-differential sensitivity analysis with respect to model discrepancy: Posterior optimal solution sampling Next Article An operator learning perspective on parameter-to-observable maps

Stacked networks improve physics-informed training: Applications to neural networks and deep operator networks

1.
Pacific Northwest National Laboratory, USA
2.
University of North Carolina, Charlotte, USA
3.
Department of Applied Mathematics, University of Washington, USA
4.
Division of Applied Mathematics, Brown University, USA

^*Corresponding author: Panos Stinis
^*Corresponding author: Panos Stinis

Received: December 2023

Revised: May 2024

Early access: June 2024

Published: March 2025

Abstract / Introduction Full Text(HTML) Figure(22) / Table(6) Related Papers Cited by

Abstract

Physics-informed neural networks and operator networks have shown promise for effectively solving equations modeling physical systems. However, these networks can be difficult or impossible to train accurately for some systems of equations. We present a novel multifidelity framework for stacking physics-informed neural networks and operator networks that facilitates training. We successively build a chain of networks, where the output at one step can act as a low-fidelity input for training the next step, gradually increasing the expressivity of the learned model. The equations imposed at each step of the iterative process can be the same or different (akin to simulated annealing). The iterative (stacking) nature of the proposed method allows us to progressively learn features of a solution that are hard to learn directly. Through benchmark problems including a nonlinear pendulum, the wave equation, and the viscous Burgers equation, we show how stacking can be used to improve the accuracy and reduce the required size of physics-informed neural networks and operator networks.

Keywords:

Mathematics Subject Classification: Primary: 68T07; Secondary: 68T99.

Citation:

Full Text(HTML)

Figure 1. Graphical abstract. A previous prediction is used as the low-fidelity input for a physics-informed multifidelity neural network to generate a more accurate prediction as the output

Download: Full-size image PowerPoint slide

Figure 2. Physics-informed neural network

Download: Full-size image PowerPoint slide

Figure 3. PINN results $ s_1 $ (left) and $ s_2 $ (right) as a function of time for the pendulum problem for ten random initial seedings. In each case, the PINN solution decays and does not agree well with the exact solution

Download: Full-size image PowerPoint slide

Figure 4. Multifidelity physics-informed neural network

Download: Full-size image PowerPoint slide

Figure 5. Stacking multifidelity physics-informed neural network

Download: Full-size image PowerPoint slide

Figure 6. Stacking PINN training for the pendulum problem

Download: Full-size image PowerPoint slide

Figure 7. Results for the multiscale problem. (A) Single fidelity results for a variety of network sizes. (B) Multifidelity stacking results. (C) Relative $ \ell_2 $ errors for the stacking PINN. The black dashed line in (C) is the lowest error from the single fidelity training for the $ 5\times 64 $ network. For the stacking PINN, step 0 is the single fidelity step and step 1 is the first multifidelity step

Download: Full-size image PowerPoint slide

Figure 8. Results for the stacking PINN for the wave equation for Case 1. The top left figure is the exact solution. The results from training steps 0, 2, and 14 are in the middle column. Their respective absolute errors are in the right column

Download: Full-size image PowerPoint slide

Figure 9. Results for the stacking PINN for the wave equation for Case 3. The left column has the exact solution for the value of $ c $ used for a given stacking step. The results from training steps 0, 2, and 10 are in the middle column. Their respective absolute errors are in the right column

Download: Full-size image PowerPoint slide

Figure 10. Relative $ \ell_2 $ errors for the stacking PINNs for the wave equations for Case 1 (circles), Case 2 (triangles), and Case 3 (squares). The relative $ \ell_2 $ is calculated using the exact solution for the wave equation with $ c = 2 $

Download: Full-size image PowerPoint slide

Figure 11. Results for the 2D wave equation at two different times, $ t = 0.6 $ (A) and $ t = 1 $ (B)

Download: Full-size image PowerPoint slide

Figure 12. Relative $ \ell_2 $ error for the 2D wave equation calculated at 125,000 evenly spaced grid points in the training domain

Download: Full-size image PowerPoint slide

Figure 13. Single fidelity DeepONet results for a random initial condition not seen during training. The exact solution is in the top left. Training with fixed weights is in the top middle column, and training with NTK weights is in the top right column. The bottom has slices of the solution at $ t = 0, 0.5 $, and $ 1 $. Clearly, the addition of the NTK weights improves the accuracy of the results

Download: Full-size image PowerPoint slide

Figure 14. Evolution of the relative $ \ell_2 $ error for the four DeepONet cases considered. The relative $ \ell_2 $ error is calculated with respect to the numerically generated solution for 100 solutions in the test set for $ \nu = 0.0001 $. "Fixed" denotes cases where $ \nu $ is fixed, and "Change" denotes cases where $ \nu $ starts large and is decreased. All errors are calculated with respect to the exact solutions with $ \nu = 0.0001 $

Download: Full-size image PowerPoint slide

Figure 15. Results for a stacking DeepONet with fixed $ \nu $ and fixed weights. Representative steps are shown from the training, including the final stacking step 10. While the results do improve from the initial predictions in early stacking steps, the results struggle in the vicinity of the sharp gradients

Download: Full-size image PowerPoint slide

Figure 16. Results for a stacking DeepONet with changing $ \nu $ and fixed weights. Representative steps are shown from the training, including the final stacking step 10

Download: Full-size image PowerPoint slide

Figure 17. Results for a stacking DeepONet with fixed $ \nu $ and NTK weights. Representative steps are shown from the training, including the final stacking step 6

Download: Full-size image PowerPoint slide

Figure 18. Results for a stacking DeepONet with changing $ \nu $ and NTK weights. Representative steps are shown from the training, including the final stacking step 6

Download: Full-size image PowerPoint slide

Figure 19. Results for the stacking PINN for the wave equation for Case 2. The left column has the exact solution for the value of $ c $ used for a given stacking step. The results from training steps 0, 2, and 12 are in the middle column. Their respective absolute errors are in the right column

Download: Full-size image PowerPoint slide

Figure 20. Results for the stacking PINN for the wave equation for Case 4. The left column has the exact solution for the value of $ c $ used for a given stacking step. The results from training steps 0, 2, 10, and 20 are in the middle column. Their respective absolute errors are in the right column

Download: Full-size image PowerPoint slide

Figure 21. Results for the stacking PINN for the wave equation for Case 5. The left column has the exact solution for the value of $ c $ used for a given stacking step. The results from training steps 0, 2, 10, and 20 are in the middle column. Their respective absolute errors are in the right column

Download: Full-size image PowerPoint slide

Figure 22. Relative $ \ell_2 $ errors for the stacking PINN for the wave equation for Cases 4 and 5 with $ c = 4 $. The error is calculated with respect to the exact solution for the value of $ c = 4 $. In Case 4, $ c $ is gradually increased, so the relative $ \ell_2 $ error starts large and then quickly decreases once $ c = 4 $. In Case 5, $ c = 4 $ for all stacking steps, so the relative error is lower initially but plateaus at a higher value than Case 4

Download: Full-size image PowerPoint slide

Table 1. Relative $ \ell_2 $ errors and network sizes for the multiscale problem in Sec. 3.2. A network size of $ a \times b $ indicates $ a $ hidden layers with $ b $ neurons each. For the stacking results, the nonlinear network size is given first, and then the linear network size. All stacking PINNs begin with a single fidelity network with size 3$ \times $32. For the stacking cases, we report the first stacking level that has a relative $ \ell_2 $ error less than the minimum single fidelity error, and the relative $ \ell_2 $ error after the tenth stacking level

Method	Network size	Trainable parameters	Final relative error	Computational time (h)
Single fidelity	3$ \times $32	2209	1.3419	0.169
Single fidelity	4$ \times $64	12673	0.6482	0.183
Single fidelity	4$ \times $128	49921	0.1123	0.185
Single fidelity	5$ \times $64	16833	0.0265	0.196
Stacking	$ 4 \times 16 $, $ 1\times $ 5, 3 levels	4900	0.0249	0.450
Stacking	$ 4 \times 16 $, $ 1\times $ 5, 10 levels	11179	0.0061	1.579

| Show Table

DownLoad: CSV

Table 2. Schedule for increasing $ c $ with stacking steps for each of the cases considered for the wave equation case. The right column has the relative $ \ell_2 $ error for the final stacking step. Note that the relative error is calculated with respect to the exact solution with $ c = 2 $ for cases 1, 2, and 3, and $ c = 4 $ for Case 4

Case	Schedule for $ c $	Error
Case 1	$ c=2 $ for all steps	$ 2.913 \times 10^{-3} $
Case 2	$ c = [1, 1.25, 1.5, 1.75, 2, 2, 2, 2, \ldots] $	$ 4.274 \times 10^{-3} $
Case 3	$ c = [1, 2, 2, 2, 2, 2, 2, 2, \ldots] $	$ 3.805 \times 10^{-3} $
Case 4	$ c = [1, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, \ldots] $	$ 8.396 \times 10^{-3} $
Case 5	$ c=4 $ for all steps	$ 2.630 \times 10^{-2} $
Single fidelity	$ c = 2 $	$ 4.129\times 10^{-2} $
Single fidelity	$ c = 4 $	1.379

| Show Table

DownLoad: CSV

Table 3. Cases used in training for the stacking DeepONets for Burgers equation. A fixed schedule for $ \nu $ denotes that $ \nu = 0.0001 $ for all stacking levels. A changing schedule denotes that $ \nu = 0.01 $ for Step 0, $ \nu = 0.001 $ for Step 1, and then $ \nu = 0.0001 $ for all remaining stacking levels. The reported error is the relative $ \ell_2 $ error

Case	Schedule for $ \nu $	Weighting scheme	Stacking steps	Error
Case 1	Fixed	Fixed	10	$ 0.2183 \pm 0.1010 $
Case 2	Changing	Fixed	10	$ 0.1463 \pm 0.0908 $
Case 3	Fixed	NTK	6	$ 0.1006 \pm 0.0627 $
Case 4	Changing	NTK	6	$ 0.0592 \pm 0.0488 $
Single fidelity	–	Fixed	–	$ 0.2660 \pm 0.1352 $
Single fidelity	–	NTK	–	$ 0.1119 \pm 0.0636 $

| Show Table

DownLoad: CSV

Table 4. Training parameters for the results in Secs. 3.1 and 3.2. For the learning rate, the triplet $ (a, b, c) $ denotes the $ \texttt{exponential}\_\texttt{decay}$ function in JAX [10] with learning rate $ a $, decay steps $ b $, and decay rate $ c $. The PINN parameters refer to the parameters for cases without stacking

	Sec. 3.1	Sec. 3.2
PINN parameters
Learning rate	($ 10^{-3} $, 2000, .99)	($ 10^{-3} $, 2000, .99)
Network size	[1, 200, 200, 200, 2]	Varies
Activation function	$ \texttt{swish}$	$ \texttt{swish}$
BC batch size	1	1
Residual batch size	200	400
Iterations	400000	400000
$ \lambda_{r} $	1.0	10.0
$ \lambda_{bc} $	–	–
$ \lambda_{ic} $	20.0	1.0
Stacking parameters
Step 0 learning rate	($ 10^{-3} $, 2000, .99)	($ 10^{-3} $, 2000, .99)
Step 0 network size	[1, 100, 100, 100, 2]	[1, 32, 32, 32, 1]
Nonlinear network size	[3, 50, 50, 50, 50, 50, 2]	[2, 16, 16, 16, 16, 1]
Linear network size	[2, 20, 2]	[1, 5, 1]
MF learning rate	($ 10^{-3} $, 2000, .99)	($ 10^{-3} $, 2000, .99)
Activation function	$ \texttt{swish}$	$ \texttt{swish}$
BC batch size	1	1
Residual batch size	200	400
Iterations	100000	200000
$ \lambda_{r} $	1.0	10.0
$ \lambda_{bc} $	–	–
$ \lambda_{ic} $	1.0	1.0

| Show Table

DownLoad: CSV

Table 5. Training parameters for the results in Secs. 3.3 and 3.4. For the learning rate, the triplet $ (a, b, c) $ denotes the $ \texttt{exponential}\_\texttt{decay}$ function in JAX [10] with learning rate $ a $, decay steps $ b $, and decay rate $ c $. The PINN parameters refer to the parameters for cases without stacking

	Sec. 3.3	Sec. 3.4
PINN parameters
Learning rate	($ 10^{-4} $, 2000, .99)
Network width	100
Network layers	5
Activation function	$ \texttt{tanh}$
BC batch size	300
Residual batch size	300
Iterations	400000
$ \lambda_{r} $	1.0
$ \lambda_{bc} $	1.0
$ \lambda_{ic} $	20.0
Stacking parameters
Step 0 learning rate	($ 10^{-4} $, 2000, .99)	($ 10^{-4} $, 2000, .99)
Step 0 network width	100	100
Step 0 network layers	5	5
Nonlinear network width	100	100
Nonlinear network layers	5	5
Linear network size	[1, 1]	[1, 1]
MF learning rate	($ 5\times 10^{-4} $, 2000, .99)	($ 5\times 10^{-4} $, 2000, .99)
Activation function	$ \texttt{tanh}$	$ \texttt{sin}$
BC batch size	300	300
Residual batch size	300	1000
Iterations	100000	100000
$ \lambda_{r} $	1.0	1.0
$ \lambda_{bc} $	1.0	1.0
$ \lambda_{ic} $	20.0	20.0

| Show Table

DownLoad: CSV

Table 6. Training parameters for the DeepONet results in Sec. 4. For the learning rate, the triplet $ (a, b, c) $ denotes the $ \texttt{exponential}\_\texttt{decay}$ function in JAX with learning rate $ a $, decay steps $ b $, and decay rate $ c $. $ N_u $ is the number of initial conditions used in the training set

DeepONet parameters
Learning rate	($ 10^{-3} $, 5000, .9)
Branch network size	$ [101,100,100,100,100,100,100,100] $
Trunk network size	$ [2,100,100,100,100,100,100,100] $
Activation function	$ \texttt{tanh}$
BC batch size	10000
Residual batch size	10000
Iterations	200000
$ \lambda_{r} $	1.0
$ \lambda_{bc} $	10.0
$ \lambda_{ic} $	10.0
$ N_u $	1000
Stacking DeepONet parameters
Step 0 learning rate	($ 10^{-3} $, 5000, .9)
Step 0 branch network size	$ [101,100,100,100,100,100,100,100] $
Step 0 trunk network size	$ [2,100,100,100,100,100,100,100] $
Step 0 iterations, fixed $ \nu $	200000
Step 0 iterations, changing $ \nu $	100000
Learning rate	($ 5\times 10^{-4} $, 5000, .95)
Nonlinear branch network size	$ [102,100,100,100,100,100,100,100] $
Nonlinear trunk network size	$ [2,100,100,100,100,100,100,100] $
Linear branch network size	$ [1, 20] $
Linear trunk network size	$ [2, 20] $
Activation function	$ \texttt{tanh}$
BC batch size	10000
Residual batch size	10000
Iterations, fixed weights	100000
Iterations, NTK weights	200000
$ \lambda_{r} $	1.0
$ \lambda_{bc} $	10.0
$ \lambda_{ic} $	10.0
$ N_u $	1000

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	M. Ainsworth and J. Dong, Galerkin neural networks: A framework for approximating variational equations with error control, SIAM Journal on Scientific Computing, 43 (2021), A2474-A2501. doi: 10.1137/20M1366587.
[2]	M. Ainsworth and J. Dong, Galerkin neural network approximation of singularly-perturbed elliptic systems, Computer Methods in Applied Mechanics and Engineering, 402 (2022), 115169. doi: 10.1016/j.cma.2022.115169.
[3]	Z. Aldirany, R. Cottereau, M. Laforest and S. Prudhomme, Multi-level neural networks for accurate solutions of boundary-value problems, Comput. Methods Appl. Mech. Engrg, 419 (2024), 23.
[4]	M. Aliakbari, M. Mahmoudi, P. Vadasz and A. Arzani, Predicting high-fidelity multiphysics data from low-fidelity fluid flow and transport solvers using physics-informed neural networks, International Journal of Heat and Fluid Flow, 96 (2022), 109002. doi: 10.1016/j.ijheatfluidflow.2022.109002.
[5]	Z. Allen-Zhu, Y. Li and Z. Song, A convergence theory for deep learning via over-parameterization, International Conference on Machine Learning, PMLR, (2019), 242-252.
[6]	C. Bajaj, L. McLennan, T. Andeen and A. Roy, Recipes for when physics fails: Recovering robust learning of physics informed neural networks, Machine Learning: Science and Technology, 4 (2023), 015013. doi: 10.1088/2632-2153/acb416.
[7]	N. Baker, F. Alexander, T. Bremer, A. Hagberg, Y. Kevrekidis, H. Najm, M. Parashar, A. Patra, J. Sethian and S. Wild, et al., Workshop Report on Basic Research Needs for Scientific Machine Learning: Core Technologies for Artificial Intelligence, Technical Report, USDOE Office of Science (SC), Washington, DC (United States), (2019). doi: 10.2172/1478744.
[8]	M. E. Bento, Physics-guided neural network for load margin assessment of power systems, IEEE Transactions on Power Systems.
[9]	D. Bertsimas and J. Tsitsiklis, Simulated annealing, Statistical Science, 8 (1993), 10-15. doi: 10.1214/ss/1177011077.
[10]	J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne and Q. Zhang, JAX: composable transformations of Python+NumPy programs, (2018), URL http://github.com/google/jax.
[11]	S. L. Brunton, Applying machine learning to study fluid mechanics, Acta Mechanica Sinica, 37 (2021), 1718-1726. doi: 10.1007/s10409-021-01143-6.
[12]	S. Cai, Z. Mao, Z. Wang, M. Yin and G. E. Karniadakis, Physics-informed neural networks (PINNs) for fluid mechanics: A review, Acta Mechanica Sinica, 37 (2021), 1727-1738. doi: 10.1007/s10409-021-01148-1.
[13]	W. Chen and P. Stinis, Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations, Comput. Phys., 498 (2024), 112683.
[14]	Y. Chen, L. Lu, G. E. Karniadakis and L. Dal Negro, Physics-informed neural networks for inverse problems in nano-optics and metamaterials, Optics Express, 28 (2020), 11618-11633. doi: 10.1364/OE.384875.
[15]	P.-Y. Chuang and L. A. Barba, Experience report of physics-informed neural networks in fluid simulations: Pitfalls and frustration, arXiv Preprint arXiv: 2205.14249.
[16]	P.-Y. Chuang and L. A. Barba, Predictive limitations of physics-informed neural networks in vortex shedding, arXiv Preprint arXiv: 2306.00230.
[17]	S. Cuomo, V. S. Di Cola, F. Giampaolo, G. Rozza, M. Raissi and F. Piccialli, Scientific machine learning through physics-informed neural networks: Where we are and what's next, Journal of Scientific Computing, 92 (2022), 88. doi: 10.1007/s10915-022-01939-z.
[18]	A. Daw, J. Bu, S. Wang, P. Perdikaris and A. Karpatne, Mitigating propagation failures in physics-informed neural networks using retain-resample-release (R3) sampling, arXiv Preprint arXiv: 2207.02338.
[19]	S. De, M. Reynolds, M. Hassanaly, R. N. King and A. Doostan, Bi-fidelity modeling of uncertain and partially unknown systems using DeepONets, Computational Mechanics, 71 (2023), 1251-1267. doi: 10.1007/s00466-023-02272-4.
[20]	V. Dolean, A. Heinlein, S. Mishra and B. Moseley, Multilevel domain decomposition-based architectures for physics-informed neural networks, Comput. Methods Appl. Mech. Engrg., 429 (2024), 117116.
[21]	T. A. Driscoll, N. Hale and L. N. Trefethen, Chebfun Guide, Pafnuty Publications, (2014), URL http://www.chebfun.org/docs/guide/.
[22]	S. Du, J. Lee, H. Li, L. Wang and X. Zhai, Gradient descent finds global minima of deep neural networks, International Conference on Machine Learning, PMLR, (2019), 1675-1685.
[23]	P. Dubois, T. Gomez, L. Planckaert and L. Perret, Machine learning for fluid flow reconstruction from limited measurements, Journal of Computational Physics, 448 (2022), 110733.
[24]	Z. Fang and J. Zhan, Deep physical informed neural networks for metamaterial design, IEEE Access, 8 (2019), 24506-24513. doi: 10.1109/ACCESS.2019.2963375.
[25]	Z. Gao, L. Yan and T. Zhou, Failure-informed adaptive sampling for PINNs, SIAM Journal on Scientific Computing, 45 (2023), A1971-A1994. doi: 10.1137/22M1527763.
[26]	S. Goswami, A. Bora, Y. Yu and G. E. Karniadakis, Physics-informed deep neural operators networks, arXiv Preprint arXiv: 2207.05748.
[27]	S. Goswami, K. Kontolati, M. D. Shields and G. E. Karniadakis, Deep transfer operator learning for partial differential equations under conditional shift, Nature Machine Intelligence, 4 (2022), 1155-1164. doi: 10.1038/s42256-022-00569-2.
[28]	S. Goswami, M. Yin, Y. Yu and G. E. Karniadakis, A physics-informed variational DeepONet for predicting crack path in quasi-brittle materials, Computer Methods in Applied Mechanics and Engineering, 391 (2022), 114587. doi: 10.1016/j.cma.2022.114587.
[29]	Y. Hao, P. C. Di Leoni, O. Marxen, C. Meneveau, G. E. Karniadakis and T. A. Zaki, Instability-wave prediction in hypersonic boundary layers with physics-informed neural operators, Journal of Computational Science, 102120.
[30]	Q. He, M. Perego, A. A. Howard, G. E. Karniadakis and P. Stinis, A hybrid deep neural operator/finite element method for ice-sheet modeling, Journal of Computational Physics, 492 (2023), 0112428. doi: 10.1016/j.jcp.2023.112428.
[31]	J. Hou, Y. Li and S. Ying, Enhancing pinns for solving pdes via adaptive collocation point movement and adaptive loss weighting, Nonlinear Dynamics, 111 (2023), 15233-15261. doi: 10.1007/s11071-023-08654-w.
[32]	J. Hou, Y. Li and S. Ying, Enhancing PINNs for solving PDEs via adaptive collocation point movement and adaptive loss weighting, Nonlinear Dynamics, 111 (2023), 15233-15261. doi: 10.1007/s11071-023-08654-w.
[33]	A. Howard, Y. Fu and P. Stinis, A multifidelity approach to continual learning for physical systems, Machine Learning: Science and Technology, 5 (2024), 025042.
[34]	A. A. Howard, M. Perego, G. E. Karniadakis and P. Stinis, Multifidelity deep operator networks for data-driven and physics-informed problems, Journal of Computational Physics, 493 (2023), 112462. doi: 10.1016/j.jcp.2023.112462.
[35]	B. Huang and J. Wang, Applications of physics-informed neural networks in power systems-a review, IEEE Transactions on Power Systems, 38 (2022), 572-588. doi: 10.1109/TPWRS.2022.3162473.
[36]	A. Jacot, F. Gabriel and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, Advances in Neural Information Processing Systems, 31 (2021).
[37]	A. D. Jagtap, D. Mitsotakis and G. E. Karniadakis, Deep learning of inverse water waves problems using multi-fidelity data: Application to Serre–Green–Naghdi equations, Ocean Engineering, 248 (2022), 110775. doi: 10.1016/j.oceaneng.2022.110775.
[38]	X. Jin, S. Cai, H. Li and G. E. Karniadakis, NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations, Journal of Computational Physics, 426 (2021), 109951. doi: 10.1016/j.jcp.2020.109951.
[39]	A. S. Joglekar and A. G. R. Thomas, Machine learning of hidden variables in multiscale fluid simulation, Machine Learning: Science and Technology, 4 (2023), 035049. doi: 10.1088/2632-2153/acf81a.
[40]	G. E. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang and L. Yang, Physics-informed machine learning, Nature Reviews Physics, 3 (2021), 422-440. doi: 10.1038/s42254-021-00314-5.
[41]	S. Koric and D. W. Abueidda, Data-driven and physics-informed deep learning operators for solution of heat conduction equation with parametric heat source, International Journal of Heat and Mass Transfer, 203 (2023), 123809. doi: 10.1016/j.ijheatmasstransfer.2022.123809.
[42]	A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby and M. W. Mahoney, Characterizing possible failure modes in physics-informed neural networks, Advances in Neural Information Processing Systems, 34 (2021), 26548-26560.
[43]	V. Kumar, S. Goswami, D. J. Smith and G. E. Karniadakis, Real-time prediction of multiple output states in diesel engines using a deep neural operator framework, arXiv Preprint arXiv: 2304.00567.
[44]	Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart and A. Anandkumar, Fourier neural operator for parametric partial differential equations, arXiv Preprint arXiv: 2010.08895.
[45]	Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart and A. Anandkumar, Neural operator: Graph kernel network for partial differential equations, arXiv Preprint arXiv: 2003.03485.
[46]	D. Liu and Y. Wang, Multi-fidelity physics-constrained neural network and its application in materials modeling, Journal of Mechanical Design, 141 (2019), 121403. doi: 10.1115/1.4044400.
[47]	L. Liu, K. Nath and W. Cai, A causality-DeepONet for causal responses of linear dynamical systems, arXiv Preprint arXiv: 2209.08397.
[48]	Y.-T. Liu, C.-Y. Wu, T. Chen and Y. Yao, Multi-fidelity surrogate modeling for chemical processes with physics-informed neural networks, Computer Aided Chemical Engineering, Elsevier, 52 (2023), 57-63. doi: 10.1016/B978-0-443-15274-0.50010-X.
[49]	L. Lu, P. Jin and G. E. Karniadakis, DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators, arXiv Preprint arXiv: 1910.03193.
[50]	L. Lu, P. Jin, G. Pang, Z. Zhang and G. E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence, 3 (2021), 218-229. doi: 10.1038/s42256-021-00302-5.
[51]	L. Lu, R. Pestourie, S. G. Johnson and G. Romano, Multifidelity deep neural operators for efficient learning of partial differential equations with application to fast inverse design of nanoscale heat transport, Physical Review Research, 4 (2022), 023210. doi: 10.1103/PhysRevResearch.4.023210.
[52]	Z. Mao, A. D. Jagtap and G. E. Karniadakis, Physics-informed neural networks for high-speed flows, Computer Methods in Applied Mechanics and Engineering, 360 (2020), 112789. doi: 10.1016/j.cma.2019.112789.
[53]	R. Mattey and S. Ghosh, A novel sequential method to train physics informed neural networks for Allen Cahn and Cahn Hilliard equations, Computer Methods in Applied Mechanics and Engineering, 390 (2022), 114474. doi: 10.1016/j.cma.2021.114474.
[54]	L. McClenny and U. Braga-Neto, Self-adaptive physics-informed neural networks using a soft attention mechanism, arXiv Preprint arXiv: 2009.04544.
[55]	X. Meng and G. E. Karniadakis, A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems, Journal of Computational Physics.
[56]	G. S. Misyris, A. Venzke and S. Chatzivasileiadis, Physics-informed neural networks for power systems, 2020 IEEE Power & Energy Society General Meeting (PESGM), IEEE, (2020), 1-5. doi: 10.1109/PESGM41954.2020.9282004.
[57]	R. Mojgani, M. Balajewicz and P. Hassanzadeh, Lagrangian PINNs: A causality-conforming solution to failure modes of physics-informed neural networks, arXiv Preprint arXiv: 2205.02902.
[58]	J. J. J. Molina, K. Ogawa and T. Taniguchi, Stokesian processes: Inferring Stokes flows using physics-informed Gaussian processes, Machine Learning: Science and Technology.
[59]	B. Moseley, A. Markham and T. Nissen-Meyer, Finite basis physics-informed neural networks (FBPINNs): A scalable domain decomposition approach for solving differential equations, Advances in Computational Mathematics, 49 (2023), 62. doi: 10.1007/s10444-023-10065-9.
[60]	C. Moya and G. Lin, DAE-PINN: A physics-informed neural network model for simulating differential algebraic equations with application to power networks, Neural Computing and Applications, 35 (2023), 3789-3804. doi: 10.1007/s00521-022-07886-y.
[61]	M. A. Nabian, R. J. Gladstone and H. Meidani, Efficient training of physics-informed neural networks via importance sampling, Computer-Aided Civil and Infrastructure Engineering, 36 (2021), 962-977. doi: 10.1111/mice.12685.
[62]	M. Penwarden, A. D. Jagtap, S. Zhe, G. E. Karniadakis and R. M. Kirby, A unified scalable framework for causal sweeping strategies for physics-informed neural networks (PINNs) and their temporal decompositions, arXiv Preprint arXiv: 2302.14227.
[63]	M. Penwarden, S. Zhe, A. Narayan and R. M. Kirby, Multifidelity modeling for physics-informed neural networks (PINNs), Journal of Computational Physics, 451 (2022), 110844. doi: 10.1016/j.jcp.2021.110844.
[64]	S. Qadeer, A. Engel, A. Howard, A. Tsou, M. Vargas, P. Stinis and T. Chiang, Efficient kernel surrogates for neural network-based regression, arXiv Preprint arXiv: 2310.18612.
[65]	M. Raissi, P. Perdikaris and G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics, 378 (2019), 686-707. doi: 10.1016/j.jcp.2018.10.045.
[66]	M. Raissi, A. Yazdani and G. E. Karniadakis, Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations, Science, 367 (2020), 1026-1030. doi: 10.1126/science.aaw4741.
[67]	M. Ramezankhani, A. Nazemi, A. Narayan, H. Voggenreiter, M. Harandi, R. Seethaler and A. S. Milani, A data-driven multi-fidelity physics-informed learning framework for smart manufacturing: a composites processing case study, 2022 IEEE 5th International Conference on Industrial Cyber-Physical Systems (ICPS), IEEE, (2022), 1-7. doi: 10.1109/ICPS51978.2022.9816983.
[68]	F. Regazzoni, S. Pagani, A. Cosenza, A. Lombardi and A. Quarteroni, A physics-informed multi-fidelity approach for the estimation of differential equations parameters in low-data or large-noise regimes, Rendiconti Lincei, 32 (2021), 437-470. doi: 10.1007/s12210-021-01000-5.
[69]	F. M. Rohrhofer, S. Posch, C. Gößnitzer and B. Geiger, On the role of fixed points of dynamical systems in training physics-informed neural networks, Transactions on Machine Learning Research, 2023 (2023), 490.
[70]	D. Shu, Z. Li and A. B. Farimani, A physics-informed diffusion model for high-fidelity flow field reconstruction, Journal of Computational Physics, 478 (2023), 111972. doi: 10.1016/j.jcp.2023.111972.
[71]	S. Wang and P. Perdikaris, Long-time integration of parametric evolution equations with physics-informed DeepONets, Journal of Computational Physics, 475 (2023), 111855. doi: 10.1016/j.jcp.2022.111855.
[72]	S. Wang, S. Sankaran and P. Perdikaris, Respecting causality is all you need for training physics-informed neural networks, arXiv Preprint arXiv: 2203.07404.
[73]	S. Wang, H. Wang and P. Perdikaris, Learning the solution operator of parametric partial differential equations with physics-informed DeepONets, Science Advances, 7 (2021), eabi8605. doi: 10.1126/sciadv.abi8605.
[74]	S. Wang, H. Wang and P. Perdikaris, Improved architectures and training algorithms for deep operator networks, Journal of Scientific Computing, 92 (2022), 35. doi: 10.1007/s10915-022-01881-0.
[75]	S. Wang, X. Yu and P. Perdikaris, When and why PINNs fail to train: A neural tangent kernel perspective, Journal of Computational Physics, 449 (2022), 110768. doi: 10.1016/j.jcp.2021.110768.
[76]	Y. Wang and C.-Y. Lai, Multi-stage neural networks: Function approximator of machine precision, arXiv Preprint arXiv: 2307.08934.
[77]	Z. Wang, T. Hong, H. Li and M. A. Piette, Predicting city-scale daily electricity consumption using data-driven models, Advances in Applied Energy, 2 (2021), 100025. doi: 10.1016/j.adapen.2021.100025.
[78]	G. Wen, Z. Li, K. Azizzadenesheli, A. Anandkumar and S. M. Benson, U-FNO—an enhanced Fourier neural operator-based deep-learning model for multiphase flow, Advances in Water Resources, 163 (2022), 104180. doi: 10.1016/j.advwatres.2022.104180.
[79]	C. L. Wight and J. Zhao, Solving Allen-Cahn and Cahn-Hilliard equations using the adaptive physics informed neural networks, arXiv Preprint arXiv: 2007.04542.
[80]	C. Wu, M. Zhu, Q. Tan, Y. Kartha and L. Lu, A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks, Computer Methods in Applied Mechanics and Engineering, 403 (2023), 115671. doi: 10.1016/j.cma.2022.115671.
[81]	Z. Xiang, W. Peng, X. Liu and W. Yao, Self-adaptive loss balanced physics-informed neural networks, Neurocomputing, 496 (2022), 11-34. doi: 10.1016/j.neucom.2022.05.015.
[82]	W. Xu, Y. Lu and L. Wang, Transfer learning enhanced DeepONet for long-time prediction of evolution equations, Proceedings of the AAAI Conference on Artificial Intelligence, 37 (2023), 10629-10636. doi: 10.1609/aaai.v37i9.26262.
[83]	M. Yin, E. Zhang, Y. Yu and G. E. Karniadakis, Interfacing finite elements with deep neural operators for fast multiscale modeling of mechanics problems, Computer Methods in Applied Mechanics and Engineering, 402 (2022), 115027. doi: 10.1016/j.cma.2022.115027.