
-
Previous Article
Comparative study of macroscopic traffic flow models at road junctions
- NHM Home
- This Issue
-
Next Article
A new mixed finite element method for the n-dimensional Boussinesq problem with temperature-dependent viscosity
Deep neural network approach to forward-inverse problems
1. | Department of Mathematics, Pohang University of Science and Technology, South Korea |
2. | Department of Mathematics and Statistics, California State University Long Beach, US |
In this paper, we construct approximated solutions of Differential Equations (DEs) using the Deep Neural Network (DNN). Furthermore, we present an architecture that includes the process of finding model parameters through experimental data, the inverse problem. That is, we provide a unified framework of DNN architecture that approximates an analytic solution and its model parameters simultaneously. The architecture consists of a feed forward DNN with non-linear activation functions depending on DEs, automatic differentiation [
References:
[1] |
W. Arloff, K. R. B. Schmitt and L. J. Venstrom,
A parameter estimation method for stiff ordinary differential equations using particle swarm optimisation, Int. J. Comput. Sci. Math., 9 (2018), 419-432.
doi: 10.1504/IJCSM.2018.095506. |
[2] |
A. G. Baydin, B. A. Pearlmutter, A. A. Radul and J. M. Siskind, Automatic differentiation in machine learning: A survey, J. Mach. Learn. Res., 18 (2017), 43pp. |
[3] |
J. Berg and K. Nystr{ö}m, Neural network augmented inverse problems for PDEs, preprint, arXiv: 1712.09685. |
[4] |
J. Berg and K. Nystr{ö}m,
A unified deep artificial neural network approach to partial differential equations in complex geometries, Neurocomputing, 317 (2018), 28-41.
doi: 10.1016/j.neucom.2018.06.056. |
[5] |
G. Chavet, Nonlinear Least Squares for Inverse Problems. Theoretical Foundations and Step-By-Step Guide for Applications, Scientific Computation, Springer, New York, 2009.
doi: 10.1007/978-90-481-2785-6. |
[6] |
N. E. Cotter,
The Stone-Weierstrass theorem and its application to neural networks, IEEE Trans. Neural Networks, 1 (1990), 290-295.
doi: 10.1109/72.80265. |
[7] |
R. Courant, K. Friedrichs and H. Lewy,
On the partial difference equations of mathematical physics, IBM J. Res. Develop., 11 (1967), 215-234.
doi: 10.1147/rd.112.0215. |
[8] |
G. Cybenko,
Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems, 2 (1989), 303-314.
doi: 10.1007/BF02551274. |
[9] |
L. C. Evans, Partial Differential Equations, Graduate Studies in Mathematics, 19, American Mathematical Society, Providence, RI, 2010.
doi: 10.1090/gsm/019. |
[10] |
G. E. Fasshauer,
Solving partial differential equations by collocation with radial basis functions, Proceedings of Chamonix, 1997 (1996), 1-8.
|
[11] |
K. Hornik, M. Stinchcombe and H. White,
Multilayer feedforward networks are universal approximators, Neural Networks, 2 (1989), 359-366.
doi: 10.1016/0893-6080(89)90020-8. |
[12] |
D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, preprint, arXiv: 1412.6980. |
[13] |
I. E. Lagaris, A. Likas and D. I. Fotiadis,
Artificial neural networks for solving ordinary and partial differential equations, IEEE Trans. Neural Networks, 9 (1998), 987-1000.
doi: 10.1109/72.712178. |
[14] |
I. E. Lagaris, A. C. Likas and D. G. Papageorgiou,
Neural-network methods for boundary value problems with irregular boundaries, IEEE Trans. Neural Networks, 11 (2000), 1041-1049.
doi: 10.1109/72.870037. |
[15] |
K. Levenberg,
A method for the solution of certain non-linear problems in least squares, Quart. Appl. Math., 2 (1944), 164-168.
doi: 10.1090/qam/10666. |
[16] |
L. Jianyu, L. Siwei, Q. Yingjian and H. Yaping,
Numerical solution of elliptic partial differential equation using radial basis function neural networks, Neural Networks, 16 (2003), 729-734.
doi: 10.1016/S0893-6080(03)00083-2. |
[17] |
J. Li and X. Li, Particle swarm optimization iterative identification algorithm and gradient iterative identification algorithm for Wiener systems with colored noise, Complexity, 2018 (2018), 8pp.
doi: 10.1155/2018/7353171. |
[18] |
X. Li,
Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer, Neurocomputing, 12 (1996), 327-343.
doi: 10.1016/0925-2312(95)00070-4. |
[19] |
D. W. Marquardt,
An algorithm for least-squares estimation of nonlinear parameters, J. Soc. Indust. Appl. Math., 11 (1963), 431-441.
doi: 10.1137/0111030. |
[20] |
W. S. McCulloch and W. Pitts,
A logical calculus of the ideas immanent in nervous activity, Bull. Math. Biophys., 5 (1943), 115-133.
doi: 10.1007/BF02478259. |
[21] |
A. Paszke, et al., Automatic differentiation in PyTorch, Computer Science, (2017). |
[22] |
M. Raissi, P. Perdikaris and G. E. Karniadakis,
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378 (2019), 686-707.
doi: 10.1016/j.jcp.2018.10.045. |
[23] |
S. J. Reddi, S. Kale and S. Kumar, On the convergence of ADAM and beyond, preprint, arXiv: 1904.09237. |
[24] |
S. A. Sarra,
Adaptive radial basis function methods for time dependent partial differential equations, Appl. Numer. Math., 54 (2005), 79-94.
doi: 10.1016/j.apnum.2004.07.004. |
[25] |
P. Tsilifis, I. Bilionis, I. Katsounaros and N. Zabaras, Computationally efficient variational approximations for Bayesian inverse problems, J. Verif. Valid. Uncert., 1 (2016), 13pp.
doi: 10.1115/1.4034102. |
[26] |
F. Yaman, V. G. Yakhno and R. Potthast, A survey on inverse problems for applied sciences, Math. Probl. Eng., 2013 (2013), 19pp.
doi: 10.1155/2013/976837. |
show all references
References:
[1] |
W. Arloff, K. R. B. Schmitt and L. J. Venstrom,
A parameter estimation method for stiff ordinary differential equations using particle swarm optimisation, Int. J. Comput. Sci. Math., 9 (2018), 419-432.
doi: 10.1504/IJCSM.2018.095506. |
[2] |
A. G. Baydin, B. A. Pearlmutter, A. A. Radul and J. M. Siskind, Automatic differentiation in machine learning: A survey, J. Mach. Learn. Res., 18 (2017), 43pp. |
[3] |
J. Berg and K. Nystr{ö}m, Neural network augmented inverse problems for PDEs, preprint, arXiv: 1712.09685. |
[4] |
J. Berg and K. Nystr{ö}m,
A unified deep artificial neural network approach to partial differential equations in complex geometries, Neurocomputing, 317 (2018), 28-41.
doi: 10.1016/j.neucom.2018.06.056. |
[5] |
G. Chavet, Nonlinear Least Squares for Inverse Problems. Theoretical Foundations and Step-By-Step Guide for Applications, Scientific Computation, Springer, New York, 2009.
doi: 10.1007/978-90-481-2785-6. |
[6] |
N. E. Cotter,
The Stone-Weierstrass theorem and its application to neural networks, IEEE Trans. Neural Networks, 1 (1990), 290-295.
doi: 10.1109/72.80265. |
[7] |
R. Courant, K. Friedrichs and H. Lewy,
On the partial difference equations of mathematical physics, IBM J. Res. Develop., 11 (1967), 215-234.
doi: 10.1147/rd.112.0215. |
[8] |
G. Cybenko,
Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems, 2 (1989), 303-314.
doi: 10.1007/BF02551274. |
[9] |
L. C. Evans, Partial Differential Equations, Graduate Studies in Mathematics, 19, American Mathematical Society, Providence, RI, 2010.
doi: 10.1090/gsm/019. |
[10] |
G. E. Fasshauer,
Solving partial differential equations by collocation with radial basis functions, Proceedings of Chamonix, 1997 (1996), 1-8.
|
[11] |
K. Hornik, M. Stinchcombe and H. White,
Multilayer feedforward networks are universal approximators, Neural Networks, 2 (1989), 359-366.
doi: 10.1016/0893-6080(89)90020-8. |
[12] |
D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, preprint, arXiv: 1412.6980. |
[13] |
I. E. Lagaris, A. Likas and D. I. Fotiadis,
Artificial neural networks for solving ordinary and partial differential equations, IEEE Trans. Neural Networks, 9 (1998), 987-1000.
doi: 10.1109/72.712178. |
[14] |
I. E. Lagaris, A. C. Likas and D. G. Papageorgiou,
Neural-network methods for boundary value problems with irregular boundaries, IEEE Trans. Neural Networks, 11 (2000), 1041-1049.
doi: 10.1109/72.870037. |
[15] |
K. Levenberg,
A method for the solution of certain non-linear problems in least squares, Quart. Appl. Math., 2 (1944), 164-168.
doi: 10.1090/qam/10666. |
[16] |
L. Jianyu, L. Siwei, Q. Yingjian and H. Yaping,
Numerical solution of elliptic partial differential equation using radial basis function neural networks, Neural Networks, 16 (2003), 729-734.
doi: 10.1016/S0893-6080(03)00083-2. |
[17] |
J. Li and X. Li, Particle swarm optimization iterative identification algorithm and gradient iterative identification algorithm for Wiener systems with colored noise, Complexity, 2018 (2018), 8pp.
doi: 10.1155/2018/7353171. |
[18] |
X. Li,
Simultaneous approximations of multivariate functions and their derivatives by neural networks with one hidden layer, Neurocomputing, 12 (1996), 327-343.
doi: 10.1016/0925-2312(95)00070-4. |
[19] |
D. W. Marquardt,
An algorithm for least-squares estimation of nonlinear parameters, J. Soc. Indust. Appl. Math., 11 (1963), 431-441.
doi: 10.1137/0111030. |
[20] |
W. S. McCulloch and W. Pitts,
A logical calculus of the ideas immanent in nervous activity, Bull. Math. Biophys., 5 (1943), 115-133.
doi: 10.1007/BF02478259. |
[21] |
A. Paszke, et al., Automatic differentiation in PyTorch, Computer Science, (2017). |
[22] |
M. Raissi, P. Perdikaris and G. E. Karniadakis,
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 378 (2019), 686-707.
doi: 10.1016/j.jcp.2018.10.045. |
[23] |
S. J. Reddi, S. Kale and S. Kumar, On the convergence of ADAM and beyond, preprint, arXiv: 1904.09237. |
[24] |
S. A. Sarra,
Adaptive radial basis function methods for time dependent partial differential equations, Appl. Numer. Math., 54 (2005), 79-94.
doi: 10.1016/j.apnum.2004.07.004. |
[25] |
P. Tsilifis, I. Bilionis, I. Katsounaros and N. Zabaras, Computationally efficient variational approximations for Bayesian inverse problems, J. Verif. Valid. Uncert., 1 (2016), 13pp.
doi: 10.1115/1.4034102. |
[26] |
F. Yaman, V. G. Yakhno and R. Potthast, A survey on inverse problems for applied sciences, Math. Probl. Eng., 2013 (2013), 19pp.
doi: 10.1155/2013/976837. |







Algorithm 1: Training |
1: procedure train(number of epochs) 2: Initialize the nerural network. 3: For number of epochs do 4: sample 5: sample 6: sample 7: sample k observation points 8: Find the true value 9: Update the neural network by descending its stochastic gradient : 10: end for 11: end procedure |
Algorithm 1: Training |
1: procedure train(number of epochs) 2: Initialize the nerural network. 3: For number of epochs do 4: sample 5: sample 6: sample 7: sample k observation points 8: Find the true value 9: Update the neural network by descending its stochastic gradient : 10: end for 11: end procedure |
Data Generation | |||
Grid Range | Number of Grid Points | Number of Observations | |
1D Transport | 17 | ||
2D Heat | 13 | ||
2D Wave | 61 | ||
Lotka-Volterra | 20,000 | 40 |
Data Generation | |||
Grid Range | Number of Grid Points | Number of Observations | |
1D Transport | 17 | ||
2D Heat | 13 | ||
2D Wave | 61 | ||
Lotka-Volterra | 20,000 | 40 |
Neural Network Architecture | |||
Fully Connected Layers | Activation Functions | Learning Rate | |
1D Transport | 2(input)-128-256-128-1(output) | ReLU | |
2D Heat | 3(input)-128-128-1(output) | Sin, Sigmoid | |
2D Wave | 3(input)-128-256-128-1(output) | Sin, Tanh | |
Lotka-Volterra | 1(input)-64-64-2(output) | Sin |
Neural Network Architecture | |||
Fully Connected Layers | Activation Functions | Learning Rate | |
1D Transport | 2(input)-128-256-128-1(output) | ReLU | |
2D Heat | 3(input)-128-128-1(output) | Sin, Sigmoid | |
2D Wave | 3(input)-128-256-128-1(output) | Sin, Tanh | |
Lotka-Volterra | 1(input)-64-64-2(output) | Sin |
[1] |
Lars Grüne. Computing Lyapunov functions using deep neural networks. Journal of Computational Dynamics, 2021, 8 (2) : 131-152. doi: 10.3934/jcd.2021006 |
[2] |
Zheng Chen, Liu Liu, Lin Mu. Solving the linear transport equation by a deep neural network approach. Discrete and Continuous Dynamical Systems - S, 2022, 15 (4) : 669-686. doi: 10.3934/dcdss.2021070 |
[3] |
Graciela Canziani, Rosana Ferrati, Claudia Marinelli, Federico Dukatz. Artificial neural networks and remote sensing in the analysis of the highly variable Pampean shallow lakes. Mathematical Biosciences & Engineering, 2008, 5 (4) : 691-711. doi: 10.3934/mbe.2008.5.691 |
[4] |
Ruhua Wang, Senjian An, Wanquan Liu, Ling Li. Fixed-point algorithms for inverse of residual rectifier neural networks. Mathematical Foundations of Computing, 2021, 4 (1) : 31-44. doi: 10.3934/mfc.2020024 |
[5] |
Daniel G. Alfaro Vigo, Amaury C. Álvarez, Grigori Chapiro, Galina C. García, Carlos G. Moreira. Solving the inverse problem for an ordinary differential equation using conjugation. Journal of Computational Dynamics, 2020, 7 (2) : 183-208. doi: 10.3934/jcd.2020008 |
[6] |
Seiyed Hadi Abtahi, Hamidreza Rahimi, Maryam Mosleh. Solving fuzzy volterra-fredholm integral equation by fuzzy artificial neural network. Mathematical Foundations of Computing, 2021, 4 (3) : 209-219. doi: 10.3934/mfc.2021013 |
[7] |
H. N. Mhaskar, T. Poggio. Function approximation by deep networks. Communications on Pure and Applied Analysis, 2020, 19 (8) : 4085-4095. doi: 10.3934/cpaa.2020181 |
[8] |
Bingyan Liu, Xiongbing Ye, Xianzhou Dong, Lei Ni. Branching improved Deep Q Networks for solving pursuit-evasion strategy solution of spacecraft. Journal of Industrial and Management Optimization, 2022, 18 (2) : 1223-1245. doi: 10.3934/jimo.2021016 |
[9] |
Seonho Park, Maciej Rysz, Kaitlin L. Fair, Panos M. Pardalos. Synthetic-Aperture Radar image based positioning in GPS-denied environments using Deep Cosine Similarity Neural Networks. Inverse Problems and Imaging, 2021, 15 (4) : 763-785. doi: 10.3934/ipi.2021013 |
[10] |
Weishi Yin, Jiawei Ge, Pinchao Meng, Fuheng Qu. A neural network method for the inverse scattering problem of impenetrable cavities. Electronic Research Archive, 2020, 28 (2) : 1123-1142. doi: 10.3934/era.2020062 |
[11] |
Hermann Gross, Sebastian Heidenreich, Mark-Alexander Henn, Markus Bär, Andreas Rathsfeld. Modeling aspects to improve the solution of the inverse problem in scatterometry. Discrete and Continuous Dynamical Systems - S, 2015, 8 (3) : 497-519. doi: 10.3934/dcdss.2015.8.497 |
[12] |
Christopher Oballe, David Boothe, Piotr J. Franaszczuk, Vasileios Maroulas. ToFU: Topology functional units for deep learning. Foundations of Data Science, 2021 doi: 10.3934/fods.2021021 |
[13] |
Richard Archibald, Feng Bao, Yanzhao Cao, He Zhang. A backward SDE method for uncertainty quantification in deep learning. Discrete and Continuous Dynamical Systems - S, 2022 doi: 10.3934/dcdss.2022062 |
[14] |
Ziju Shen, Yufei Wang, Dufan Wu, Xu Yang, Bin Dong. Learning to scan: A deep reinforcement learning approach for personalized scanning in CT imaging. Inverse Problems and Imaging, 2022, 16 (1) : 179-195. doi: 10.3934/ipi.2021045 |
[15] |
Nguyen Huy Tuan, Mokhtar Kirane, Long Dinh Le, Van Thinh Nguyen. On an inverse problem for fractional evolution equation. Evolution Equations and Control Theory, 2017, 6 (1) : 111-134. doi: 10.3934/eect.2017007 |
[16] |
Ying Sue Huang. Resynchronization of delayed neural networks. Discrete and Continuous Dynamical Systems, 2001, 7 (2) : 397-401. doi: 10.3934/dcds.2001.7.397 |
[17] |
Zengyun Wang, Jinde Cao, Zuowei Cai, Lihong Huang. Finite-time stability of impulsive differential inclusion: Applications to discontinuous impulsive neural networks. Discrete and Continuous Dynamical Systems - B, 2021, 26 (5) : 2677-2692. doi: 10.3934/dcdsb.2020200 |
[18] |
Zuowei Cai, Jianhua Huang, Lihong Huang. Generalized Lyapunov-Razumikhin method for retarded differential inclusions: Applications to discontinuous neural networks. Discrete and Continuous Dynamical Systems - B, 2017, 22 (9) : 3591-3614. doi: 10.3934/dcdsb.2017181 |
[19] |
Yegana Ashrafova, Kamil Aida-Zade. Numerical solution to an inverse problem on a determination of places and capacities of sources in the hyperbolic systems. Journal of Industrial and Management Optimization, 2020, 16 (6) : 3011-3033. doi: 10.3934/jimo.2019091 |
[20] |
Li-Fang Dai, Mao-Lin Liang, Wei-Yuan Ma. Optimization problems on the rank of the solution to left and right inverse eigenvalue problem. Journal of Industrial and Management Optimization, 2015, 11 (1) : 171-183. doi: 10.3934/jimo.2015.11.171 |
2021 Impact Factor: 1.41
Tools
Metrics
Other articles
by authors
[Back to Top]