A novel recurrent neural network of gated unit based on Euler's method and application

Pinchao Meng; Lin Wang; Weishi Yin; Linhua Zhou

doi:10.3934/cac.2023007

Article Contents

2023, Volume 1, Issue 2: 116-134. Doi: 10.3934/cac.2023007

This issue Previous Article New results for variational regularization with oversmoothing penalty term in Banach spaces Next Article Denoising convolution algorithms and applications to SAR signal processing

A novel recurrent neural network of gated unit based on Euler's method and application

School of Mathematics and Statistics, Changchun University of Science and Technology, Changchun, China

^*Corresponding author: Pinchao Meng
^*Corresponding author: Pinchao Meng

Received: January 2023

Revised: March 2023

Early access: March 2023

Published: June 2023

This research is supported by the Jilin Provincial Science Foundation and Technology Program. [Project No. 20220101040JC].

Abstract / Introduction Full Text(HTML) Figure(12) / Table(1) Related Papers Cited by

Abstract

The main focus of this paper is to interpret the neural networks as the discretizations of differential equations, which has more benefits for analyzing the intrinsic mechanisms of neural networks. Under this theoretical framework, we propose a conditionally stable network unit called the GUEM, which is based on the Euler's method for ordinary differential equations and the gated thought in recurrent neural networks. Moreover, we build a sequence-to-sequence recurrent neural network based on the GUEM and fully connected layers, which does not amplify perturbations caused by noises of the input features. Finally, the numerical experiments of inverse scattering problems demonstrate the effectiveness and efficiency of our network.

Keywords:

Mathematics Subject Classification: 34D20, 65L07, 35R30, 68T07.

Citation:

FullText(HTML)

Figure 1. The architecture diagram of GUEM with $ \mu = 1 $

Download: Full-size image PowerPoint slide

Figure 2. Visualization of the dynamics with weight matrices $ W_{\tilde{c}h}^{1} $, $ W_{\tilde{c}h}^{2} $ and $ W_{\tilde{c}h}^{3} $ respectively, where the stars represent the four initial hidden states in two dimensions

Download: Full-size image PowerPoint slide

Figure 3. Visualization of the dynamics with weight matrices $ W_{\tilde{c}h}^{4} $, $ W_{\tilde{c}h}^{5} $ and $ W_{\tilde{c}h}^{6} $ respectively, where the stars represent the four initial hidden states in three dimensions

Download: Full-size image PowerPoint slide

Figure 4. The architecture diagram of the recurrent neural network

Download: Full-size image PowerPoint slide

Figure 5. Recoveries of a peanut-shape obstacle and a kite-shape obstacle by the GUEM-RNN with the learning rate $ \eta = 0.001, 0.0001, 0.00001 $, respectively

Download: Full-size image PowerPoint slide

Figure 6. Reconstruction losses of peanut-shape obstacles and kite-shape obstacles by the GUEM-RNN with the learning rate $ \eta = 0.001, 0.0001, 0.00001 $, respectively

Download: Full-size image PowerPoint slide

Figure 7. Recoveries of two forms of peanut-shape obstacles by the GUEM-RNN with $ n' = 15, 25, 35 $, respectively

Download: Full-size image PowerPoint slide

Figure 8. Recoveries of two forms of kite-shape obstacles by the GUEM-RNN with $ n' = 15, 25, 35 $, respectively

Download: Full-size image PowerPoint slide

Figure 9. Reconstruction losses of peanut-shape obstacles and kite-shape obstacles by the GUEM-RNN with $ n' = 15, 25, 35 $, respectively

Download: Full-size image PowerPoint slide

Figure 10. Recoveries of a peanut-shape obstacle and a kite-shape obstacle by the GUEM-RNN with $ 5\%, 10\%, 20\% $ noise levels, respectively

Download: Full-size image PowerPoint slide

Figure 11. Reconstruction losses of peanut-shape obstacles and kite-shape obstacles by the GUEM-RNN with $ 5\%, 10\%, 20\% $ noise levels, respectively

Download: Full-size image PowerPoint slide

Figure 12. Recoveries of a peanut-shape obstacle by the GUEM-RNN with the observation aperture $ \left[0, 2\pi\right] $, $ \left[-\frac{\pi}{2}, \frac{\pi}{2}\right] $, $ \left[-\frac{\pi}{4}, \frac{\pi}{4}\right] $, respectively. Figure 12(d) shows the reconstruction losses with different observation apertures by the GUEM-RNN

Download: Full-size image PowerPoint slide

Table 1. The hyper parameters of GUEM-RNN

Parameter	Hidden Neurons	Learning Rate	Weight-Decay	Batch Size	Iteration Epochs
Value	128	$ 0.0001 $	$ 0.00001 $	64	20

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	A. Adler and A. Boyle, Lectrical impedance tomography: Tissue properties to image measures, IEEE Transactions on Biomedical Engineering, 64 (2017), 2494-2504.
[2]	Y. Bengio, Learning deep architectures for AI, Foundations and Trends in Machine Learning, 2 (2009), 1-127.
[3]	X. Cao, H. Diao and H. Liu, et al., On nodal and singular structures of Laplacian eigenfunctions and applications to inverse scattering problems, Journal de Mathematiques Pures et Appliquees, 143 (2020), 116-161. doi: 10.1016/j.matpur.2020.09.011.
[4]	X. Cao, H. Diao and H. Liu, et al., On novel geometric structures of Laplacian eigenfunctions in $\mathbb{R}^3$ and applications to inverse problems, SIAM Journal on Mathematical Analysis, 53 (2021), 1263-1294. doi: 10.1137/19M1292989.
[5]	B. Chang, M. Chen, E. Haber, et al., AntisymmetricRNN: A dynamical system view on recurrent neural networks, Proceedings of the International Conference on Learning Representations, New Orleans, USA, (2019), 1-15.
[6]	B. Chang, L. Meng, E. Haber, et al., Reversible architectures for arbitrarily deep residual neural networks, Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, USA, (2018), 2811-2818.
[7]	R. Chen, Y. Rubanova, J. Bettencourt, et al., Neural ordinary differential equations, Proceedings of the International Conference on Neural Information Processing Systems, Montreal, Canada, (2018), 6572-6583.
[8]	K. Cho, B. Merri$\ddot{e}$nboer, C. Gulcehre, et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation, Proceedings of the Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, (2014), 1724-1734.
[9]	D. Colton, J. Coyle and P. Monk, Recent developments in inverse acoustic scattering theory, Siam Review, 42 (2000), 369-414. doi: 10.1137/S0036144500367337.
[10]	G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems, 2 (1989), 303-314. doi: 10.1007/BF02551274.
[11]	Y. Deng, J. Li and H. Liu, On identifying magnetized anomalies using geomagnetic monitoring, Archive for Rational Mechanics and Analysis, 231 (2019), 153-187. doi: 10.1007/s00205-018-1276-7.
[12]	Y. Deng, J. Li and H. Liu, On identifying magnetized anomalies using geomagnetic monitoring within a magnetohydrodynamic model, Archive for Rational Mechanics and Analysis, 235 (2020), 691-721. doi: 10.1007/s00205-019-01429-x.
[13]	Y. Deng, H. Liu and W. Tsui, Identifying variations of magnetic anomalies using geomagnetic monitoring, Discrete and Continuous Dynamical Systems, 40 (2020), 6411-6440. doi: 10.3934/dcds.2020285.
[14]	Y. Deng, H. Liu and G. Uhlmann, On an inverse boundary problem arising in brain imaging, Journal of Differential Equations, 267 (2019), 2471-2502. doi: 10.1016/j.jde.2019.03.019.
[15]	W. E, A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, 5 (2017), 1-11. doi: 10.1007/s40304-017-0103-z.
[16]	Y. Gao, H. Liu, X. Wang, et al., On an artificial neural network for inverse scattering problems, Journal of Computational Physics, 448 (2022), 110771, 15 pp. doi: 10.1016/j.jcp.2021.110771.
[17]	Y. Gao and K. Zhang, Machine learning based data retrieval for inverse scattering problems with incomplete data, Journal of Inverse and Ill-Posed Problems, 29 (2021), 249-266. doi: 10.1515/jiip-2019-0101.
[18]	Y. Guo, P. Monk and D. Colton, The linear sampling method for sparse small aperture data, Applicable Analysis, 95 (2016), 1599-1615. doi: 10.1080/00036811.2015.1065317.
[19]	E. Haber, K. Lensink, E. Treister, et al., IMEXnet: A forward stable deep neural network, Proceedings of the International Conference on Machine Learning, California, USA, (2019), 2525-2534.
[20]	E. Haber and L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2018), 014004, 22 pp. doi: 10.1088/1361-6420/aa9a90.
[21]	S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Computation, 9 (1997), 1735-1780.
[22]	J. Jia, B. Wu, J. Peng, et al., Recursive linearization method for inverse medium scattering problems with complex mixture Gaussian error learning, Inverse Probelms, 35 (2019), 075003, 29 pp. doi: 10.1088/1361-6420/ab08f2.
[23]	J. Jia, S. Yue and J. Peng, et al., Infinite-dimensional Bayesian approach for inverse scattering problems of a fractional Helmholtz equation, Journal of Functional Analysis, 275 (2018), 2299-2332. doi: 10.1016/j.jfa.2018.08.002.
[24]	J. Jumper, R. Evans and A. Pritzel, et al., Highly accurate protein structure prediction with AlphaFold, Nature, 596 (2021), 583-589.
[25]	J. Kaipio and E. Somersalo, Statistical and Computational Inverse Problems, $1^{st}$ edition, Applied Mathematical Sciences, 160. Springer-Verlag, New York, 2005.
[26]	S. Kanai, Y. Fujiwara and S. Iwamura, Preventing gradient explosions in gated recurrent units, Proceedings of the International Conference on Neural Information Processing Systems, California, USA, (2017), 435-444.
[27]	Y. Khoo and L. Ying, SwitchNet: A neural network model for forward and inverse scattering problems, SIAM Journal on Scientific Computing, 41 (2019), A3182-A3201. doi: 10.1137/18M1222399.
[28]	J. Li and H. Liu, Recovering a polyhedral obstacle by a few backscattering measurements, Journal of Differential Equations, 259 (2015), 2101-2120. doi: 10.1016/j.jde.2015.03.030.
[29]	J. Li, H. Liu and W. Tsui, et al., An inverse scattering approach for geometric body generation: A machine learning perspective, Mathematics in Engineering, 1 (2019), 800-823. doi: 10.3934/mine.2019.4.800.
[30]	J. Li, H. Liu and Y. Wang, Recovering an electromagnetic obstacle by a few phaseless backscattering measurements, Inverse Problems, 33 (2017), 035011, 20 pp. doi: 10.1088/1361-6420/aa5bf3.
[31]	H. Liu, M. Petrini and L. Rondi, et al., Stable determination of sound-hard polyhedral scatterers by a minimal number of scattering measurements, Journal of Differential Equations, 262 (2017), 1631-1670. doi: 10.1016/j.jde.2016.10.021.
[32]	H. Liu and J. Zou, Uniqueness in an inverse acoustic obstacle scattering problem for both sound-hard and sound-soft polyhedral scatterers, Inverse Problems, 22 (2006), 515-524. doi: 10.1088/0266-5611/22/2/008.
[33]	Z. Long, Y. Lu and B. Dong, PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network, Journal of Computational Physics, 399 (2019), 108925, 17 pp. doi: 10.1016/j.jcp.2019.108925.
[34]	Y. Lu, A. Zhong, Q. Li, et al., Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations, Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, (2018), 3276-3285.
[35]	P. Meng, L. Su and W. Yin, et al., Solving a kind of inverse scattering problem of acoustic waves based on linear sampling method and neural network, Journal of Alexandria Engineering, 59 (2020), 1451-1462.
[36]	P. Rauber, S. Fadel and A. Falcao, et al., Visualizing the hidden activity of artificial neural networks, IEEE Transactions on Visualization and Computer Graphics, 23 (2017), 101-110.
[37]	L. Ruthotto and E. Haber, Deep neural networks motivated by partial differential equations, Journal of Mathematical Imaging and Vision, 62 (2020), 352-364. doi: 10.1007/s10851-019-00903-1.
[38]	D. Silver, A. Huang and C. Maddison, et al., Mastering the game of Go with deep neural networks and tree search, Nature, 529 (2016), 484-489.
[39]	D. Silver, J. Schrittwieser and K. Simonyan, et al., Mastering the game of Go without human knowledge, Nature, 550 (2017), 354-359.
[40]	C. Tallec and Y. Ollivier, Can recurrent neural networks warp time?, Proceedings of the International Conference on Learning Representations, Vancouver, Canada, (2018), 1-13.
[41]	Z. Xu, Y. Zhang and T. Luo, et al., Frequency principle: Fourier analysis sheds light on deep neural networks, Communications in Computational Physics, 28 (2020), 1746-1767. doi: 10.4208/cicp.OA-2020-0085.
[42]	W. Yin, W. Yang and H. Liu, A neural network scheme for recovering scattering obstacles with limited phaseless far-field data, Journal of Computational Physics, 417 (2020), 109594, 18 pp. doi: 10.1016/j.jcp.2020.109594.
[43]	Y. Yin, W. Yin and P. Meng, et al., On a hybrid approach for recovering multiple obstacles, Communications in Computational Physics, 31 (2022), 869-892. doi: 10.4208/cicp.OA-2021-0124.
[44]	Y. Yin, W. Yin and P. Meng, et al., The interior inverse scattering problem for a two-layered cavity using the Bayesian method, Inverse Problems and Imaging, 16 (2022), 673-690. doi: 10.3934/ipi.2021069.
[45]	B. Zhou, A. Khosla, A. Lapedriza, et al., Learning deep features for discriminative localization, IEEE Conference on Computer Vision and Pattern Recognition, (2016), 2921-2929.