2014, 4(3): 193-207. doi: 10.3934/naco.2014.4.193

Convergence analysis of the weighted state space search algorithm for recurrent neural networks

1. 

Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong

2. 

Department of Mathematics, Cleveland State University, Cleveland, OH 44115

Received  April 2013 Revised  July 2014 Published  September 2014

Recurrent neural networks (RNNs) have emerged as a promising tool in modeling nonlinear dynamical systems. The convergence is one of the most important issues of concern among the dynamical properties for the RNNs in practical applications. The reason is that the viability of many applications of RNNs depends on their convergence properties. We study in this paper the convergence properties of the weighted state space search algorithm (WSSSA) -- a derivative-free and non-random learning algorithm which searches the neighborhood of the target trajectory in the state space instead of the parameter space. Because there is no computation of partial derivatives involved, the WSSSA has a couple of salient features such as simple, fast and cost effective. In this study we provide a necessary and sufficient condition that required for the convergence of the WSSSA. Restrictions are offered that may help assure convergence of the of the WSSSA to the desired solution. The asymptotic rate of convergence is also analyzed. Our study gives insights into the problem and provides useful information for the actual design of the RNNs. A numerical example is given to support the theoretical analysis and to demonstrate that it is applicable to many applications.
Citation: Leong-Kwan Li, Sally Shao. Convergence analysis of the weighted state space search algorithm for recurrent neural networks. Numerical Algebra, Control & Optimization, 2014, 4 (3) : 193-207. doi: 10.3934/naco.2014.4.193
References:
[1]

A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence,, IEEE Transcations on Neural Networks, 11 (2000), 697.   Google Scholar

[2]

R. A. Conn, K. Scheinberg and N. L. Vicente, Introduction to Derivative-free Optimization,, SIAM, (2009).  doi: 10.1137/1.9780898718768.  Google Scholar

[3]

L. Jin, N. Nikifork and M. M. Gupta, Absolute stability conditions for discrete-time neural networks,, IEEE Tranc. Neural Networks, 5 (1994), 954.   Google Scholar

[4]

L. K. Li, Learning sunspot series dynamics by recurrent neural networks,, Advances in Data Mining and Modeling (eds. W. K. Ching and K. P. Ng), (2003), 107.   Google Scholar

[5]

L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations,, International Journal of Pure and Applied Mathematics, 39 (2007), 545.   Google Scholar

[6]

L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates,, Applied Mathematical Sciences, 2 (2008), 1705.   Google Scholar

[7]

L. K. Li, Sally Shao and K. F. Cedric Yiu, Nonlinear dynamical system modeling via recurrent neural networks and a weighted wtate space search algorithm,, Journal of Industrial and Management Optimization, 7 (2011), 385.  doi: 10.3934/jimo.2011.7.385.  Google Scholar

[8]

Q. Liu and J. Wang, Finite-time convergent recurrent neural network with a hard-liming activation function for constrained optimization with piecewise-linear objective functions,, IEEE Transactions on Neural Networks, 22 (2011), 601.   Google Scholar

[9]

D. T. Mirikitani and N. Nikolaev, Recursive Bayesian recurrent neural networks for time-series modeling,, IEEE Transactions on Neural Networks, 2 (2010), 262.   Google Scholar

[10]

Q. Song, On the weight convergence of Elman networks,, IEEE Transactions on Neural Networks, 21 (2010), 463.   Google Scholar

[11]

X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks,, Journal of Computer and System Sciences, 45 (1992), 1.  doi: 10.1016/0022-0000(92)90038-K.  Google Scholar

[12]

X. Wang and H. Huang, Convergence Study in Extended Kalman Filter-based Training of Recurrent Neural Networks,, IEEE Trans. on Neural Networks, 22 (2011), 588.   Google Scholar

[13]

L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service,, Journal of Industrial and Management Optimization, 1 (2005), 389.  doi: 10.3934/jimo.2005.1.337.  Google Scholar

[14]

F. Xu and Z. Yi, Convergence Analysis of a class of simplified background netural networks with subnetworks,, Neurocomputing, 74 (2011), 3877.   Google Scholar

[15]

J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex,, Neural Computation, 34 (2000), 79.   Google Scholar

[16]

K. F. C. Yiu, S. Wang, K. L. Teo and A. C. Tsoi, Nonlinear System modeling via knot-optimizing B-splines networks,, IEEE Transactions on Neural Networks, 12 (2001), 1013.   Google Scholar

[17]

Y. Zhang and K. K. Tan, Convergence Analysis of Recurrent Neural Networks., Kluwer, (2004).  doi: 10.1007/978-1-4757-3819-3.  Google Scholar

[18]

L. Zhang and Z. Yi., Selectable and unselectable sets of neurons in recurrent neural networks with saturated piecewise linear transfer function,, IEEE Transactions on Neural Networks, 22 (2011), 1021.   Google Scholar

show all references

References:
[1]

A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence,, IEEE Transcations on Neural Networks, 11 (2000), 697.   Google Scholar

[2]

R. A. Conn, K. Scheinberg and N. L. Vicente, Introduction to Derivative-free Optimization,, SIAM, (2009).  doi: 10.1137/1.9780898718768.  Google Scholar

[3]

L. Jin, N. Nikifork and M. M. Gupta, Absolute stability conditions for discrete-time neural networks,, IEEE Tranc. Neural Networks, 5 (1994), 954.   Google Scholar

[4]

L. K. Li, Learning sunspot series dynamics by recurrent neural networks,, Advances in Data Mining and Modeling (eds. W. K. Ching and K. P. Ng), (2003), 107.   Google Scholar

[5]

L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations,, International Journal of Pure and Applied Mathematics, 39 (2007), 545.   Google Scholar

[6]

L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates,, Applied Mathematical Sciences, 2 (2008), 1705.   Google Scholar

[7]

L. K. Li, Sally Shao and K. F. Cedric Yiu, Nonlinear dynamical system modeling via recurrent neural networks and a weighted wtate space search algorithm,, Journal of Industrial and Management Optimization, 7 (2011), 385.  doi: 10.3934/jimo.2011.7.385.  Google Scholar

[8]

Q. Liu and J. Wang, Finite-time convergent recurrent neural network with a hard-liming activation function for constrained optimization with piecewise-linear objective functions,, IEEE Transactions on Neural Networks, 22 (2011), 601.   Google Scholar

[9]

D. T. Mirikitani and N. Nikolaev, Recursive Bayesian recurrent neural networks for time-series modeling,, IEEE Transactions on Neural Networks, 2 (2010), 262.   Google Scholar

[10]

Q. Song, On the weight convergence of Elman networks,, IEEE Transactions on Neural Networks, 21 (2010), 463.   Google Scholar

[11]

X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks,, Journal of Computer and System Sciences, 45 (1992), 1.  doi: 10.1016/0022-0000(92)90038-K.  Google Scholar

[12]

X. Wang and H. Huang, Convergence Study in Extended Kalman Filter-based Training of Recurrent Neural Networks,, IEEE Trans. on Neural Networks, 22 (2011), 588.   Google Scholar

[13]

L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service,, Journal of Industrial and Management Optimization, 1 (2005), 389.  doi: 10.3934/jimo.2005.1.337.  Google Scholar

[14]

F. Xu and Z. Yi, Convergence Analysis of a class of simplified background netural networks with subnetworks,, Neurocomputing, 74 (2011), 3877.   Google Scholar

[15]

J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex,, Neural Computation, 34 (2000), 79.   Google Scholar

[16]

K. F. C. Yiu, S. Wang, K. L. Teo and A. C. Tsoi, Nonlinear System modeling via knot-optimizing B-splines networks,, IEEE Transactions on Neural Networks, 12 (2001), 1013.   Google Scholar

[17]

Y. Zhang and K. K. Tan, Convergence Analysis of Recurrent Neural Networks., Kluwer, (2004).  doi: 10.1007/978-1-4757-3819-3.  Google Scholar

[18]

L. Zhang and Z. Yi., Selectable and unselectable sets of neurons in recurrent neural networks with saturated piecewise linear transfer function,, IEEE Transactions on Neural Networks, 22 (2011), 1021.   Google Scholar

[1]

George W. Patrick. The geometry of convergence in numerical analysis. Journal of Computational Dynamics, 2021, 8 (1) : 33-58. doi: 10.3934/jcd.2021003

[2]

Matúš Tibenský, Angela Handlovičová. Convergence analysis of the discrete duality finite volume scheme for the regularised Heston model. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1181-1195. doi: 10.3934/dcdss.2020226

[3]

Ziang Long, Penghang Yin, Jack Xin. Global convergence and geometric characterization of slow to fast weight evolution in neural network training for classifying linearly non-separable data. Inverse Problems & Imaging, 2021, 15 (1) : 41-62. doi: 10.3934/ipi.2020077

[4]

Lars Grüne. Computing Lyapunov functions using deep neural networks. Journal of Computational Dynamics, 2020  doi: 10.3934/jcd.2021006

[5]

Leslaw Skrzypek, Yuncheng You. Feedback synchronization of FHN cellular neural networks. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2021001

[6]

Wei Ouyang, Li Li. Hölder strong metric subregularity and its applications to convergence analysis of inexact Newton methods. Journal of Industrial & Management Optimization, 2021, 17 (1) : 169-184. doi: 10.3934/jimo.2019105

[7]

Matania Ben–Artzi, Joseph Falcovitz, Jiequan Li. The convergence of the GRP scheme. Discrete & Continuous Dynamical Systems - A, 2009, 23 (1&2) : 1-27. doi: 10.3934/dcds.2009.23.1

[8]

Thierry Horsin, Mohamed Ali Jendoubi. On the convergence to equilibria of a sequence defined by an implicit scheme. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020465

[9]

Philipp Harms. Strong convergence rates for markovian representations of fractional processes. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020367

[10]

Alberto Bressan, Carlotta Donadello. On the convergence of viscous approximations after shock interactions. Discrete & Continuous Dynamical Systems - A, 2009, 23 (1&2) : 29-48. doi: 10.3934/dcds.2009.23.29

[11]

Editorial Office. Retraction: Honggang Yu, An efficient face recognition algorithm using the improved convolutional neural network. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 901-901. doi: 10.3934/dcdss.2019060

[12]

Cheng Peng, Zhaohui Tang, Weihua Gui, Qing Chen, Jing He. A bidirectional weighted boundary distance algorithm for time series similarity computation based on optimized sliding window size. Journal of Industrial & Management Optimization, 2021, 17 (1) : 205-220. doi: 10.3934/jimo.2019107

[13]

Zi Xu, Siwen Wang, Jinjin Huang. An efficient low complexity algorithm for box-constrained weighted maximin dispersion problem. Journal of Industrial & Management Optimization, 2021, 17 (2) : 971-979. doi: 10.3934/jimo.2020007

[14]

Parikshit Upadhyaya, Elias Jarlebring, Emanuel H. Rubensson. A density matrix approach to the convergence of the self-consistent field iteration. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 99-115. doi: 10.3934/naco.2020018

[15]

Gang Luo, Qingzhi Yang. The point-wise convergence of shifted symmetric higher order power method. Journal of Industrial & Management Optimization, 2021, 17 (1) : 357-368. doi: 10.3934/jimo.2019115

[16]

Toshiko Ogiwara, Danielle Hilhorst, Hiroshi Matano. Convergence and structure theorems for order-preserving dynamical systems with mass conservation. Discrete & Continuous Dynamical Systems - A, 2020, 40 (6) : 3883-3907. doi: 10.3934/dcds.2020129

[17]

Xiuli Xu, Xueke Pu. Optimal convergence rates of the magnetohydrodynamic model for quantum plasmas with potential force. Discrete & Continuous Dynamical Systems - B, 2021, 26 (2) : 987-1010. doi: 10.3934/dcdsb.2020150

[18]

Takiko Sasaki. Convergence of a blow-up curve for a semilinear wave equation. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1133-1143. doi: 10.3934/dcdss.2020388

[19]

Adrian Viorel, Cristian D. Alecsa, Titus O. Pinţa. Asymptotic analysis of a structure-preserving integrator for damped Hamiltonian systems. Discrete & Continuous Dynamical Systems - A, 2020  doi: 10.3934/dcds.2020407

[20]

Thomas Frenzel, Matthias Liero. Effective diffusion in thin structures via generalized gradient systems and EDP-convergence. Discrete & Continuous Dynamical Systems - S, 2021, 14 (1) : 395-425. doi: 10.3934/dcdss.2020345

 Impact Factor: 

Metrics

  • PDF downloads (68)
  • HTML views (0)
  • Cited by (1)

Other articles
by authors

[Back to Top]