Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm

Previous Article
An integrated approach for the operations of distribution and lateral transshipment for seasonal products - A case study in household product industry
JIMO Home
This Issue
Next Article
Multi-objective aggregate production planning decisions using two-phase fuzzy goal programming method

April 2011, 7(2): 385-400. doi: 10.3934/jimo.2011.7.385

Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm

Leong-Kwan Li ^1, , Sally Shao ^2, and K. F. Cedric Yiu ^1,

1.	Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China
2.	Department of Mathematics, Cleveland State University, Cleveland, OH 44115, United States

Received October 2009 Revised January 2011 Published April 2011

Abstract
Related Papers

Given a task of tracking a trajectory, a recurrent neural network may be considered as a black-box nonlinear regression model for tracking unknown dynamic systems. An error function is used to measure the difference between the system outputs and the desired trajectory that formulates a nonlinear least square problem with dynamical constraints. With the dynamical constraints, classical gradient type methods are difficult and time consuming due to the involving of the computation of the partial derivatives along the trajectory. We develop an alternative learning algorithm, namely the weighted state space search algorithm, which searches the neighborhood of the target trajectory in the state space instead of the parameter space. Since there is no computation of partial derivatives involved, our algorithm is simple and fast. We demonstrate our approach by modeling the short-term foreign exchange rates. The empirical results show that the weighted state space search method is very promising and effective in solving least square problems with dynamical constraints. Numerical costs between the gradient method and our the proposed method are provided.

Keywords: state space search., recurrent neural network, Nonlinear dynamical system.

Mathematics Subject Classification: 68T05, 65K9.

Citation: Leong-Kwan Li, Sally Shao, K. F. Cedric Yiu. Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm. Journal of Industrial and Management Optimization, 2011, 7 (2) : 385-400. doi: 10.3934/jimo.2011.7.385

References:

[1]	A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence, IEEE Transcations on Neural Networks, 11 (2000), 697-709. doi: 10.1109/72.846741.
[2]	Y. Fang and T. W. S. Chow, Non-linear dynamical systems control using a new RNN temporal learning strategy, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 719-723.
[3]	R. A. Conn, K. Scheinberg and N. L. Vicente, "Introduction to Derivative-free Optimization," SIAM, 2009. doi: 10.1137/1.9780898718768.
[4]	J. F. G. Freitas, M. Niranjan, A. H. Gee and A. Doucet, Sequential Monte Carlo methods to train neural network models, Neural Computation, 12 (2000), 955-993. doi: 10.1162/089976600300015664.
[5]	L. K. Li, Learning sunspot series dynamics by recurrent neural networks, in "Advances in Data Mining and Modeling" (eds. W. K. Ching and K. P. Ng), World Science, (2003), 107-115. doi: 10.1142/9789812704955_0009.
[6]	L. K. Li, W. K. Pang, W. T. Yu and M. D. Trout, Forecasting short-term exchange Rates: a recurrent neural network approach, in "Neural Networks in Business Forecasting" (eds. G. P. Zhang), Idea Group Publishing, (2004), 195-212. doi: 10.4018/9781591401766.ch010.
[7]	L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations, International Journal of Pure and Applied Mathematics, 39 (2007), 545-562.
[8]	L. K. Li and S. Shao, A neural network approach for global optimization with applications, Neural Network World, 18 (2008), 365-379.
[9]	L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates, Applied Mathematical Sciences, 2 (2008), 1705-1728.
[10]	X. D. Li, J. K. L. Ho and T. W. S. Chow, Approximation of dynamical time-variant systems by continuous-time recurrent neural networks, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 656-660.
[11]	X. B. Liang and J. Wang, A recurrent neural network for nonlinear optimization with a continuously differentiable objective function and bound constraints, IEEE Transactions on Neural Networks, 11 (2000), 1251-1262. doi: 10.1109/72.883412.
[12]	Z. Liu and I. Elhanany, A Fast and Scalable Recurrent Neural Network Based on Stochastic Meta Descent, IEEE Transactions on Neural Networks, 19 (2008), 1652-1658. doi: 10.1109/TNN.2008.2000838.
[13]	S. Wang, Q. Shao and X. Zhou, Knot-optimizing spline networks (KOSNETS) for nonparametric regression, Journal of Industrial and Management Optimization, 4 (2008), 33?52.
[14]	X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks, Journal of Computer and System Sciences, 45 (1992), 1-19. doi: 10.1016/0022-0000(92)90038-K.
[15]	R. J. Williams and D. Zipser, A learning algorithm for continually running fully recurrent neural networks, Neural Computation, 1 (1989), 270-280. doi: 10.1162/neco.1989.1.2.270.
[16]	L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service, Journal of Industrial and Management Optimization, 1 (2005), 389-404.
[17]	J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex, Neural Computation, 34 (2000), 79-98.
[18]	K. F. C. Yiu, S. Wang, K. L. Teo and A. H. Tsoi, Nonlinear system modeling via knot-optimizing B-spline networks, IEEE Transactions on Neural Networks, 12 (2001), 1013-1022. doi: 10.1109/72.950131.
[19]	K. F. C. Yiu, Y. Liu and K. L. Teo, A hybrid descent method for global optimization, Journal of Global Optimization, 28 (2004), 229-238. doi: 10.1023/B:JOGO.0000015313.93974.b0.

show all references

References:

[1]	A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence, IEEE Transcations on Neural Networks, 11 (2000), 697-709. doi: 10.1109/72.846741.
[2]	Y. Fang and T. W. S. Chow, Non-linear dynamical systems control using a new RNN temporal learning strategy, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 719-723.
[3]	R. A. Conn, K. Scheinberg and N. L. Vicente, "Introduction to Derivative-free Optimization," SIAM, 2009. doi: 10.1137/1.9780898718768.
[4]	J. F. G. Freitas, M. Niranjan, A. H. Gee and A. Doucet, Sequential Monte Carlo methods to train neural network models, Neural Computation, 12 (2000), 955-993. doi: 10.1162/089976600300015664.
[5]	L. K. Li, Learning sunspot series dynamics by recurrent neural networks, in "Advances in Data Mining and Modeling" (eds. W. K. Ching and K. P. Ng), World Science, (2003), 107-115. doi: 10.1142/9789812704955_0009.
[6]	L. K. Li, W. K. Pang, W. T. Yu and M. D. Trout, Forecasting short-term exchange Rates: a recurrent neural network approach, in "Neural Networks in Business Forecasting" (eds. G. P. Zhang), Idea Group Publishing, (2004), 195-212. doi: 10.4018/9781591401766.ch010.
[7]	L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations, International Journal of Pure and Applied Mathematics, 39 (2007), 545-562.
[8]	L. K. Li and S. Shao, A neural network approach for global optimization with applications, Neural Network World, 18 (2008), 365-379.
[9]	L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates, Applied Mathematical Sciences, 2 (2008), 1705-1728.
[10]	X. D. Li, J. K. L. Ho and T. W. S. Chow, Approximation of dynamical time-variant systems by continuous-time recurrent neural networks, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 656-660.
[11]	X. B. Liang and J. Wang, A recurrent neural network for nonlinear optimization with a continuously differentiable objective function and bound constraints, IEEE Transactions on Neural Networks, 11 (2000), 1251-1262. doi: 10.1109/72.883412.
[12]	Z. Liu and I. Elhanany, A Fast and Scalable Recurrent Neural Network Based on Stochastic Meta Descent, IEEE Transactions on Neural Networks, 19 (2008), 1652-1658. doi: 10.1109/TNN.2008.2000838.
[13]	S. Wang, Q. Shao and X. Zhou, Knot-optimizing spline networks (KOSNETS) for nonparametric regression, Journal of Industrial and Management Optimization, 4 (2008), 33?52.
[14]	X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks, Journal of Computer and System Sciences, 45 (1992), 1-19. doi: 10.1016/0022-0000(92)90038-K.
[15]	R. J. Williams and D. Zipser, A learning algorithm for continually running fully recurrent neural networks, Neural Computation, 1 (1989), 270-280. doi: 10.1162/neco.1989.1.2.270.
[16]	L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service, Journal of Industrial and Management Optimization, 1 (2005), 389-404.
[17]	J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex, Neural Computation, 34 (2000), 79-98.
[18]	K. F. C. Yiu, S. Wang, K. L. Teo and A. H. Tsoi, Nonlinear system modeling via knot-optimizing B-spline networks, IEEE Transactions on Neural Networks, 12 (2001), 1013-1022. doi: 10.1109/72.950131.
[19]	K. F. C. Yiu, Y. Liu and K. L. Teo, A hybrid descent method for global optimization, Journal of Global Optimization, 28 (2004), 229-238. doi: 10.1023/B:JOGO.0000015313.93974.b0.

[1]	Leong-Kwan Li, Sally Shao. Convergence analysis of the weighted state space search algorithm for recurrent neural networks. Numerical Algebra, Control and Optimization, 2014, 4 (3) : 193-207. doi: 10.3934/naco.2014.4.193
[2]	K. L. Mak, J. G. Peng, Z. B. Xu, K. F. C. Yiu. A novel neural network for associative memory via dynamical systems. Discrete and Continuous Dynamical Systems - B, 2006, 6 (3) : 573-590. doi: 10.3934/dcdsb.2006.6.573
[3]	Lixin Xu, Wanquan Liu. A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service. Journal of Industrial and Management Optimization, 2005, 1 (3) : 389-404. doi: 10.3934/jimo.2005.1.389
[4]	Sanjay K. Mazumdar, Cheng-Chew Lim. A neural network based anti-skid brake system. Discrete and Continuous Dynamical Systems, 1999, 5 (2) : 321-338. doi: 10.3934/dcds.1999.5.321
[5]	Léo Bois, Emmanuel Franck, Laurent Navoret, Vincent Vigon. A neural network closure for the Euler-Poisson system based on kinetic simulations. Kinetic and Related Models, 2022, 15 (1) : 49-89. doi: 10.3934/krm.2021044
[6]	Meiyu Sui, Yejuan Wang, Peter E. Kloeden. Pullback attractors for stochastic recurrent neural networks with discrete and distributed delays. Electronic Research Archive, 2021, 29 (2) : 2187-2221. doi: 10.3934/era.2020112
[7]	Jianfeng Feng, Mariya Shcherbina, Brunello Tirozzi. Stability of the dynamics of an asymmetric neural network. Communications on Pure and Applied Analysis, 2009, 8 (2) : 655-671. doi: 10.3934/cpaa.2009.8.655
[8]	Yang Mi, Kang Zheng, Song Wang. Homography estimation along short videos by recurrent convolutional regression network. Mathematical Foundations of Computing, 2020, 3 (2) : 125-140. doi: 10.3934/mfc.2020014
[9]	Thi Tuyet Trang Chau, Pierre Ailliot, Valérie Monbet, Pierre Tandeo. Comparison of simulation-based algorithms for parameter estimation and state reconstruction in nonlinear state-space models. Discrete and Continuous Dynamical Systems - S, 2022 doi: 10.3934/dcdss.2022054
[10]	Vena Pearl Bongolan-walsh, David Cheban, Jinqiao Duan. Recurrent motions in the nonautonomous Navier-Stokes system. Discrete and Continuous Dynamical Systems - B, 2003, 3 (2) : 255-262. doi: 10.3934/dcdsb.2003.3.255
[11]	Ndolane Sene. Fractional input stability and its application to neural network. Discrete and Continuous Dynamical Systems - S, 2020, 13 (3) : 853-865. doi: 10.3934/dcdss.2020049
[12]	Ying Sue Huang, Chai Wah Wu. Stability of cellular neural network with small delays. Conference Publications, 2005, 2005 (Special) : 420-426. doi: 10.3934/proc.2005.2005.420
[13]	King Hann Lim, Hong Hui Tan, Hendra G. Harno. Approximate greatest descent in neural network optimization. Numerical Algebra, Control and Optimization, 2018, 8 (3) : 327-336. doi: 10.3934/naco.2018021
[14]	Shyan-Shiou Chen, Chih-Wen Shih. Asymptotic behaviors in a transiently chaotic neural network. Discrete and Continuous Dynamical Systems, 2004, 10 (3) : 805-826. doi: 10.3934/dcds.2004.10.805
[15]	Zhigang Zeng, Tingwen Huang. New passivity analysis of continuous-time recurrent neural networks with multiple discrete delays. Journal of Industrial and Management Optimization, 2011, 7 (2) : 283-289. doi: 10.3934/jimo.2011.7.283
[16]	John R. Tucker. Attractors and kernels: Linking nonlinear PDE semigroups to harmonic analysis state-space decomposition. Conference Publications, 2001, 2001 (Special) : 366-370. doi: 10.3934/proc.2001.2001.366
[17]	Irena Pawłow, Wojciech M. Zajączkowski. Unique solvability of a nonlinear thermoviscoelasticity system in Sobolev space with a mixed norm. Discrete and Continuous Dynamical Systems - S, 2011, 4 (2) : 441-466. doi: 10.3934/dcdss.2011.4.441
[18]	Mo Chen. Recurrent solutions of the Schrödinger-KdV system with boundary forces. Discrete and Continuous Dynamical Systems - B, 2021, 26 (9) : 5149-5170. doi: 10.3934/dcdsb.2020337
[19]	Fred C. Pinto. Nonlinear stability and dynamical properties for a Kuramoto-Sivashinsky equation in space dimension two. Discrete and Continuous Dynamical Systems, 1999, 5 (1) : 117-136. doi: 10.3934/dcds.1999.5.117
[20]	Karl Peter Hadeler. Structured populations with diffusion in state space. Mathematical Biosciences & Engineering, 2010, 7 (1) : 37-49. doi: 10.3934/mbe.2010.7.37

2021 Impact Factor: 1.411

Tools

Metrics

Other articles
by authors

on AIMS
Leong-Kwan Li Sally Shao K. F. Cedric Yiu
on Google Scholar
Leong-Kwan Li Sally Shao K. F. Cedric Yiu

[Back to Top]