Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm

Previous Article
Multi-objective aggregate production planning decisions using two-phase fuzzy goal programming method
JIMO Home
This Issue
Next Article
An integrated approach for the operations of distribution and lateral transshipment for seasonal products - A case study in household product industry

April 2011, 7(2): 385-400. doi: 10.3934/jimo.2011.7.385

Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm

Leong-Kwan Li ^1, , Sally Shao ^2, and K. F. Cedric Yiu ^1,

1.	Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China
2.	Department of Mathematics, Cleveland State University, Cleveland, OH 44115, United States

Received October 2009 Revised January 2011 Published April 2011

Abstract
Related Papers

Given a task of tracking a trajectory, a recurrent neural network may be considered as a black-box nonlinear regression model for tracking unknown dynamic systems. An error function is used to measure the difference between the system outputs and the desired trajectory that formulates a nonlinear least square problem with dynamical constraints. With the dynamical constraints, classical gradient type methods are difficult and time consuming due to the involving of the computation of the partial derivatives along the trajectory. We develop an alternative learning algorithm, namely the weighted state space search algorithm, which searches the neighborhood of the target trajectory in the state space instead of the parameter space. Since there is no computation of partial derivatives involved, our algorithm is simple and fast. We demonstrate our approach by modeling the short-term foreign exchange rates. The empirical results show that the weighted state space search method is very promising and effective in solving least square problems with dynamical constraints. Numerical costs between the gradient method and our the proposed method are provided.

Keywords: state space search., recurrent neural network, Nonlinear dynamical system.

Mathematics Subject Classification: 68T05, 65K9.

Citation: Leong-Kwan Li, Sally Shao, K. F. Cedric Yiu. Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm. Journal of Industrial & Management Optimization, 2011, 7 (2) : 385-400. doi: 10.3934/jimo.2011.7.385

References:

[1]	A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence,, IEEE Transcations on Neural Networks, 11 (2000), 697. doi: 10.1109/72.846741.
[2]	Y. Fang and T. W. S. Chow, Non-linear dynamical systems control using a new RNN temporal learning strategy,, IEEE Trans on Circuit and Systems, 52 (2005), 719.
[3]	R. A. Conn, K. Scheinberg and N. L. Vicente, "Introduction to Derivative-free Optimization,", SIAM, (2009). doi: 10.1137/1.9780898718768.
[4]	J. F. G. Freitas, M. Niranjan, A. H. Gee and A. Doucet, Sequential Monte Carlo methods to train neural network models,, Neural Computation, 12 (2000), 955. doi: 10.1162/089976600300015664.
[5]	L. K. Li, Learning sunspot series dynamics by recurrent neural networks,, in, (2003), 107. doi: 10.1142/9789812704955_0009.
[6]	L. K. Li, W. K. Pang, W. T. Yu and M. D. Trout, Forecasting short-term exchange Rates: a recurrent neural network approach,, in, (2004), 195. doi: 10.4018/9781591401766.ch010.
[7]	L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations,, International Journal of Pure and Applied Mathematics, 39 (2007), 545.
[8]	L. K. Li and S. Shao, A neural network approach for global optimization with applications,, Neural Network World, 18 (2008), 365.
[9]	L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates,, Applied Mathematical Sciences, 2 (2008), 1705.
[10]	X. D. Li, J. K. L. Ho and T. W. S. Chow, Approximation of dynamical time-variant systems by continuous-time recurrent neural networks,, IEEE Trans on Circuit and Systems, 52 (2005), 656.
[11]	X. B. Liang and J. Wang, A recurrent neural network for nonlinear optimization with a continuously differentiable objective function and bound constraints,, IEEE Transactions on Neural Networks, 11 (2000), 1251. doi: 10.1109/72.883412.
[12]	Z. Liu and I. Elhanany, A Fast and Scalable Recurrent Neural Network Based on Stochastic Meta Descent,, IEEE Transactions on Neural Networks, 19 (2008), 1652. doi: 10.1109/TNN.2008.2000838.
[13]	S. Wang, Q. Shao and X. Zhou, Knot-optimizing spline networks (KOSNETS) for nonparametric regression,, Journal of Industrial and Management Optimization, 4 (2008).
[14]	X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks,, Journal of Computer and System Sciences, 45 (1992), 1. doi: 10.1016/0022-0000(92)90038-K.
[15]	R. J. Williams and D. Zipser, A learning algorithm for continually running fully recurrent neural networks,, Neural Computation, 1 (1989), 270. doi: 10.1162/neco.1989.1.2.270.
[16]	L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service,, Journal of Industrial and Management Optimization, 1 (2005), 389.
[17]	J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex,, Neural Computation, 34 (2000), 79.
[18]	K. F. C. Yiu, S. Wang, K. L. Teo and A. H. Tsoi, Nonlinear system modeling via knot-optimizing B-spline networks,, IEEE Transactions on Neural Networks, 12 (2001), 1013. doi: 10.1109/72.950131.
[19]	K. F. C. Yiu, Y. Liu and K. L. Teo, A hybrid descent method for global optimization,, Journal of Global Optimization, 28 (2004), 229. doi: 10.1023/B:JOGO.0000015313.93974.b0.

show all references

References:

[1]	A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence,, IEEE Transcations on Neural Networks, 11 (2000), 697. doi: 10.1109/72.846741.
[2]	Y. Fang and T. W. S. Chow, Non-linear dynamical systems control using a new RNN temporal learning strategy,, IEEE Trans on Circuit and Systems, 52 (2005), 719.
[3]	R. A. Conn, K. Scheinberg and N. L. Vicente, "Introduction to Derivative-free Optimization,", SIAM, (2009). doi: 10.1137/1.9780898718768.
[4]	J. F. G. Freitas, M. Niranjan, A. H. Gee and A. Doucet, Sequential Monte Carlo methods to train neural network models,, Neural Computation, 12 (2000), 955. doi: 10.1162/089976600300015664.
[5]	L. K. Li, Learning sunspot series dynamics by recurrent neural networks,, in, (2003), 107. doi: 10.1142/9789812704955_0009.
[6]	L. K. Li, W. K. Pang, W. T. Yu and M. D. Trout, Forecasting short-term exchange Rates: a recurrent neural network approach,, in, (2004), 195. doi: 10.4018/9781591401766.ch010.
[7]	L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations,, International Journal of Pure and Applied Mathematics, 39 (2007), 545.
[8]	L. K. Li and S. Shao, A neural network approach for global optimization with applications,, Neural Network World, 18 (2008), 365.
[9]	L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates,, Applied Mathematical Sciences, 2 (2008), 1705.
[10]	X. D. Li, J. K. L. Ho and T. W. S. Chow, Approximation of dynamical time-variant systems by continuous-time recurrent neural networks,, IEEE Trans on Circuit and Systems, 52 (2005), 656.
[11]	X. B. Liang and J. Wang, A recurrent neural network for nonlinear optimization with a continuously differentiable objective function and bound constraints,, IEEE Transactions on Neural Networks, 11 (2000), 1251. doi: 10.1109/72.883412.
[12]	Z. Liu and I. Elhanany, A Fast and Scalable Recurrent Neural Network Based on Stochastic Meta Descent,, IEEE Transactions on Neural Networks, 19 (2008), 1652. doi: 10.1109/TNN.2008.2000838.
[13]	S. Wang, Q. Shao and X. Zhou, Knot-optimizing spline networks (KOSNETS) for nonparametric regression,, Journal of Industrial and Management Optimization, 4 (2008).
[14]	X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks,, Journal of Computer and System Sciences, 45 (1992), 1. doi: 10.1016/0022-0000(92)90038-K.
[15]	R. J. Williams and D. Zipser, A learning algorithm for continually running fully recurrent neural networks,, Neural Computation, 1 (1989), 270. doi: 10.1162/neco.1989.1.2.270.
[16]	L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service,, Journal of Industrial and Management Optimization, 1 (2005), 389.
[17]	J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex,, Neural Computation, 34 (2000), 79.
[18]	K. F. C. Yiu, S. Wang, K. L. Teo and A. H. Tsoi, Nonlinear system modeling via knot-optimizing B-spline networks,, IEEE Transactions on Neural Networks, 12 (2001), 1013. doi: 10.1109/72.950131.
[19]	K. F. C. Yiu, Y. Liu and K. L. Teo, A hybrid descent method for global optimization,, Journal of Global Optimization, 28 (2004), 229. doi: 10.1023/B:JOGO.0000015313.93974.b0.

[1]	Leong-Kwan Li, Sally Shao. Convergence analysis of the weighted state space search algorithm for recurrent neural networks. Numerical Algebra, Control & Optimization, 2014, 4 (3) : 193-207. doi: 10.3934/naco.2014.4.193
[2]	K. L. Mak, J. G. Peng, Z. B. Xu, K. F. C. Yiu. A novel neural network for associative memory via dynamical systems. Discrete & Continuous Dynamical Systems - B, 2006, 6 (3) : 573-590. doi: 10.3934/dcdsb.2006.6.573
[3]	Lixin Xu, Wanquan Liu. A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service. Journal of Industrial & Management Optimization, 2005, 1 (3) : 389-404. doi: 10.3934/jimo.2005.1.389
[4]	Sanjay K. Mazumdar, Cheng-Chew Lim. A neural network based anti-skid brake system. Discrete & Continuous Dynamical Systems - A, 1999, 5 (2) : 321-338. doi: 10.3934/dcds.1999.5.321
[5]	Jianfeng Feng, Mariya Shcherbina, Brunello Tirozzi. Stability of the dynamics of an asymmetric neural network. Communications on Pure & Applied Analysis, 2009, 8 (2) : 655-671. doi: 10.3934/cpaa.2009.8.655
[6]	Vena Pearl Bongolan-walsh, David Cheban, Jinqiao Duan. Recurrent motions in the nonautonomous Navier-Stokes system. Discrete & Continuous Dynamical Systems - B, 2003, 3 (2) : 255-262. doi: 10.3934/dcdsb.2003.3.255
[7]	Ying Sue Huang, Chai Wah Wu. Stability of cellular neural network with small delays. Conference Publications, 2005, 2005 (Special) : 420-426. doi: 10.3934/proc.2005.2005.420
[8]	King Hann Lim, Hong Hui Tan, Hendra G. Harno. Approximate greatest descent in neural network optimization. Numerical Algebra, Control & Optimization, 2018, 8 (3) : 327-336. doi: 10.3934/naco.2018021
[9]	Shyan-Shiou Chen, Chih-Wen Shih. Asymptotic behaviors in a transiently chaotic neural network. Discrete & Continuous Dynamical Systems - A, 2004, 10 (3) : 805-826. doi: 10.3934/dcds.2004.10.805
[10]	Ndolane Sene. Fractional input stability and its application to neural network. Discrete & Continuous Dynamical Systems - S, 2018, 0 (0) : 853-865. doi: 10.3934/dcdss.2020049
[11]	Zhigang Zeng, Tingwen Huang. New passivity analysis of continuous-time recurrent neural networks with multiple discrete delays. Journal of Industrial & Management Optimization, 2011, 7 (2) : 283-289. doi: 10.3934/jimo.2011.7.283
[12]	Irena Pawłow, Wojciech M. Zajączkowski. Unique solvability of a nonlinear thermoviscoelasticity system in Sobolev space with a mixed norm. Discrete & Continuous Dynamical Systems - S, 2011, 4 (2) : 441-466. doi: 10.3934/dcdss.2011.4.441
[13]	John R. Tucker. Attractors and kernels: Linking nonlinear PDE semigroups to harmonic analysis state-space decomposition. Conference Publications, 2001, 2001 (Special) : 366-370. doi: 10.3934/proc.2001.2001.366
[14]	Rui Hu, Yuan Yuan. Stability, bifurcation analysis in a neural network model with delay and diffusion. Conference Publications, 2009, 2009 (Special) : 367-376. doi: 10.3934/proc.2009.2009.367
[15]	Hui-Qiang Ma, Nan-Jing Huang. Neural network smoothing approximation method for stochastic variational inequality problems. Journal of Industrial & Management Optimization, 2015, 11 (2) : 645-660. doi: 10.3934/jimo.2015.11.645
[16]	Yixin Guo, Aijun Zhang. Existence and nonexistence of traveling pulses in a lateral inhibition neural network. Discrete & Continuous Dynamical Systems - B, 2016, 21 (6) : 1729-1755. doi: 10.3934/dcdsb.2016020
[17]	Jianhong Wu, Ruyuan Zhang. A simple delayed neural network with large capacity for associative memory. Discrete & Continuous Dynamical Systems - B, 2004, 4 (3) : 851-863. doi: 10.3934/dcdsb.2004.4.851
[18]	Honggang Yu. An efficient face recognition algorithm using the improved convolutional neural network. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 901-914. doi: 10.3934/dcdss.2019060
[19]	Lidong Liu, Fajie Wei, Shenghan Zhou. Major project risk assessment method based on BP neural network. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1053-1064. doi: 10.3934/dcdss.2019072
[20]	Zhuwei Qin, Fuxun Yu, Chenchen Liu, Xiang Chen. How convolutional neural networks see the world --- A survey of convolutional neural network visualization methods. Mathematical Foundations of Computing, 2018, 1 (2) : 149-180. doi: 10.3934/mfc.2018008

2018 Impact Factor: 1.025

American Institute of Mathematical Sciences