American Institute of Mathematical Sciences

Advanced
  • Previous Article
    An integrated approach for the operations of distribution and lateral transshipment for seasonal products - A case study in household product industry
  • JIMO Home
  • This Issue
  • Next Article
    Multi-objective aggregate production planning decisions using two-phase fuzzy goal programming method
April  2011, 7(2): 385-400. doi: 10.3934/jimo.2011.7.385

Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm

Leong-Kwan Li 1, , Sally Shao 2, and  K. F. Cedric Yiu 1,

1. 

Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong, China
2. 

Department of Mathematics, Cleveland State University, Cleveland, OH 44115, United States

Received  October 2009 Revised  January 2011 Published  April 2011

Given a task of tracking a trajectory, a recurrent neural network may be considered as a black-box nonlinear regression model for tracking unknown dynamic systems. An error function is used to measure the difference between the system outputs and the desired trajectory that formulates a nonlinear least square problem with dynamical constraints. With the dynamical constraints, classical gradient type methods are difficult and time consuming due to the involving of the computation of the partial derivatives along the trajectory. We develop an alternative learning algorithm, namely the weighted state space search algorithm, which searches the neighborhood of the target trajectory in the state space instead of the parameter space. Since there is no computation of partial derivatives involved, our algorithm is simple and fast. We demonstrate our approach by modeling the short-term foreign exchange rates. The empirical results show that the weighted state space search method is very promising and effective in solving least square problems with dynamical constraints. Numerical costs between the gradient method and our the proposed method are provided.
Keywords: state space search., recurrent neural network, Nonlinear dynamical system.
Mathematics Subject Classification: 68T05, 65K9.
Citation: Leong-Kwan Li, Sally Shao, K. F. Cedric Yiu. Nonlinear dynamical system modeling via recurrent neural networks and a weighted state space search algorithm. Journal of Industrial & Management Optimization, 2011, 7 (2) : 385-400. doi: 10.3934/jimo.2011.7.385
References:
[1]

A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence, IEEE Transcations on Neural Networks, 11 (2000), 697-709. doi: 10.1109/72.846741.  Google Scholar

[2]

Y. Fang and T. W. S. Chow, Non-linear dynamical systems control using a new RNN temporal learning strategy, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 719-723. Google Scholar

[3]

R. A. Conn, K. Scheinberg and N. L. Vicente, "Introduction to Derivative-free Optimization," SIAM, 2009. doi: 10.1137/1.9780898718768.  Google Scholar

[4]

J. F. G. Freitas, M. Niranjan, A. H. Gee and A. Doucet, Sequential Monte Carlo methods to train neural network models, Neural Computation, 12 (2000), 955-993. doi: 10.1162/089976600300015664.  Google Scholar

[5]

L. K. Li, Learning sunspot series dynamics by recurrent neural networks, in "Advances in Data Mining and Modeling" (eds. W. K. Ching and K. P. Ng), World Science, (2003), 107-115. doi: 10.1142/9789812704955_0009.  Google Scholar

[6]

L. K. Li, W. K. Pang, W. T. Yu and M. D. Trout, Forecasting short-term exchange Rates: a recurrent neural network approach, in "Neural Networks in Business Forecasting" (eds. G. P. Zhang), Idea Group Publishing, (2004), 195-212. doi: 10.4018/9781591401766.ch010.  Google Scholar

[7]

L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations, International Journal of Pure and Applied Mathematics, 39 (2007), 545-562.  Google Scholar

[8]

L. K. Li and S. Shao, A neural network approach for global optimization with applications, Neural Network World, 18 (2008), 365-379. Google Scholar

[9]

L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates, Applied Mathematical Sciences, 2 (2008), 1705-1728.  Google Scholar

[10]

X. D. Li, J. K. L. Ho and T. W. S. Chow, Approximation of dynamical time-variant systems by continuous-time recurrent neural networks, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 656-660. Google Scholar

[11]

X. B. Liang and J. Wang, A recurrent neural network for nonlinear optimization with a continuously differentiable objective function and bound constraints, IEEE Transactions on Neural Networks, 11 (2000), 1251-1262. doi: 10.1109/72.883412.  Google Scholar

[12]

Z. Liu and I. Elhanany, A Fast and Scalable Recurrent Neural Network Based on Stochastic Meta Descent, IEEE Transactions on Neural Networks, 19 (2008), 1652-1658. doi: 10.1109/TNN.2008.2000838.  Google Scholar

[13]

S. Wang, Q. Shao and X. Zhou, Knot-optimizing spline networks (KOSNETS) for nonparametric regression, Journal of Industrial and Management Optimization, 4 (2008), 33?52. Google Scholar

[14]

X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks, Journal of Computer and System Sciences, 45 (1992), 1-19. doi: 10.1016/0022-0000(92)90038-K.  Google Scholar

[15]

R. J. Williams and D. Zipser, A learning algorithm for continually running fully recurrent neural networks, Neural Computation, 1 (1989), 270-280. doi: 10.1162/neco.1989.1.2.270.  Google Scholar

[16]

L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service, Journal of Industrial and Management Optimization, 1 (2005), 389-404. Google Scholar

[17]

J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex, Neural Computation, 34 (2000), 79-98. Google Scholar

[18]

K. F. C. Yiu, S. Wang, K. L. Teo and A. H. Tsoi, Nonlinear system modeling via knot-optimizing B-spline networks, IEEE Transactions on Neural Networks, 12 (2001), 1013-1022. doi: 10.1109/72.950131.  Google Scholar

[19]

K. F. C. Yiu, Y. Liu and K. L. Teo, A hybrid descent method for global optimization, Journal of Global Optimization, 28 (2004), 229-238. doi: 10.1023/B:JOGO.0000015313.93974.b0.  Google Scholar

show all references

References:
[1]

A. F. Atiya and A. G. Parlos, New results on recurrent network training: Unifying the algorithms and accelerating convergence, IEEE Transcations on Neural Networks, 11 (2000), 697-709. doi: 10.1109/72.846741.  Google Scholar

[2]

Y. Fang and T. W. S. Chow, Non-linear dynamical systems control using a new RNN temporal learning strategy, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 719-723. Google Scholar

[3]

R. A. Conn, K. Scheinberg and N. L. Vicente, "Introduction to Derivative-free Optimization," SIAM, 2009. doi: 10.1137/1.9780898718768.  Google Scholar

[4]

J. F. G. Freitas, M. Niranjan, A. H. Gee and A. Doucet, Sequential Monte Carlo methods to train neural network models, Neural Computation, 12 (2000), 955-993. doi: 10.1162/089976600300015664.  Google Scholar

[5]

L. K. Li, Learning sunspot series dynamics by recurrent neural networks, in "Advances in Data Mining and Modeling" (eds. W. K. Ching and K. P. Ng), World Science, (2003), 107-115. doi: 10.1142/9789812704955_0009.  Google Scholar

[6]

L. K. Li, W. K. Pang, W. T. Yu and M. D. Trout, Forecasting short-term exchange Rates: a recurrent neural network approach, in "Neural Networks in Business Forecasting" (eds. G. P. Zhang), Idea Group Publishing, (2004), 195-212. doi: 10.4018/9781591401766.ch010.  Google Scholar

[7]

L. K. Li and S. Shao, Dynamic properties of recurrent neural networks and its approximations, International Journal of Pure and Applied Mathematics, 39 (2007), 545-562.  Google Scholar

[8]

L. K. Li and S. Shao, A neural network approach for global optimization with applications, Neural Network World, 18 (2008), 365-379. Google Scholar

[9]

L. K. Li, S. Shao and T. Zheleva, A state space search algorithm and its application to learn the short-term foreign exchange rates, Applied Mathematical Sciences, 2 (2008), 1705-1728.  Google Scholar

[10]

X. D. Li, J. K. L. Ho and T. W. S. Chow, Approximation of dynamical time-variant systems by continuous-time recurrent neural networks, IEEE Trans on Circuit and Systems, Part II, 52 (2005), 656-660. Google Scholar

[11]

X. B. Liang and J. Wang, A recurrent neural network for nonlinear optimization with a continuously differentiable objective function and bound constraints, IEEE Transactions on Neural Networks, 11 (2000), 1251-1262. doi: 10.1109/72.883412.  Google Scholar

[12]

Z. Liu and I. Elhanany, A Fast and Scalable Recurrent Neural Network Based on Stochastic Meta Descent, IEEE Transactions on Neural Networks, 19 (2008), 1652-1658. doi: 10.1109/TNN.2008.2000838.  Google Scholar

[13]

S. Wang, Q. Shao and X. Zhou, Knot-optimizing spline networks (KOSNETS) for nonparametric regression, Journal of Industrial and Management Optimization, 4 (2008), 33?52. Google Scholar

[14]

X. Wang and E. K. Blum, Discrete-time versus continuous-time models of neural networks, Journal of Computer and System Sciences, 45 (1992), 1-19. doi: 10.1016/0022-0000(92)90038-K.  Google Scholar

[15]

R. J. Williams and D. Zipser, A learning algorithm for continually running fully recurrent neural networks, Neural Computation, 1 (1989), 270-280. doi: 10.1162/neco.1989.1.2.270.  Google Scholar

[16]

L. Xu and W. Liu, A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service, Journal of Industrial and Management Optimization, 1 (2005), 389-404. Google Scholar

[17]

J. Yao and C. L. Tan, A case study on using neural networks to perform technical forecasting of forex, Neural Computation, 34 (2000), 79-98. Google Scholar

[18]

K. F. C. Yiu, S. Wang, K. L. Teo and A. H. Tsoi, Nonlinear system modeling via knot-optimizing B-spline networks, IEEE Transactions on Neural Networks, 12 (2001), 1013-1022. doi: 10.1109/72.950131.  Google Scholar

[19]

K. F. C. Yiu, Y. Liu and K. L. Teo, A hybrid descent method for global optimization, Journal of Global Optimization, 28 (2004), 229-238. doi: 10.1023/B:JOGO.0000015313.93974.b0.  Google Scholar

[1]

Leong-Kwan Li, Sally Shao. Convergence analysis of the weighted state space search algorithm for recurrent neural networks. Numerical Algebra, Control & Optimization, 2014, 4 (3) : 193-207. doi: 10.3934/naco.2014.4.193
[2]

K. L. Mak, J. G. Peng, Z. B. Xu, K. F. C. Yiu. A novel neural network for associative memory via dynamical systems. Discrete & Continuous Dynamical Systems - B, 2006, 6 (3) : 573-590. doi: 10.3934/dcdsb.2006.6.573
[3]

Lixin Xu, Wanquan Liu. A new recurrent neural network adaptive approach for host-gate way rate control protocol within intranets using ATM ABR service. Journal of Industrial & Management Optimization, 2005, 1 (3) : 389-404. doi: 10.3934/jimo.2005.1.389
[4]

Sanjay K. Mazumdar, Cheng-Chew Lim. A neural network based anti-skid brake system. Discrete & Continuous Dynamical Systems, 1999, 5 (2) : 321-338. doi: 10.3934/dcds.1999.5.321
[5]

Meiyu Sui, Yejuan Wang, Peter E. Kloeden. Pullback attractors for stochastic recurrent neural networks with discrete and distributed delays. Electronic Research Archive, 2021, 29 (2) : 2187-2221. doi: 10.3934/era.2020112
[6]

Jianfeng Feng, Mariya Shcherbina, Brunello Tirozzi. Stability of the dynamics of an asymmetric neural network. Communications on Pure & Applied Analysis, 2009, 8 (2) : 655-671. doi: 10.3934/cpaa.2009.8.655
[7]

Yang Mi, Kang Zheng, Song Wang. Homography estimation along short videos by recurrent convolutional regression network. Mathematical Foundations of Computing, 2020, 3 (2) : 125-140. doi: 10.3934/mfc.2020014
[8]

Vena Pearl Bongolan-walsh, David Cheban, Jinqiao Duan. Recurrent motions in the nonautonomous Navier-Stokes system. Discrete & Continuous Dynamical Systems - B, 2003, 3 (2) : 255-262. doi: 10.3934/dcdsb.2003.3.255
[9]

Ndolane Sene. Fractional input stability and its application to neural network. Discrete & Continuous Dynamical Systems - S, 2020, 13 (3) : 853-865. doi: 10.3934/dcdss.2020049
[10]

Ying Sue Huang, Chai Wah Wu. Stability of cellular neural network with small delays. Conference Publications, 2005, 2005 (Special) : 420-426. doi: 10.3934/proc.2005.2005.420
[11]

King Hann Lim, Hong Hui Tan, Hendra G. Harno. Approximate greatest descent in neural network optimization. Numerical Algebra, Control & Optimization, 2018, 8 (3) : 327-336. doi: 10.3934/naco.2018021
[12]

Shyan-Shiou Chen, Chih-Wen Shih. Asymptotic behaviors in a transiently chaotic neural network. Discrete & Continuous Dynamical Systems, 2004, 10 (3) : 805-826. doi: 10.3934/dcds.2004.10.805
[13]

Zhigang Zeng, Tingwen Huang. New passivity analysis of continuous-time recurrent neural networks with multiple discrete delays. Journal of Industrial & Management Optimization, 2011, 7 (2) : 283-289. doi: 10.3934/jimo.2011.7.283
[14]

John R. Tucker. Attractors and kernels: Linking nonlinear PDE semigroups to harmonic analysis state-space decomposition. Conference Publications, 2001, 2001 (Special) : 366-370. doi: 10.3934/proc.2001.2001.366
[15]

Irena Pawłow, Wojciech M. Zajączkowski. Unique solvability of a nonlinear thermoviscoelasticity system in Sobolev space with a mixed norm. Discrete & Continuous Dynamical Systems - S, 2011, 4 (2) : 441-466. doi: 10.3934/dcdss.2011.4.441
[16]

Mo Chen. Recurrent solutions of the Schrödinger-KdV system with boundary forces. Discrete & Continuous Dynamical Systems - B, 2021, 26 (9) : 5149-5170. doi: 10.3934/dcdsb.2020337
[17]

Fred C. Pinto. Nonlinear stability and dynamical properties for a Kuramoto-Sivashinsky equation in space dimension two. Discrete & Continuous Dynamical Systems, 1999, 5 (1) : 117-136. doi: 10.3934/dcds.1999.5.117
[18]

Karl Peter Hadeler. Structured populations with diffusion in state space. Mathematical Biosciences & Engineering, 2010, 7 (1) : 37-49. doi: 10.3934/mbe.2010.7.37
[19]

Rui Hu, Yuan Yuan. Stability, bifurcation analysis in a neural network model with delay and diffusion. Conference Publications, 2009, 2009 (Special) : 367-376. doi: 10.3934/proc.2009.2009.367
[20]

Hui-Qiang Ma, Nan-Jing Huang. Neural network smoothing approximation method for stochastic variational inequality problems. Journal of Industrial & Management Optimization, 2015, 11 (2) : 645-660. doi: 10.3934/jimo.2015.11.645

2020 Impact Factor: 1.801

Tools

Metrics

  • PDF downloads (139)
  • HTML views (0)
  • Cited by (1)

Other articles
by authors

[Back to Top]

Copyright © 2021 American Institute of Mathematical Sciences