September  2016, 6(3): 363-389. doi: 10.3934/mcrf.2016007

A sparse Markov chain approximation of LQ-type stochastic control problems

1. 

School of Mathematics, University of Edinburgh, Edinburgh EH9 3JZ, Scotland, United Kingdom

2. 

Institut für Mathematik, Brandenburgische Technische Universitat Cottbus-Senftenberg, Platz der Deutschen Einheit 1, 03046 Cottbus, Germany

Received  March 2015 Revised  October 2015 Published  August 2016

We propose a novel Galerkin discretization scheme for stochastic optimal control problems on an indefinite time horizon. The control problems are linear-quadratic in the controls, but possibly nonlinear in the state variables, and the discretization is based on the fact that problems of this kind admit a dual formulation in terms of linear boundary value problems. We show that the discretized linear problem is dual to a Markov decision problem, prove an $L^{2}$ error bound for the general scheme and discuss the sparse discretization using a basis of so-called committor functions as a special case; the latter is particularly suited when the dynamics are metastable, e.g., when controlling biomolecular systems. We illustrate the method with several numerical examples, one being the optimal control of Alanine dipeptide to its helical conformation.
Citation: Ralf Banisch, Carsten Hartmann. A sparse Markov chain approximation of LQ-type stochastic control problems. Mathematical Control and Related Fields, 2016, 6 (3) : 363-389. doi: 10.3934/mcrf.2016007
References:
[1]

T. Belytschko, Y. Krongauz, D. Organ, M. Fleming and P. Krysl, Meshless methods: An overview and recent developments, Comput. Methods Appl. M., 139 (1995), 3-47. doi: 10.1016/S0045-7825(96)01078-X.

[2]

A. Berman and R. Plemmons, Nonnegative Matrices in the Mathematical Sciences, Academic Press, 1979.

[3]

O. Bokanowski, J. Garcke, M. Griebel and I. Klompmaker, An adaptive sparse grid semi-lagrangian scheme for first order Hamilton-Jacobi-Bellman equations, J. Sci. Comput., 55 (2013), 575-605. doi: 10.1007/s10915-012-9648-x.

[4]

M. Boulbrachene and M. Haiour, The finite element approximation of Hamilton-Jacobi-Bellman equations, Comput. Math. Appl., 41 (2001), 993-1007. doi: 10.1016/S0898-1221(00)00334-5.

[5]

A. Bovier, Methods of Contemporary Statistical Mechanics, Metastability, Springer, 2009. doi: 10.1007/978-3-540-92796-9.

[6]

D. Braess, Finite Elements: Theory, Fast Solvers, and Applications in Solid Mechanics, Cambridge University Press, 2007. doi: 10.1088/0957-0233/13/9/704.

[7]

G. Bussi, D. Donadio and M. Parrinello, Canonical sampling through velocity rescaling, J. Chem. Phys., 126 (2007), 014101. doi: 10.1063/1.2408420.

[8]

E. Carlini, M. Falcone and R. Ferretti, An efficient algorithm for Hamilton-Jacobi equations in high dimension, Comput. Visual. Sci., 7 (2004), 15-29. doi: 10.1007/s00791-004-0124-5.

[9]

T. Cecil, J. Qian and S. Osher, Numerical methods for high dimensional Hamilton-Jacobi equations using radial basis functions, J. Comput. Phys., 196 (2004), 327-347. doi: 10.1016/j.jcp.2003.11.010.

[10]

P. Dai Pra, L. Meneghini and W. Runggaldier, Connections between stochastic control and dynamic games, Math. Control Signals Systems, 9 (1996), 303-326. doi: 10.1007/BF01211853.

[11]

T. Darden, D. York and L. Pedersen, Particle mesh Ewald: An $N\cdot\log(N)$ method for Ewald sums in large systems}, J. Chem. Phys., 98 (1993), 10089-10092. doi: 10.1063/1.464397.

[12]

N. Djurdjevac, M. Sarich and C. Schütte, Estimating the eigenvalue error of Markov state models, Multiscale Model. Simul., 10 (2012), 61-81. doi: 10.1137/100798910.

[13]

P. Dupuis, K. Spiliopoulos and H. Wang, Importance sampling for multiscale diffusions, Multiscale Model. Simul., 10 (2012), 1-27. doi: 10.1137/110842545.

[14]

A. Faradjian and R. Elber, Computing time scales from reaction coordinates by milestoning, J. Chem. Phys., 120 (2004), 10880-10889. doi: 10.1063/1.1738640.

[15]

W. Fleming, Exit probabilities and optimal stochastic control, Appl. Math. Optim., 4 (1977), 329-346. doi: 10.1007/BF01442148.

[16]

W. Fleming and H. Soner, Controlled Markov Processes and Viscosity Solutions, Springer, 2006.

[17]

I. Gikhman and A. Skorokhod, The Theory of Stochastic Processes II, Springer-Verlag, Berlin, 2004.

[18]

C. Hartmann, R. Banisch, M. Sarich, T. Badowski and C. Schütte, Characterization of rare events in molecular dynamics, Entropy, 16 (2014), 350-376. doi: 10.3390/e16010350.

[19]

C. Hartmann, J. Latorre, G. Pavliotis and W. Zhang, Optimal control of multiscale systems using reduced-order models, J. Comput. Dynamics, 1 (2014), 279-306. doi: 10.3934/jcd.2014.1.279.

[20]

C. Hartmann and C. Schütte, Efficient rare event simulation by optimal nonequilibrium forcing, J. Stat. Mech. Theor. Exp., 11 (2012), 4pp.

[21]

B. Hess, H. Bekker, H. Berendsen and J. Fraaije, LINCS: a linear constraint solver for molecular simulations, J. Comp. Chem., 18 (1997), 1463-1472.

[22]

R. H. Hoppe, Multi-grid methods for Hamilton-Jacobi-Bellman equations, Numer. Math., 49 (1986), 239-254. doi: 10.1007/BF01389627.

[23]

V. Hornak, R. Abel, A. Okur, B. Strockbine, A. Roitberg and C. Simmerling, Comparison of multiple amber force fields and development of improved protein backbone parameters, Proteins, 65 (2006), 712-725. doi: 10.1002/prot.21123.

[24]

C.-S. Huang, S. Wang, C. Chen and Z.-C. Li, A radial basis collocation method for Hamilton-Jacobi-Bellman equations, Automatica, 42 (2006), 2201-2207. doi: 10.1016/j.automatica.2006.07.013.

[25]

H. Kramers, Brownian motion in a field of force and the diffusion model of chemical reactions, Physica, 7 (1940), 284-304. doi: 10.1016/S0031-8914(40)90098-2.

[26]

H. J. Kushner, A survey of some applications of probability and stochastic control theory to finite difference methods for degenerate elliptic and parabolic equations, SIAM Review, 18 (1976), 545-577. doi: 10.1137/1018112.

[27]

H. J. Kushner, Numerical methods for stochastic control problems in finance, in Mathematics of Derivative Securities, Cambridge University Press, 15 (1997), 504-527.

[28]

H. Kushner and P. Dupuis, Numerical Methods for Stochastic Control Problems in Continuous Time, Springer Verlag, 1992. doi: 10.1007/978-1-4684-0441-8.

[29]

J. Mattingly, A. Stuart and D. Higham, Ergodicity for SDEs and approximations: Locally Lipschitz vector fields and degenerate noise, Stochastic Process. Appl., 101 (2002), 185-232. doi: 10.1016/S0304-4149(02)00150-3.

[30]

J. Menaldi, Some estimates for finite difference approximations, SIAM J. Control Optim., 27 (1989), 579-607. doi: 10.1137/0327031.

[31]

R. H. Momeya and Z. B. Salah, The minimal entropy martingale measure (memm) for a markov-modulated exponential lévy model, Asia-Pacific Financial Markets, 19 (2012), 63-98. doi: 10.1007/s10690-011-9142-8.

[32]

B. Øksendal, Stochastic Differential Equations: An Introduction With Applications, Springer, 2003. doi: 10.1007/978-3-642-14394-6.

[33]

M. Peletier, G. Savaré and M. Veneroni, Chemical reactions as $\Gamma$-limit of diffusion, SIAM Review, 54 (2012), 327-352. doi: 10.1137/110858781.

[34]

H. Rabitz, R. de Vivie-Riedle, M. Motzkus and K. Kompa, Whither the future of controlling quantum phenomena?, Science, 288 (2000), 824-828. doi: 10.1126/science.288.5467.824.

[35]

M. Sarich, Projected Transfer Operators, PhD thesis, FU Berlin, 2011.

[36]

M. Sarich, F. Noé and C. Schütte, On the approximation quality of Markov state models, Multiscale Model. Simul., 8 (2010), 1154-1177. doi: 10.1137/090764049.

[37]

M. Sarich and C. Schütte, Metastability and Markov State Models in Molecular Dynamics: Modeling, Analysis, Algorithmic Approaches, AMS, Providence, RI, 2013.

[38]

C. Schütte, F. Noé, J. Lu, M. Sarich and E. Vanden-Eijnden, Markov state models based on milestoning, J. Chem. Phys., 134 (2011), 204105.

[39]

C. Schütte, S. Winkelmann and C. Hartmann, Optimal control of molecular dynamics using Markov state models, Math. Program. (Series B), 134 (2012), 259-282. doi: 10.1007/s10107-012-0547-6.

[40]

S. Sheu, Stochastic control and exit probabilities of jump processes, J. Control Optim., 23 (1985), 306-328. doi: 10.1137/0323022.

[41]

Q. Song, Convergence of Markov chain approximation on generalized HJB equation and its applications, Automatica, 44 (2008), 761-766. doi: 10.1016/j.automatica.2007.07.014.

[42]

Q. Song, G. Yin and Z. Zhang, Numerical methods for controlled regime-switching diffusions and regime-switching jump diffusions, Automatica, 42 (2006), 1147-1157. doi: 10.1016/j.automatica.2006.03.016.

[43]

H. Stapelfeldt, Laser aligned molecules: Applications in physics and chemistry, Physica Scripta, 2004 (2004), 132-136. doi: 10.1238/Physica.Topical.110a00132.

[44]

A. Steinbrecher, Optimal control of robot guided laser material treatment, in Progress in Industrial Mathematics at ECMI 2008 (eds. A. Fitt, J. Norbury, H. Ockendon and E. Wilson), Springer Berlin Heidelberg, 15 (2010), 505-511. doi: 10.1007/978-3-642-12110-4_79.

[45]

M. Sun, Domain decomposition algorithms for solving Hamilton-Jacobi-Bellman equations, Numer. Func. Anal. Optim., 14 (1993), 145-166. doi: 10.1080/01630569308816513.

[46]

D. Van Der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. Mark and H. C. Berendsen, Gromacs: Fast, flexible, and free, J. Comp. Chem., 26 (2005), 1701-1718.

[47]

E. Vanden-Eijnden, Transition path theory, Lect. Notes Phys., 703 (2006), 439-478.

[48]

S. Wang, L. Jennings and K. Teo, Numerical solution of Hamilton-Jacobi-Bellman equations by an upwind finite volume method, J. Global Optim., 27 (2003), 177-192. doi: 10.1023/A:1024980623095.

[49]

H. Wendland, Meshless galerkin methods using radial basis functions, Math. Comput., 68 (1999), 1521-1531. doi: 10.1090/S0025-5718-99-01102-3.

[50]

M. Zhong and E. Todorov, Moving least-squares approximations for linearly-solvable stochastic optimal control problems, J. Control Theory Appl., 9 (2011), 451-463. doi: 10.1007/s11768-011-0275-0.

show all references

References:
[1]

T. Belytschko, Y. Krongauz, D. Organ, M. Fleming and P. Krysl, Meshless methods: An overview and recent developments, Comput. Methods Appl. M., 139 (1995), 3-47. doi: 10.1016/S0045-7825(96)01078-X.

[2]

A. Berman and R. Plemmons, Nonnegative Matrices in the Mathematical Sciences, Academic Press, 1979.

[3]

O. Bokanowski, J. Garcke, M. Griebel and I. Klompmaker, An adaptive sparse grid semi-lagrangian scheme for first order Hamilton-Jacobi-Bellman equations, J. Sci. Comput., 55 (2013), 575-605. doi: 10.1007/s10915-012-9648-x.

[4]

M. Boulbrachene and M. Haiour, The finite element approximation of Hamilton-Jacobi-Bellman equations, Comput. Math. Appl., 41 (2001), 993-1007. doi: 10.1016/S0898-1221(00)00334-5.

[5]

A. Bovier, Methods of Contemporary Statistical Mechanics, Metastability, Springer, 2009. doi: 10.1007/978-3-540-92796-9.

[6]

D. Braess, Finite Elements: Theory, Fast Solvers, and Applications in Solid Mechanics, Cambridge University Press, 2007. doi: 10.1088/0957-0233/13/9/704.

[7]

G. Bussi, D. Donadio and M. Parrinello, Canonical sampling through velocity rescaling, J. Chem. Phys., 126 (2007), 014101. doi: 10.1063/1.2408420.

[8]

E. Carlini, M. Falcone and R. Ferretti, An efficient algorithm for Hamilton-Jacobi equations in high dimension, Comput. Visual. Sci., 7 (2004), 15-29. doi: 10.1007/s00791-004-0124-5.

[9]

T. Cecil, J. Qian and S. Osher, Numerical methods for high dimensional Hamilton-Jacobi equations using radial basis functions, J. Comput. Phys., 196 (2004), 327-347. doi: 10.1016/j.jcp.2003.11.010.

[10]

P. Dai Pra, L. Meneghini and W. Runggaldier, Connections between stochastic control and dynamic games, Math. Control Signals Systems, 9 (1996), 303-326. doi: 10.1007/BF01211853.

[11]

T. Darden, D. York and L. Pedersen, Particle mesh Ewald: An $N\cdot\log(N)$ method for Ewald sums in large systems}, J. Chem. Phys., 98 (1993), 10089-10092. doi: 10.1063/1.464397.

[12]

N. Djurdjevac, M. Sarich and C. Schütte, Estimating the eigenvalue error of Markov state models, Multiscale Model. Simul., 10 (2012), 61-81. doi: 10.1137/100798910.

[13]

P. Dupuis, K. Spiliopoulos and H. Wang, Importance sampling for multiscale diffusions, Multiscale Model. Simul., 10 (2012), 1-27. doi: 10.1137/110842545.

[14]

A. Faradjian and R. Elber, Computing time scales from reaction coordinates by milestoning, J. Chem. Phys., 120 (2004), 10880-10889. doi: 10.1063/1.1738640.

[15]

W. Fleming, Exit probabilities and optimal stochastic control, Appl. Math. Optim., 4 (1977), 329-346. doi: 10.1007/BF01442148.

[16]

W. Fleming and H. Soner, Controlled Markov Processes and Viscosity Solutions, Springer, 2006.

[17]

I. Gikhman and A. Skorokhod, The Theory of Stochastic Processes II, Springer-Verlag, Berlin, 2004.

[18]

C. Hartmann, R. Banisch, M. Sarich, T. Badowski and C. Schütte, Characterization of rare events in molecular dynamics, Entropy, 16 (2014), 350-376. doi: 10.3390/e16010350.

[19]

C. Hartmann, J. Latorre, G. Pavliotis and W. Zhang, Optimal control of multiscale systems using reduced-order models, J. Comput. Dynamics, 1 (2014), 279-306. doi: 10.3934/jcd.2014.1.279.

[20]

C. Hartmann and C. Schütte, Efficient rare event simulation by optimal nonequilibrium forcing, J. Stat. Mech. Theor. Exp., 11 (2012), 4pp.

[21]

B. Hess, H. Bekker, H. Berendsen and J. Fraaije, LINCS: a linear constraint solver for molecular simulations, J. Comp. Chem., 18 (1997), 1463-1472.

[22]

R. H. Hoppe, Multi-grid methods for Hamilton-Jacobi-Bellman equations, Numer. Math., 49 (1986), 239-254. doi: 10.1007/BF01389627.

[23]

V. Hornak, R. Abel, A. Okur, B. Strockbine, A. Roitberg and C. Simmerling, Comparison of multiple amber force fields and development of improved protein backbone parameters, Proteins, 65 (2006), 712-725. doi: 10.1002/prot.21123.

[24]

C.-S. Huang, S. Wang, C. Chen and Z.-C. Li, A radial basis collocation method for Hamilton-Jacobi-Bellman equations, Automatica, 42 (2006), 2201-2207. doi: 10.1016/j.automatica.2006.07.013.

[25]

H. Kramers, Brownian motion in a field of force and the diffusion model of chemical reactions, Physica, 7 (1940), 284-304. doi: 10.1016/S0031-8914(40)90098-2.

[26]

H. J. Kushner, A survey of some applications of probability and stochastic control theory to finite difference methods for degenerate elliptic and parabolic equations, SIAM Review, 18 (1976), 545-577. doi: 10.1137/1018112.

[27]

H. J. Kushner, Numerical methods for stochastic control problems in finance, in Mathematics of Derivative Securities, Cambridge University Press, 15 (1997), 504-527.

[28]

H. Kushner and P. Dupuis, Numerical Methods for Stochastic Control Problems in Continuous Time, Springer Verlag, 1992. doi: 10.1007/978-1-4684-0441-8.

[29]

J. Mattingly, A. Stuart and D. Higham, Ergodicity for SDEs and approximations: Locally Lipschitz vector fields and degenerate noise, Stochastic Process. Appl., 101 (2002), 185-232. doi: 10.1016/S0304-4149(02)00150-3.

[30]

J. Menaldi, Some estimates for finite difference approximations, SIAM J. Control Optim., 27 (1989), 579-607. doi: 10.1137/0327031.

[31]

R. H. Momeya and Z. B. Salah, The minimal entropy martingale measure (memm) for a markov-modulated exponential lévy model, Asia-Pacific Financial Markets, 19 (2012), 63-98. doi: 10.1007/s10690-011-9142-8.

[32]

B. Øksendal, Stochastic Differential Equations: An Introduction With Applications, Springer, 2003. doi: 10.1007/978-3-642-14394-6.

[33]

M. Peletier, G. Savaré and M. Veneroni, Chemical reactions as $\Gamma$-limit of diffusion, SIAM Review, 54 (2012), 327-352. doi: 10.1137/110858781.

[34]

H. Rabitz, R. de Vivie-Riedle, M. Motzkus and K. Kompa, Whither the future of controlling quantum phenomena?, Science, 288 (2000), 824-828. doi: 10.1126/science.288.5467.824.

[35]

M. Sarich, Projected Transfer Operators, PhD thesis, FU Berlin, 2011.

[36]

M. Sarich, F. Noé and C. Schütte, On the approximation quality of Markov state models, Multiscale Model. Simul., 8 (2010), 1154-1177. doi: 10.1137/090764049.

[37]

M. Sarich and C. Schütte, Metastability and Markov State Models in Molecular Dynamics: Modeling, Analysis, Algorithmic Approaches, AMS, Providence, RI, 2013.

[38]

C. Schütte, F. Noé, J. Lu, M. Sarich and E. Vanden-Eijnden, Markov state models based on milestoning, J. Chem. Phys., 134 (2011), 204105.

[39]

C. Schütte, S. Winkelmann and C. Hartmann, Optimal control of molecular dynamics using Markov state models, Math. Program. (Series B), 134 (2012), 259-282. doi: 10.1007/s10107-012-0547-6.

[40]

S. Sheu, Stochastic control and exit probabilities of jump processes, J. Control Optim., 23 (1985), 306-328. doi: 10.1137/0323022.

[41]

Q. Song, Convergence of Markov chain approximation on generalized HJB equation and its applications, Automatica, 44 (2008), 761-766. doi: 10.1016/j.automatica.2007.07.014.

[42]

Q. Song, G. Yin and Z. Zhang, Numerical methods for controlled regime-switching diffusions and regime-switching jump diffusions, Automatica, 42 (2006), 1147-1157. doi: 10.1016/j.automatica.2006.03.016.

[43]

H. Stapelfeldt, Laser aligned molecules: Applications in physics and chemistry, Physica Scripta, 2004 (2004), 132-136. doi: 10.1238/Physica.Topical.110a00132.

[44]

A. Steinbrecher, Optimal control of robot guided laser material treatment, in Progress in Industrial Mathematics at ECMI 2008 (eds. A. Fitt, J. Norbury, H. Ockendon and E. Wilson), Springer Berlin Heidelberg, 15 (2010), 505-511. doi: 10.1007/978-3-642-12110-4_79.

[45]

M. Sun, Domain decomposition algorithms for solving Hamilton-Jacobi-Bellman equations, Numer. Func. Anal. Optim., 14 (1993), 145-166. doi: 10.1080/01630569308816513.

[46]

D. Van Der Spoel, E. Lindahl, B. Hess, G. Groenhof, A. Mark and H. C. Berendsen, Gromacs: Fast, flexible, and free, J. Comp. Chem., 26 (2005), 1701-1718.

[47]

E. Vanden-Eijnden, Transition path theory, Lect. Notes Phys., 703 (2006), 439-478.

[48]

S. Wang, L. Jennings and K. Teo, Numerical solution of Hamilton-Jacobi-Bellman equations by an upwind finite volume method, J. Global Optim., 27 (2003), 177-192. doi: 10.1023/A:1024980623095.

[49]

H. Wendland, Meshless galerkin methods using radial basis functions, Math. Comput., 68 (1999), 1521-1531. doi: 10.1090/S0025-5718-99-01102-3.

[50]

M. Zhong and E. Todorov, Moving least-squares approximations for linearly-solvable stochastic optimal control problems, J. Control Theory Appl., 9 (2011), 451-463. doi: 10.1007/s11768-011-0275-0.

[1]

Ralf Banisch, Carsten Hartmann. Addendum to "A sparse Markov chain approximation of LQ-type stochastic control problems". Mathematical Control and Related Fields, 2017, 7 (4) : 623-623. doi: 10.3934/mcrf.2017023

[2]

Kun Fan, Yang Shen, Tak Kuen Siu, Rongming Wang. On a Markov chain approximation method for option pricing with regime switching. Journal of Industrial and Management Optimization, 2016, 12 (2) : 529-541. doi: 10.3934/jimo.2016.12.529

[3]

Xian Chen, Zhi-Ming Ma. A transformation of Markov jump processes and applications in genetic study. Discrete and Continuous Dynamical Systems, 2014, 34 (12) : 5061-5084. doi: 10.3934/dcds.2014.34.5061

[4]

Jingzhi Tie, Qing Zhang. An optimal mean-reversion trading rule under a Markov chain model. Mathematical Control and Related Fields, 2016, 6 (3) : 467-488. doi: 10.3934/mcrf.2016012

[5]

Samuel N. Cohen, Lukasz Szpruch. On Markovian solutions to Markov Chain BSDEs. Numerical Algebra, Control and Optimization, 2012, 2 (2) : 257-269. doi: 10.3934/naco.2012.2.257

[6]

Ajay Jasra, Kody J. H. Law, Yaxian Xu. Markov chain simulation for multilevel Monte Carlo. Foundations of Data Science, 2021, 3 (1) : 27-47. doi: 10.3934/fods.2021004

[7]

Lin Xu, Rongming Wang. Upper bounds for ruin probabilities in an autoregressive risk model with a Markov chain interest rate. Journal of Industrial and Management Optimization, 2006, 2 (2) : 165-175. doi: 10.3934/jimo.2006.2.165

[8]

H.Thomas Banks, Shuhua Hu. Nonlinear stochastic Markov processes and modeling uncertainty in populations. Mathematical Biosciences & Engineering, 2012, 9 (1) : 1-25. doi: 10.3934/mbe.2012.9.1

[9]

Felix X.-F. Ye, Yue Wang, Hong Qian. Stochastic dynamics: Markov chains and random transformations. Discrete and Continuous Dynamical Systems - B, 2016, 21 (7) : 2337-2361. doi: 10.3934/dcdsb.2016050

[10]

H. W. J. Lee, Y. C. E. Lee, Kar Hung Wong. Differential equation approximation and enhancing control method for finding the PID gain of a quarter-car suspension model with state-dependent ODE. Journal of Industrial and Management Optimization, 2020, 16 (5) : 2305-2330. doi: 10.3934/jimo.2019055

[11]

Michael C. Fu, Bingqing Li, Rongwen Wu, Tianqi Zhang. Option pricing under a discrete-time Markov switching stochastic volatility with co-jump model. Frontiers of Mathematical Finance, 2022, 1 (1) : 137-160. doi: 10.3934/fmf.2021005

[12]

Xin Zhao, Tao Feng, Liang Wang, Zhipeng Qiu. Threshold dynamics and sensitivity analysis of a stochastic semi-Markov switched SIRS epidemic model with nonlinear incidence and vaccination. Discrete and Continuous Dynamical Systems - B, 2021, 26 (12) : 6131-6154. doi: 10.3934/dcdsb.2021010

[13]

Hyukjin Lee, Cheng-Chew Lim, Jinho Choi. Joint backoff control in time and frequency for multichannel wireless systems and its Markov model for analysis. Discrete and Continuous Dynamical Systems - B, 2011, 16 (4) : 1083-1099. doi: 10.3934/dcdsb.2011.16.1083

[14]

Andrzej Nowakowski, Jan Sokolowski. On dual dynamic programming in shape control. Communications on Pure and Applied Analysis, 2012, 11 (6) : 2473-2485. doi: 10.3934/cpaa.2012.11.2473

[15]

Kehan Si, Zhenda Xu, Ka Fai Cedric Yiu, Xun Li. Open-loop solvability for mean-field stochastic linear quadratic optimal control problems of Markov regime-switching system. Journal of Industrial and Management Optimization, 2021  doi: 10.3934/jimo.2021074

[16]

Mark F. Demers, Christopher J. Ianzano, Philip Mayer, Peter Morfe, Elizabeth C. Yoo. Limiting distributions for countable state topological Markov chains with holes. Discrete and Continuous Dynamical Systems, 2017, 37 (1) : 105-130. doi: 10.3934/dcds.2017005

[17]

Mingzheng Wang, M. Montaz Ali, Guihua Lin. Sample average approximation method for stochastic complementarity problems with applications to supply chain supernetworks. Journal of Industrial and Management Optimization, 2011, 7 (2) : 317-345. doi: 10.3934/jimo.2011.7.317

[18]

Lin Xu, Rongming Wang, Dingjun Yao. Optimal stochastic investment games under Markov regime switching market. Journal of Industrial and Management Optimization, 2014, 10 (3) : 795-815. doi: 10.3934/jimo.2014.10.795

[19]

Demetris Hadjiloucas. Stochastic matrix-valued cocycles and non-homogeneous Markov chains. Discrete and Continuous Dynamical Systems, 2007, 17 (4) : 731-738. doi: 10.3934/dcds.2007.17.731

[20]

Daoyi Xu, Yumei Huang, Zhiguo Yang. Existence theorems for periodic Markov process and stochastic functional differential equations. Discrete and Continuous Dynamical Systems, 2009, 24 (3) : 1005-1023. doi: 10.3934/dcds.2009.24.1005

2020 Impact Factor: 1.284

Metrics

  • PDF downloads (186)
  • HTML views (0)
  • Cited by (4)

Other articles
by authors

[Back to Top]