Maximum principle for discrete-time stochastic optimal control problem and stochastic game

Zhen Wu; Feng Zhang

doi:10.3934/mcrf.2021031

Article Contents

2022, Volume 12, Issue 2: 475-493. Doi: 10.3934/mcrf.2021031

This issue Previous Article Local well-posedness for a class of 1D Boussinesq systems Next Article Existence and cost of boundary controls for a degenerate/singular parabolic equation

Maximum principle for discrete-time stochastic optimal control problem and stochastic game

Zhen Wu^1, and
Feng Zhang^2, ,

1.
School of Mathematics, Shandong University, Jinan 250100, Shandong Province, China
2.
School of Mathematics and Quantitative Economics, Shandong University of Finance and Economics, Jinan 250014, Shandong Province, China

^* Corresponding author: Feng Zhang
^* Corresponding author: Feng Zhang

Received: October 2020

Revised: February 2021

Early access: June 2021

Published: June 2022

Abstract / Introduction Full Text(HTML) Figure(4) Related Papers Cited by

Abstract

This paper is first concerned with one kind of discrete-time stochastic optimal control problem with convex control domains, for which necessary condition in the form of Pontryagin's maximum principle and sufficient condition of optimality are derived. The results are then extended to two kinds of discrete-time stochastic games. Two illustrative examples are studied, for which the explicit optimal strategies are given. This paper establishes a rigorous version of discrete-time stochastic maximum principle in a clear and concise way and paves a road for further related topics.

Keywords:

Mathematics Subject Classification: Primary: 93E20, 93C55; Secondary: 91A15.

Citation:

Full Text(HTML)

Figure 1. The sequences $ \{\alpha_{1,k}\} $ and $ \{\beta_{1,k}\} $

Download: Full-size image PowerPoint slide

Figure 2. The sequences $ \alpha_{2,k} $ and $\beta_{2,k} $

Download: Full-size image PowerPoint slide

Figure 3. The sequences $ \Psi_{k} $ and $ \Phi_{k} $

Download: Full-size image PowerPoint slide

Figure 4. The sequences $ \{\psi_{k}\} $ and $ \{\phi_{k}\} $

Download: Full-size image PowerPoint slide

Related Papers

Cited by

References

[1]	T. T. K. An and B. Øksendal, Maximum principle for stochastic differential games with partial information, J. Optim. Theory Appl., 139 (2008), 463-483. doi: 10.1007/s10957-008-9398-y.
[2]	A. Beghi and D. D'Alessandro, Discrete-time optimal control with control-dependent noise and generalized Riccati difference equations, Automatica, 34 (1998), 1031-1034. doi: 10.1016/S0005-1098(98)00044-2.
[3]	L. Chen and Z. Y. Yu, Maximum principle for nonzero-sum stochastic differential game with delays, IEEE Trans. Automat. Control, 60 (2015), 1422-1426. doi: 10.1109/TAC.2014.2352731.
[4]	S. N. Cohen and R. J. Elliott, A general theory of finite state backward stochastic difference equations, Stoch. Proc. Appl., 120 (2010), 442-466. doi: 10.1016/j.spa.2010.01.004.
[5]	S. N. Cohen and R. J. Elliott, Backward stochastic difference equations and nearly time-consistent nonlinear expectations, SIAM J. Control Optim., 49 (2011), 125-139. doi: 10.1137/090763688.
[6]	O. L. V. Costa and A. de Oliveira, Optimal mean-variance control for discrete-time linear systems with Markovian jumps and multiplicative noises, Automatica, 48 (2012), 304-315. doi: 10.1016/j.automatica.2011.11.009.
[7]	K. Du and Q. X. Meng, A maximum principle for optimal control of stochastic evolution equations, SIAM J. Control Optim., 51 (2013), 4343-4362. doi: 10.1137/120882433.
[8]	R. J. Elliott, X. Li and Y. H. Ni, Discrete time mean-field stochastic linear-quadratic optimal control problems, Automatica, 49 (2013), 3222-3233. doi: 10.1016/j.automatica.2013.08.017.
[9]	H. Halkin, A maximum principle of the pontryagin type for systems described by nonlinear difference equations, SIAM J. Control Optim., 4 (1966), 90-111. doi: 10.1137/0304009.
[10]	Y. C. Han, S. G. Peng and Z. Wu, Maximum principle for backward doubly stochastic control systems with applications, SIAM J. Control Optim., 48 (2010), 4224-4241. doi: 10.1137/080743561.
[11]	E. C. M. Hui and H. Xiao, Maximum principle for differential games of forward-backward stochastic systems with applications, J. Math. Anal. Appl., 386 (2012), 412-427. doi: 10.1016/j.jmaa.2011.08.009.
[12]	R. Isaacs, Differential Games, John Wiley and Sons, New York, 1965.
[13]	S. L. Ji and H. D. Liu, Maximum principle for stochastic optimal control problem of forward-backward stochastic difference systems, Int. J. Control, (2021). doi: 10.1080/00207179.2021.1889033.
[14]	X. S. Jiang, S. P. Tian, T. L. Zhang and W. H. Zhang, Stability and stabilization of nonlinear discrete-time stochastic systems, Int. J. Robust Nonlin., 29 (2019), 6419-6437. doi: 10.1002/rnc.4733.
[15]	D. Li and C. W. Schmidt, Cost smoothing in discrete-time linear-quadratic control, Automatica, 33 (1997), 447-452. doi: 10.1016/S0005-1098(96)00171-9.
[16]	X. Y. Lin and W. H. Zhang, A maximum principle for optimal control of discrete-time stochastic systems with multiplicative noise, IEEE Trans. Automat. Control, 60 (2015), 1121-1126. doi: 10.1109/TAC.2014.2345243.
[17]	Q. Lü and X. Zhang, General Pontryagin-type Stochastic Maximum Principle and Backward Stochastic Evolution Equations in Infinite Dimensions, SpringerBriefs in Mathematics. Springer, Cham, 2014. doi: 10.1007/978-3-319-06632-5.
[18]	J. B. Moore, X. Y. Zhou and A. E. B. Lim, Discrete time LQG controls with control dependent noise, Syst. Control Lett., 36 (1999), 199-206. doi: 10.1016/S0167-6911(98)00092-9.
[19]	Y. H. Ni, R. J. Elliott and X. Li, Discrete-time mean-field stochastic linear-quadratic optimal control problems, Ⅱ: Infinite horizon case, Automatica, 57 (2015), 65-77. doi: 10.1016/j.automatica.2015.04.002.
[20]	M. Pachter and K. D. Pham, Discrete-time linear-quadratic dynamic games, J. Optim. Theory Appl., 146 (2010), 151-179. doi: 10.1007/s10957-010-9661-x.
[21]	P. Paruchuri and D. Chatterjee, Discrete time pontryagin maximum principle under state-action-frequency constraints, IEEE Trans. Automat. Control, 64 (2019), 4202-4208. doi: 10.1109/TAC.2019.2893160.
[22]	S. G. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979. doi: 10.1137/0328054.
[23]	M. A. Rami, X. Chen and X. Y. Zhou, Discrete-time indefinite LQ control with state and control dependent noises, J. Global Optim., 23 (2002), 245-265. doi: 10.1023/A:1016578629272.
[24]	H. Y. Sun, L. Y. Jiang and W. H. Zhang, Infinite horizon linear quadratic differential games for discrete-time stochastic systems, J. Optim. Theory Appl., 10 (2012), 391-396. doi: 10.1007/s11768-012-1004-z.
[25]	G. C. Wang and Z. Y. Yu, A Pontryagin's maximum principle for non-zero sum differential games of BSDEs with applications, IEEE Trans. Automat. Control, 55 (2010), 1742-1747. doi: 10.1109/TAC.2010.2048052.
[26]	H. X. Wang, H. S. Zhang and X. Wang, Optimal control for stochastic discrete-time systems with multiple input-delays, in Proc. 10th World Congress on Intelligent Control and Automation, Beijing, (2012), 1529–1534. doi: 10.1109/WCICA.2012.6358121.
[27]	Z. Wu, A general maximum principle for optimal control of forward-backward stochastic systems, Automatica, 49 (2013), 1473-1480. doi: 10.1016/j.automatica.2013.02.005.
[28]	H. S. Zhang and X. Zhang, Second-order necessary conditions for stochastic optimal control problems, SIAM Rev., 60 (2018), 139-178. doi: 10.1137/17M1148773.
[29]	W. H. Zhang, Y. L. Huang and H. S. Zhang, Stochastic $H_2/H_\infty$ control for discrete-time systems with state and disturbance dependent noise, Automatica, 43 (2007), 513-521. doi: 10.1016/j.automatica.2006.09.015.
[30]	X. Zhang, R. J. Elliott and T. K. Siu, A stochastic maximum principle for a Markov regime-switching jump-diffusion model and its application to finance, SIAM J. Control Optim., 50 (2012), 964-990. doi: 10.1137/110839357.