• Previous Article
    Optimal control of the linear wave equation by time-depending BV-controls: A semi-smooth Newton approach
  • MCRF Home
  • This Issue
  • Next Article
    Sparse optimal control for the heat equation with mixed control-state constraints
doi: 10.3934/mcrf.2020027

Stochastic optimal control — A concise introduction

Department of Mathematics, University of Central Florida, Orlando, FL 32816, USA

Received  August 2019 Revised  January 2020 Published  June 2020

Fund Project: This work is supported in part by NSF Grant DMS-1812921

This is a concise introduction to stochastic optimal control theory. We assume that the readers have basic knowledge of real analysis, functional analysis, elementary probability, ordinary differential equations and partial differential equations. We will present the following topics: (ⅰ) A brief presentation of relevant results on stochastic analysis; (ⅱ) Formulation of stochastic optimal control problems; (ⅲ) Variational method and Pontryagin's maximum principle, together with a brief introduction of backward stochastic differential equations; (ⅳ) Dynamic programming method and viscosity solutions to Hamilton-Jacobi-Bellman equation; (ⅴ) Linear-quadratic optimal control problems, including a careful discussion on open-loop optimal controls and closed-loop optimal strategies, linear forward-backward stochastic differential equations, and Riccati equations.

Citation: Jiongmin Yong. Stochastic optimal control — A concise introduction. Mathematical Control & Related Fields, doi: 10.3934/mcrf.2020027
References:
[1]

M. G. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc., 277 (1983), 1-42.  doi: 10.1090/S0002-9947-1983-0690039-8.  Google Scholar

[2] L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, FL, 1992.   Google Scholar
[3]

W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer-Verlag, New York, 1993.  Google Scholar

[4]

A. GaryD. GreenhalghL. HuX. Mao and J. Pan, A stochastic differential equation SIS epidemic model, SIAM J. Appl. Math., 71 (2011), 876-902.  doi: 10.1137/10081856X.  Google Scholar

[5] S. HeJ. Wang and J. Yan, Semimartingale Theory and Stochastic Calculus, Science Press and CRC Press, Beijing, 1992.   Google Scholar
[6]

I. Karatzas and S. Shreve, Brownian Motion and Stochastic Calculus, Springer-Verlag, New York, 1988, 47–127. doi: 10.1007/978-1-4684-0302-2_2.  Google Scholar

[7]

E. Pardoux and S. Peng, Adapted solution of backward stochastic differential equations, Systems Control Lett., 14 (1990), 55-61.  doi: 10.1016/0167-6911(90)90082-6.  Google Scholar

[8]

S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054.  Google Scholar

[9]

J. SunX. Li and J. Yong, Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems, SIAM J. Control Optim., 54 (2016), 2274-2308.  doi: 10.1137/15M103532X.  Google Scholar

[10]

J. Sun and J. Yong, Linear quadratic stochastic differential games: Open-loop and closed-loop saddle points, SIAM J. Control Optim., 52 (2014), 4082-4121.  doi: 10.1137/140953642.  Google Scholar

[11]

E. TornatoreS. M. Buccellato and P. Vetro, Stability of a stochastic SIR system, Physica A, 354 (2005), 111-126.  doi: 10.1016/j.physa.2005.02.057.  Google Scholar

[12]

J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

show all references

References:
[1]

M. G. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc., 277 (1983), 1-42.  doi: 10.1090/S0002-9947-1983-0690039-8.  Google Scholar

[2] L. C. Evans and R. F. Gariepy, Measure Theory and Fine Properties of Functions, CRC Press, Boca Raton, FL, 1992.   Google Scholar
[3]

W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer-Verlag, New York, 1993.  Google Scholar

[4]

A. GaryD. GreenhalghL. HuX. Mao and J. Pan, A stochastic differential equation SIS epidemic model, SIAM J. Appl. Math., 71 (2011), 876-902.  doi: 10.1137/10081856X.  Google Scholar

[5] S. HeJ. Wang and J. Yan, Semimartingale Theory and Stochastic Calculus, Science Press and CRC Press, Beijing, 1992.   Google Scholar
[6]

I. Karatzas and S. Shreve, Brownian Motion and Stochastic Calculus, Springer-Verlag, New York, 1988, 47–127. doi: 10.1007/978-1-4684-0302-2_2.  Google Scholar

[7]

E. Pardoux and S. Peng, Adapted solution of backward stochastic differential equations, Systems Control Lett., 14 (1990), 55-61.  doi: 10.1016/0167-6911(90)90082-6.  Google Scholar

[8]

S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054.  Google Scholar

[9]

J. SunX. Li and J. Yong, Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems, SIAM J. Control Optim., 54 (2016), 2274-2308.  doi: 10.1137/15M103532X.  Google Scholar

[10]

J. Sun and J. Yong, Linear quadratic stochastic differential games: Open-loop and closed-loop saddle points, SIAM J. Control Optim., 52 (2014), 4082-4121.  doi: 10.1137/140953642.  Google Scholar

[11]

E. TornatoreS. M. Buccellato and P. Vetro, Stability of a stochastic SIR system, Physica A, 354 (2005), 111-126.  doi: 10.1016/j.physa.2005.02.057.  Google Scholar

[12]

J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.  Google Scholar

[1]

Steven Richardson, Song Wang. The viscosity approximation to the Hamilton-Jacobi-Bellman equation in optimal feedback control: Upper bounds for extended domains. Journal of Industrial & Management Optimization, 2010, 6 (1) : 161-175. doi: 10.3934/jimo.2010.6.161

[2]

Jean-Claude Zambrini. On the geometry of the Hamilton-Jacobi-Bellman equation. Journal of Geometric Mechanics, 2009, 1 (3) : 369-387. doi: 10.3934/jgm.2009.1.369

[3]

Daniele Castorina, Annalisa Cesaroni, Luca Rossi. On a parabolic Hamilton-Jacobi-Bellman equation degenerating at the boundary. Communications on Pure & Applied Analysis, 2016, 15 (4) : 1251-1263. doi: 10.3934/cpaa.2016.15.1251

[4]

Guy Barles, Ariela Briani, Emmanuel Trélat. Value function for regional control problems via dynamic programming and Pontryagin maximum principle. Mathematical Control & Related Fields, 2018, 8 (3&4) : 509-533. doi: 10.3934/mcrf.2018021

[5]

Joan-Andreu Lázaro-Camí, Juan-Pablo Ortega. The stochastic Hamilton-Jacobi equation. Journal of Geometric Mechanics, 2009, 1 (3) : 295-315. doi: 10.3934/jgm.2009.1.295

[6]

Mohamed Assellaou, Olivier Bokanowski, Hasnaa Zidani. Error estimates for second order Hamilton-Jacobi-Bellman equations. Approximation of probabilistic reachable sets. Discrete & Continuous Dynamical Systems - A, 2015, 35 (9) : 3933-3964. doi: 10.3934/dcds.2015.35.3933

[7]

Nicolas Forcadel, Mamdouh Zaydan. A comparison principle for Hamilton-Jacobi equation with moving in time boundary. Evolution Equations & Control Theory, 2019, 8 (3) : 543-565. doi: 10.3934/eect.2019026

[8]

Tomoki Ohsawa, Anthony M. Bloch. Nonholonomic Hamilton-Jacobi equation and integrability. Journal of Geometric Mechanics, 2009, 1 (4) : 461-481. doi: 10.3934/jgm.2009.1.461

[9]

Nalini Anantharaman, Renato Iturriaga, Pablo Padilla, Héctor Sánchez-Morgado. Physical solutions of the Hamilton-Jacobi equation. Discrete & Continuous Dynamical Systems - B, 2005, 5 (3) : 513-528. doi: 10.3934/dcdsb.2005.5.513

[10]

María Barbero-Liñán, Manuel de León, David Martín de Diego, Juan C. Marrero, Miguel C. Muñoz-Lecanda. Kinematic reduction and the Hamilton-Jacobi equation. Journal of Geometric Mechanics, 2012, 4 (3) : 207-237. doi: 10.3934/jgm.2012.4.207

[11]

Larry M. Bates, Francesco Fassò, Nicola Sansonetto. The Hamilton-Jacobi equation, integrability, and nonholonomic systems. Journal of Geometric Mechanics, 2014, 6 (4) : 441-449. doi: 10.3934/jgm.2014.6.441

[12]

Federica Masiero. Hamilton Jacobi Bellman equations in infinite dimensions with quadratic and superquadratic Hamiltonian. Discrete & Continuous Dynamical Systems - A, 2012, 32 (1) : 223-263. doi: 10.3934/dcds.2012.32.223

[13]

Yan Wang, Yanxiang Zhao, Lei Wang, Aimin Song, Yanping Ma. Stochastic maximum principle for partial information optimal investment and dividend problem of an insurer. Journal of Industrial & Management Optimization, 2018, 14 (2) : 653-671. doi: 10.3934/jimo.2017067

[14]

Shanjian Tang. A second-order maximum principle for singular optimal stochastic controls. Discrete & Continuous Dynamical Systems - B, 2010, 14 (4) : 1581-1599. doi: 10.3934/dcdsb.2010.14.1581

[15]

Shaolin Ji, Xiaole Xue. A stochastic maximum principle for linear quadratic problem with nonconvex control domain. Mathematical Control & Related Fields, 2019, 9 (3) : 495-507. doi: 10.3934/mcrf.2019022

[16]

Huaiqiang Yu, Bin Liu. Pontryagin's principle for local solutions of optimal control governed by the 2D Navier-Stokes equations with mixed control-state constraints. Mathematical Control & Related Fields, 2012, 2 (1) : 61-80. doi: 10.3934/mcrf.2012.2.61

[17]

Zaidong Zhan, Shuping Chen, Wei Wei. A unified theory of maximum principle for continuous and discrete time optimal control problems. Mathematical Control & Related Fields, 2012, 2 (2) : 195-215. doi: 10.3934/mcrf.2012.2.195

[18]

Rein Luus. Optimal control of oscillatory systems by iterative dynamic programming. Journal of Industrial & Management Optimization, 2008, 4 (1) : 1-15. doi: 10.3934/jimo.2008.4.1

[19]

Diana Keller. Optimal control of a linear stochastic Schrödinger equation. Conference Publications, 2013, 2013 (special) : 437-446. doi: 10.3934/proc.2013.2013.437

[20]

Fulvia Confortola, Elisa Mastrogiacomo. Optimal control for stochastic heat equation with memory. Evolution Equations & Control Theory, 2014, 3 (1) : 35-58. doi: 10.3934/eect.2014.3.35

2019 Impact Factor: 0.857

Article outline

[Back to Top]