# American Institute of Mathematical Sciences

September  2020, 10(3): 493-526. doi: 10.3934/mcrf.2020008

## Optimality conditions in variational form for non-linear constrained stochastic control problems

 Institute of Mathematics and Scientific Computing, University of Graz, Heinrichstrasse 36, 8010 Graz, Austria

Dedicated to Prof. Dr. Fréderic Bonnans on the occasion of his 60th birthday

Received  February 2018 Revised  December 2018 Published  September 2020 Early access  December 2019

Optimality conditions in the form of a variational inequality are proved for a class of constrained optimal control problems of stochastic differential equations. The cost function and the inequality constraints are functions of the probability distribution of the state variable at the final time. The analysis uses in an essential manner a convexity property of the set of reachable probability distributions. An augmented Lagrangian method based on the obtained optimality conditions is proposed and analyzed for solving iteratively the problem. At each iteration of the method, a standard stochastic optimal control problem is solved by dynamic programming. Two academical examples are investigated.

Citation: Laurent Pfeiffer. Optimality conditions in variational form for non-linear constrained stochastic control problems. Mathematical Control and Related Fields, 2020, 10 (3) : 493-526. doi: 10.3934/mcrf.2020008
##### References:
 [1] Y. Achdou and M. Laurière, On the system of partial differential equations arising in mean field type control, Discrete and Continuous Dynamical Systems, 35 (2015), 3879-3900.  doi: 10.3934/dcds.2015.35.3879. [2] G. Albi, Y.-P. Choi, M. Fornasier and D. Kalise, Mean field control hierarchy, Appl. Math. Optim., 76 (2017), 93-135.  doi: 10.1007/s00245-017-9429-x. [3] D. Andersson and B. Djehiche, A maximum principle for SDEs of mean-field type, Appl. Math. Optim., 63 (2011), 341-356.  doi: 10.1007/s00245-010-9123-8. [4] A. Bensoussan, J. Frehse and P. Yam, Mean Field Games and Mean Field Type Control Theory, SpringerBriefs in Mathematics, Springer, New York, 2013. doi: 10.1007/978-1-4614-8508-7. [5] J. F. Bonnans, L. Pfeiffer and O. S. Serea, Sensitivity analysis for relaxed optimal control problems with final-state constraints, Nonlinear Anal., 89 (2013), 55-80.  doi: 10.1016/j.na.2013.04.013. [6] J. F. Bonnans and A. Shapiro, Perturbation Analysis of Optimization Problems, Springer Series in Operations Research, Springer-Verlag, New York, 2000. doi: 10.1007/978-1-4612-1394-9. [7] J. F. Bonnans and F. J. Silva., First and second order necessary conditions for stochastic optimal control problems, Appl. Math. Optim., 65 (2012), 403-439.  doi: 10.1007/s00245-012-9162-4. [8] B. Bouchard, R. Elie and N. Touzi, Stochastic target problems with controlled loss, SIAM J. Control Optim., 48 (2009/10), 3123-3150.  doi: 10.1137/08073593X. [9] R. Buckdahn, B. Djehiche and J. Li, A general stochastic maximum principle for SDEs of mean-field type, Appl. Math. Optim., 64 (2011), 197-216.  doi: 10.1007/s00245-011-9136-y. [10] P. H. Calamai and J. J. Moré, Projected gradient methods for linearly constrained problems, Mathematical Programming, 39 (1987), 93-116.  doi: 10.1007/BF02592073. [11] F. Camilli and M. Falcone, An approximation scheme for the optimal control of diffusion processes, RAIRO Modél. Math. Anal. Numér., 29 (1995), 97-122.  doi: 10.1051/m2an/1995290100971. [12] P. Cardaliaguet, Notes on Mean Field Games, 2012. [13] R. Carmona and F. Delarue, Forward-backward stochastic differential equations and controlled McKean-Vlasov dynamics, Ann. Probab., 43 (2015), 2647-2700.  doi: 10.1214/14-AOP946. [14] A. R. Conn, N. I. M. Gould and P. L. Toint, LANCELOT. A Fortran Package for Large-Scale Nonlinear Optimization (Release A), Springer Series in Computational Mathematics, 17. Springer-Verlag, Berlin, 1992. doi: 10.1007/978-3-662-12211-2. [15] A. Fleig and R. Guglielmi, Optimal control of the Fokker-Planck equation with space-dependent controls, Journal of Optimization Theory and Applications, 174 (2017), 408-427.  doi: 10.1007/s10957-017-1120-5. [16] W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Second edition, Stochastic Modelling and Applied Probability, 25. Springer, New York, 2006. [17] N. V. Krylov, Controlled Diffusion Processes, Applications of Mathematics, 14. Springer-Verlag, New York-Berlin, 1980. [18] M. Laurière and O. Pironneau, Dynamic programming for mean-field type control, Journal of Optimization Theory and Applications, 169 (2016), 902-924.  doi: 10.1007/s10957-015-0785-x. [19] C. W. Miller and I. Yang, Optimal control of conditional value-at-risk in continuous time, SIAM J. Control Optim., 55 (2017), 856-884.  doi: 10.1137/16M1058492. [20] J. Nocedal and S. J. Wright, Numerical Optimization, Second edition, Springer Series in Operations Research and Financial Engineering. Springer, New York, 2006. [21] B. Øksendal, Stochastic Differential Equations: An Introduction with Applications, Hochschultext / Universitext. Springer, 2003. [22] J. L. Pedersen and G. Peskir, Optimal mean-variance portfolio selection, Math. Financ. Econ., 11 (2017), 137-160.  doi: 10.1007/s11579-016-0174-8. [23] S. G. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054. [24] L. Pfeiffer, Optimality conditions for mean-field type optimal control problems, SFB-report 2015-015, (2015). [25] L. Pfeiffer, Numerical methods for mean-field type optimal control problems, Pure and Applied Functional Analysis, 1 (2016), 629-655. [26] L. Pfeiffer, Risk-averse Merton's portfolio problem, IFAC-PapersOnLine, 49 (2016), 266-271.  doi: 10.1016/j.ifacol.2016.07.452. [27] L. Pfeiffer, Two approaches to stochastic optimal control problems with a final time expectation constraint, Appl. Math. Optim., 77 (2018), 377-404.  doi: 10.1007/s00245-016-9378-9. [28] H. Pham, Continuous-Time Stochastic Control and Optimization with Financial Applications, Stochastic Modelling and Applied Probability, 61. Springer-Verlag, Berlin, 2009. doi: 10.1007/978-3-540-89500-8. [29] H. Pham and X. L. Wei, Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics, SIAM Journal on Control and Optimization, 55 (2017), 1069-1101.  doi: 10.1137/16M1071390. [30] H. Pham and X. L. Wei, Bellman equation and viscosity solutions for mean-field stochastic control problem, ESAIM Control Optim. Calc. Var., 24 (2018), 437-461.  doi: 10.1051/cocv/2017019. [31] A. Shapiro, D. Dentcheva and A. Ruszczyński, Lectures on Stochastic Programming. Modeling and Theory, Second edition, MOS-SIAM Series on Optimization, 9. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, Mathematical Optimization Society, Philadelphia, PA, 2014. [32] X. L. Tan and N. Touzi, Optimal transportation under controlled stochastic dynamics, Ann. Probab., 41 (2013), 3201-3240.  doi: 10.1214/12-AOP797. [33] N. Touzi, Direct characterization of the value of super-replication under stochastic volatility and portfolio constraints, Stochastic Process. Appl., 88 (2000), 305-328.  doi: 10.1016/S0304-4149(00)00007-7. [34] C. Villani, Optimal Transport. Old and New, Grundlehren der Mathematischen Wissenschaften, 338. Springer-Verlag, Berlin, 2009. doi: 10.1007/978-3-540-71050-9. [35] J. M. Yong and X. Y. Zhou., Stochastic Controls: Hamiltonian Systems and HJB Equations, Applications of Mathematics (New York), 43. Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.

show all references

##### References:
 [1] Y. Achdou and M. Laurière, On the system of partial differential equations arising in mean field type control, Discrete and Continuous Dynamical Systems, 35 (2015), 3879-3900.  doi: 10.3934/dcds.2015.35.3879. [2] G. Albi, Y.-P. Choi, M. Fornasier and D. Kalise, Mean field control hierarchy, Appl. Math. Optim., 76 (2017), 93-135.  doi: 10.1007/s00245-017-9429-x. [3] D. Andersson and B. Djehiche, A maximum principle for SDEs of mean-field type, Appl. Math. Optim., 63 (2011), 341-356.  doi: 10.1007/s00245-010-9123-8. [4] A. Bensoussan, J. Frehse and P. Yam, Mean Field Games and Mean Field Type Control Theory, SpringerBriefs in Mathematics, Springer, New York, 2013. doi: 10.1007/978-1-4614-8508-7. [5] J. F. Bonnans, L. Pfeiffer and O. S. Serea, Sensitivity analysis for relaxed optimal control problems with final-state constraints, Nonlinear Anal., 89 (2013), 55-80.  doi: 10.1016/j.na.2013.04.013. [6] J. F. Bonnans and A. Shapiro, Perturbation Analysis of Optimization Problems, Springer Series in Operations Research, Springer-Verlag, New York, 2000. doi: 10.1007/978-1-4612-1394-9. [7] J. F. Bonnans and F. J. Silva., First and second order necessary conditions for stochastic optimal control problems, Appl. Math. Optim., 65 (2012), 403-439.  doi: 10.1007/s00245-012-9162-4. [8] B. Bouchard, R. Elie and N. Touzi, Stochastic target problems with controlled loss, SIAM J. Control Optim., 48 (2009/10), 3123-3150.  doi: 10.1137/08073593X. [9] R. Buckdahn, B. Djehiche and J. Li, A general stochastic maximum principle for SDEs of mean-field type, Appl. Math. Optim., 64 (2011), 197-216.  doi: 10.1007/s00245-011-9136-y. [10] P. H. Calamai and J. J. Moré, Projected gradient methods for linearly constrained problems, Mathematical Programming, 39 (1987), 93-116.  doi: 10.1007/BF02592073. [11] F. Camilli and M. Falcone, An approximation scheme for the optimal control of diffusion processes, RAIRO Modél. Math. Anal. Numér., 29 (1995), 97-122.  doi: 10.1051/m2an/1995290100971. [12] P. Cardaliaguet, Notes on Mean Field Games, 2012. [13] R. Carmona and F. Delarue, Forward-backward stochastic differential equations and controlled McKean-Vlasov dynamics, Ann. Probab., 43 (2015), 2647-2700.  doi: 10.1214/14-AOP946. [14] A. R. Conn, N. I. M. Gould and P. L. Toint, LANCELOT. A Fortran Package for Large-Scale Nonlinear Optimization (Release A), Springer Series in Computational Mathematics, 17. Springer-Verlag, Berlin, 1992. doi: 10.1007/978-3-662-12211-2. [15] A. Fleig and R. Guglielmi, Optimal control of the Fokker-Planck equation with space-dependent controls, Journal of Optimization Theory and Applications, 174 (2017), 408-427.  doi: 10.1007/s10957-017-1120-5. [16] W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Second edition, Stochastic Modelling and Applied Probability, 25. Springer, New York, 2006. [17] N. V. Krylov, Controlled Diffusion Processes, Applications of Mathematics, 14. Springer-Verlag, New York-Berlin, 1980. [18] M. Laurière and O. Pironneau, Dynamic programming for mean-field type control, Journal of Optimization Theory and Applications, 169 (2016), 902-924.  doi: 10.1007/s10957-015-0785-x. [19] C. W. Miller and I. Yang, Optimal control of conditional value-at-risk in continuous time, SIAM J. Control Optim., 55 (2017), 856-884.  doi: 10.1137/16M1058492. [20] J. Nocedal and S. J. Wright, Numerical Optimization, Second edition, Springer Series in Operations Research and Financial Engineering. Springer, New York, 2006. [21] B. Øksendal, Stochastic Differential Equations: An Introduction with Applications, Hochschultext / Universitext. Springer, 2003. [22] J. L. Pedersen and G. Peskir, Optimal mean-variance portfolio selection, Math. Financ. Econ., 11 (2017), 137-160.  doi: 10.1007/s11579-016-0174-8. [23] S. G. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979.  doi: 10.1137/0328054. [24] L. Pfeiffer, Optimality conditions for mean-field type optimal control problems, SFB-report 2015-015, (2015). [25] L. Pfeiffer, Numerical methods for mean-field type optimal control problems, Pure and Applied Functional Analysis, 1 (2016), 629-655. [26] L. Pfeiffer, Risk-averse Merton's portfolio problem, IFAC-PapersOnLine, 49 (2016), 266-271.  doi: 10.1016/j.ifacol.2016.07.452. [27] L. Pfeiffer, Two approaches to stochastic optimal control problems with a final time expectation constraint, Appl. Math. Optim., 77 (2018), 377-404.  doi: 10.1007/s00245-016-9378-9. [28] H. Pham, Continuous-Time Stochastic Control and Optimization with Financial Applications, Stochastic Modelling and Applied Probability, 61. Springer-Verlag, Berlin, 2009. doi: 10.1007/978-3-540-89500-8. [29] H. Pham and X. L. Wei, Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics, SIAM Journal on Control and Optimization, 55 (2017), 1069-1101.  doi: 10.1137/16M1071390. [30] H. Pham and X. L. Wei, Bellman equation and viscosity solutions for mean-field stochastic control problem, ESAIM Control Optim. Calc. Var., 24 (2018), 437-461.  doi: 10.1051/cocv/2017019. [31] A. Shapiro, D. Dentcheva and A. Ruszczyński, Lectures on Stochastic Programming. Modeling and Theory, Second edition, MOS-SIAM Series on Optimization, 9. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, Mathematical Optimization Society, Philadelphia, PA, 2014. [32] X. L. Tan and N. Touzi, Optimal transportation under controlled stochastic dynamics, Ann. Probab., 41 (2013), 3201-3240.  doi: 10.1214/12-AOP797. [33] N. Touzi, Direct characterization of the value of super-replication under stochastic volatility and portfolio constraints, Stochastic Process. Appl., 88 (2000), 305-328.  doi: 10.1016/S0304-4149(00)00007-7. [34] C. Villani, Optimal Transport. Old and New, Grundlehren der Mathematischen Wissenschaften, 338. Springer-Verlag, Berlin, 2009. doi: 10.1007/978-3-540-71050-9. [35] J. M. Yong and X. Y. Zhou., Stochastic Controls: Hamiltonian Systems and HJB Equations, Applications of Mathematics (New York), 43. Springer-Verlag, New York, 1999. doi: 10.1007/978-1-4612-1466-3.
Convergence results for the Test Case 1
Numerical results
Convergence results for the Test Case 2
 [1] Heinz Schättler, Urszula Ledzewicz, Helmut Maurer. Sufficient conditions for strong local optimality in optimal control problems with $L_{2}$-type objectives and control constraints. Discrete and Continuous Dynamical Systems - B, 2014, 19 (8) : 2657-2679. doi: 10.3934/dcdsb.2014.19.2657 [2] Salah Eddine Choutri, Boualem Djehiche, Hamidou Tembine. Optimal control and zero-sum games for Markov chains of mean-field type. Mathematical Control and Related Fields, 2019, 9 (3) : 571-605. doi: 10.3934/mcrf.2019026 [3] Jianhui Huang, Xun Li, Jiongmin Yong. A linear-quadratic optimal control problem for mean-field stochastic differential equations in infinite horizon. Mathematical Control and Related Fields, 2015, 5 (1) : 97-139. doi: 10.3934/mcrf.2015.5.97 [4] Jingrui Sun, Hanxiao Wang. Mean-field stochastic linear-quadratic optimal control problems: Weak closed-loop solvability. Mathematical Control and Related Fields, 2021, 11 (1) : 47-71. doi: 10.3934/mcrf.2020026 [5] Xun Li, Jingrui Sun, Jiongmin Yong. Mean-field stochastic linear quadratic optimal control problems: closed-loop solvability. Probability, Uncertainty and Quantitative Risk, 2016, 1 (0) : 2-. doi: 10.1186/s41546-016-0002-3 [6] Lucas Bonifacius, Ira Neitzel. Second order optimality conditions for optimal control of quasilinear parabolic equations. Mathematical Control and Related Fields, 2018, 8 (1) : 1-34. doi: 10.3934/mcrf.2018001 [7] Sofia O. Lopes, Fernando A. C. C. Fontes, Maria do Rosário de Pinho. On constraint qualifications for nondegenerate necessary conditions of optimality applied to optimal control problems. Discrete and Continuous Dynamical Systems, 2011, 29 (2) : 559-575. doi: 10.3934/dcds.2011.29.559 [8] Wei Zhu, Xue-Cheng Tai, Tony Chan. Augmented Lagrangian method for a mean curvature based image denoising model. Inverse Problems and Imaging, 2013, 7 (4) : 1409-1432. doi: 10.3934/ipi.2013.7.1409 [9] René Carmona, Kenza Hamidouche, Mathieu Laurière, Zongjun Tan. Linear-quadratic zero-sum mean-field type games: Optimality conditions and policy optimization. Journal of Dynamics and Games, 2021, 8 (4) : 403-443. doi: 10.3934/jdg.2021023 [10] Adel Chala, Dahbia Hafayed. On stochastic maximum principle for risk-sensitive of fully coupled forward-backward stochastic control of mean-field type with application. Evolution Equations and Control Theory, 2020, 9 (3) : 817-843. doi: 10.3934/eect.2020035 [11] Jianxiong Ye, An Li. Necessary optimality conditions for nonautonomous optimal control problems and its applications to bilevel optimal control. Journal of Industrial and Management Optimization, 2019, 15 (3) : 1399-1419. doi: 10.3934/jimo.2018101 [12] Yves Achdou, Mathieu Laurière. On the system of partial differential equations arising in mean field type control. Discrete and Continuous Dynamical Systems, 2015, 35 (9) : 3879-3900. doi: 10.3934/dcds.2015.35.3879 [13] Kehan Si, Zhenda Xu, Ka Fai Cedric Yiu, Xun Li. Open-loop solvability for mean-field stochastic linear quadratic optimal control problems of Markov regime-switching system. Journal of Industrial and Management Optimization, 2021  doi: 10.3934/jimo.2021074 [14] Shahlar F. Maharramov. Necessary optimality conditions for switching control problems. Journal of Industrial and Management Optimization, 2010, 6 (1) : 47-55. doi: 10.3934/jimo.2010.6.47 [15] Ciro D'Apice, Olha P. Kupenko, Rosanna Manzo. On boundary optimal control problem for an arterial system: First-order optimality conditions. Networks and Heterogeneous Media, 2018, 13 (4) : 585-607. doi: 10.3934/nhm.2018027 [16] Thierry Horsin, Peter I. Kogut, Olivier Wilk. Optimal $L^2$-control problem in coefficients for a linear elliptic equation. II. Approximation of solutions and optimality conditions. Mathematical Control and Related Fields, 2016, 6 (4) : 595-628. doi: 10.3934/mcrf.2016017 [17] Vladimir Gaitsgory, Alex Parkinson, Ilya Shvartsman. Linear programming based optimality conditions and approximate solution of a deterministic infinite horizon discounted optimal control problem in discrete time. Discrete and Continuous Dynamical Systems - B, 2019, 24 (4) : 1743-1767. doi: 10.3934/dcdsb.2018235 [18] Wei Zhu. A numerical study of a mean curvature denoising model using a novel augmented Lagrangian method. Inverse Problems and Imaging, 2017, 11 (6) : 975-996. doi: 10.3934/ipi.2017045 [19] EL Hassene Osmani, Mounir Haddou, Naceurdine Bensalem. A new relaxation method for optimal control of semilinear elliptic variational inequalities obstacle problems. Numerical Algebra, Control and Optimization, 2021  doi: 10.3934/naco.2021061 [20] Michael Herty, Lorenzo Pareschi, Sonja Steffensen. Mean--field control and Riccati equations. Networks and Heterogeneous Media, 2015, 10 (3) : 699-715. doi: 10.3934/nhm.2015.10.699

2020 Impact Factor: 1.284