Batch sample-wise stochastic optimal control via stochastic maximum principle

Hui Sun; Feng Bao

doi:10.3934/naco.2026009

Article Contents

2026, Volume 19: 1-34. Doi: 10.3934/naco.2026009

This volume Previous Article Next Article A $ \Sigma $-shaped bifurcation curve for a class of reaction diffusion equations and an application to an ecological model

Batch sample-wise stochastic optimal control via stochastic maximum principle

Hui Sun^1, , and
Feng Bao^{1, 2,}

1.
School of Mathematics & Physics, Xi'an Jiaotong-Liverpool University, Suzhou 215123, China
2.
Department of Mathematics, Florida State University, FL 32306, USA

^*Corresponding author: Hui Sun
^*Corresponding author: Hui Sun

Received: May 4, 2025

Revised: January 19, 2026

Early access: February 3, 2026

Published online: February 3, 2026

The first author is supported by [Research Development Fund (RDF) Xi'an Jiaotong-Liverpool University (XJTLU), No.: RDF-24-02-029].

Abstract / Introduction Full Text(HTML) Figure(5) / Table(2) Related Papers Cited by

Abstract

In this work, we study the stochastic optimal control (SOC) problem mainly from the probabilistic viewpoint, i.e., via the stochastic maximum principle (SMP) [30]. We adopt the sample-wise backpropagation scheme proposed in [1] to solve the SOC problem under the strong convexity assumption. Importantly, in the stochastic gradient descent (SGD) procedure, we use batch samples with a higher-order scheme in the forward SDE to improve the convergence rate in [1] from $ \sim \mathcal{O}(\sqrt{\frac{N}{K} + \frac{1}{N}}) $ to $ \sim \mathcal{O}(\sqrt{\frac{1}{K} + \frac{1}{N^2}}) $, and note that the main source of uncertainty originates from the scheme for the simulation of the $ Z $ term in the BSDE. In the meantime, we note the SGD procedure uses only the necessary condition of the SMP, while the batch simulation of the approximating solution of BSDEs allows one to obtain a more accurate estimate of the control $ u $ that minimizes the Hamiltonian. We then propose a damped contraction algorithm to solve the SOC problem, and a proof of convergence for a special case is attained under appropriate assumptions. We then show numerical results to check the first-order convergence rate of the projection algorithm and analyze the convergence behavior of the damped contraction algorithm. We note that comparisons between the numerical results from these two algorithms (projection and contraction) show the projection approach is still favored in terms of stability and efficiency. Lastly, we briefly discuss how to incorporate the proposed scheme in solving practical problems, especially when randomized neural networks are used. We note that in this special case, the backward propagation error can be avoided, and parameter updates can be achieved via purely algebraic computation (vector algebra), which will potentially improve the efficiency of the whole training procedure. Such ideas will require further exploration, and we will leave this as our future work.

Keywords:

Mathematics Subject Classification: Primary: 60H10, 60H30, 65C20.

Citation:

Full Text(HTML)

Figure 1. Top two figures: Demonstration of numerical solutions using the batch projection algorithm. Bottom two figures: demonstration of error decay by using different number of iterations for $ K $.

Download: Full-size image PowerPoint slide

Figure 2. Top two figures: Demonstration of numerical solutions using the damped contraction algorithm. Bottom two figures: demonstration of error decay by using different number of batch size $ M $. $ x- $axis is on log scale.

Download: Full-size image PowerPoint slide

Figure 3. Top two figures: Demonstration of numerical solutions using the batch projection algorithm. Bottom two figures: demonstration of error decay by using different number of iterations for $ K $. $ x- $axis in log scale.

Download: Full-size image PowerPoint slide

Figure 4. Top two figures: Demonstration of numerical solutions using the damped contraction algorithm. Bottom two figures: demonstration of error decay by using different batch sizes.

Download: Full-size image PowerPoint slide

Figure 5. Top figure: Comparison between numerical solutions obtained via different optimizers. Bottom figure: Comparison between numerical and exact solutions by using the Adam optimizer under different $ \lambda $'s.

Download: Full-size image PowerPoint slide

Table 1. Example 1: Controlled diffusion, efficiency comparison.

Method	$ M $ (Batch size)	$ K $ (Iterations)	$ N $	Time (s)	Relative Error
SGD [1]	1	$ 40^3 $	40	34.1s	0.0037
Batch SGD	$ N $	$ 40^2 $	40	1.03 s	0.0033
Damped Contraction	$ N^2 $	$ 710 $	40	1.96s	0.0035

| Show Table

DownLoad: CSV

Table 2. Example 1: Controlled diffusion, efficiency comparison.

Method	$ M $ (Batch size)	$ K $ (Iterations)	$ N $	Time (s)	Relative Error
SGD	1	$ 40^3 $	40	29.14s	0.00429
Batch SGD	$ N $	$ 40^2 $	40	0.91 s	0.00426
Damped Contraction	$ N^2 $	$ 620 $	40	1.81s	0.00420

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	R. Archibald, F. Bao, Y. Cao and H. Sun, Numerical analysis for convergence of a sample-wise backpropagation method for training stochastic neural networks, SIAM J. Numer. Anal., 62 (2024), 593-621. doi: 10.1137/22M1523765.
[2]	R. Archibald, F. Bao, Y. Cao and H. Zhang, A backward sde method for uncertainty quantification in deep learning, Discrete Contin. Dyn. Syst. Ser. S, 15 (2022), 2807-2835. doi: 10.3934/dcdss.2022062.
[3]	A. Bensoussan, Lecture on stochastic control, in Nonlinear Filtering and Stochastic Control, Lecture Notes in Math., 972, Springer-Verlag, Berlin, New York, 1982, 1-62. doi: 10.1007/BFb0064859.
[4]	F. Biagini, Y. Hu, B. Øksendal and A. Sulem, A stochastic maximum principle for processes driven by fractional Brownian motion, Stochastic Processes Appl., 100 (2002), 233-253. doi: 10.1016/S0304-4149(02)00105-9.
[5]	R. Carmona and M. Laurière, Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games Ⅰ: The ergodic case, SIAM J. Numer. Anal., 59 (2021), 1455-1485. doi: 10.1137/19M1274377.
[6]	R. Carmona and M. Laurière, Convergence analysis of machine learning algorithms for the numerical solution of mean field control and games Ⅱ: The finite horizon case, Ann. Appl. Probab., 32 (2022), 4065-4105. doi: 10.1214/21-AAP1715.
[7]	R. Carmona and M. Laurière, Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance, Cambridge University Press, 2023. doi: 10.1017/9781009028943.021.
[8]	R. Carmona, Lectures on BSDEs, Stochastic Control, and Stochastic Differential Games with Financial Applications, SIAM, 2016.
[9]	D. Chalishajar, D. Kasinathan, R. Kasinathan and R. Kasinathan, Trajectory controllability of higher-order Riemann-Liouville fractional stochastic systems via integral contractors in a new Banach space, Bull. Sci. Math., 204 (2025), 103659. doi: 10.1016/j.bulsci.2025.103659.
[10]	D. Chalishajar, D. Kasinathan, R. Kasinathan and R. Kasinathan, Viscoelastic Kelvin-Voigt model on Ulam-Hyers stability and T-controllability for a coupled integro fractional stochastic systems with integral boundary conditions via integral contractors, Chaos Solitons Fractals, 191 (2025), 115785. doi: 10.1016/j.chaos.2024.115785.
[11]	J. Chessari, R. Kawai, Y. Shinozaki and T. Yamada, Numerical methods for backward stochastic differential equations: A survey, Probab. Surv., 20 (2023), 486-567. doi: 10.1214/23-PS18.
[12]	N. Du, J. T. Shi and W. B. Liu, An effective gradient projection method for stochastic optimal control, Int. J. Numer. Anal. Model., 4 (2013), 757-774.
[13]	W. E, T. Li and E. Vanden-Eijinden, Applied Stochastic Analysis, Graduate Studies in Mathematics, 199, American Mathematical Society, 2019.
[14]	W. E, J. Han and A. Jentzen, Deep learning-based numerical methods for high dimensional parabolic partial differential equations and backward stochastic differential equations, Commun. Math. Stat., 5 (2017), 349-380. doi: 10.1007/s40304-017-0117-6.
[15]	E. Gobet and C. Labart, Error expansion for the discretization of backward stochastic differential equations, Stochastic Processes Appl., 117 (2007), 803-829. doi: 10.1016/j.spa.2006.10.007.
[16]	E. Gobet, J.-P. Lemor and X. Warin, A regression-based Monte Carlo method to solve backward stochastic differential equations, Ann. Appl. Probab., 15 (2005), 2172-2202. doi: 10.1214/105051605000000412.
[17]	B. Gong, W. Liu, T. Tang, W. Zhao and T. Zhou, An efficient gradient projection method for stochastic optimal control problem, SIAM J. Numer. Anal., 55 (2017), 2982-3005. doi: 10.1137/17M1123559.
[18]	Q. Han and S. Ji, A multi-step algorithm for BSDEs based on a predictor-corrector scheme and least-squares Monte Carlo, Methodol. Comput. Appl. Probab., 24 (2022), 2403-2426. doi: 10.1007/s11009-022-09943-4.
[19]	F. B. Hanson, Applied Stochastic Processes and Control for Jump-Diffusions: Modeling, Analysis, and Computation, SIAM, Philadelphia, 2007.
[20]	U. G. Haussmann, Some examples of optimal stochastic controls or: The stochastic maximum principle at work, SIAM Rev., 23 (1981), 292-307. doi: 10.1137/1023062.
[21]	R. Hilfer, Applications of Fractional Calculus in Physics, World Scientific, 2000.
[22]	J. Han and W. E, Deep learning approximation for stochastic control problems, Adv. Neural Inf. Process. Syst., 29 (2016).
[23]	C. Hur'e, H. Pham and X. Warin, Deep backward schemes for high-dimensional nonlinear PDEs, Math. Comp., 89 (2020), 1547-1579. doi: 10.1090/mcom/3514.
[24]	H. J. Kushner, Numerical methods for stochastic control problems in continuous time, SIAM J. Control Optim., 28 (1990), 999-1041. doi: 10.1137/0328056.
[25]	R. Kasinathan, R. Kasinathan, D. Chalishajar et al., Trajectory controllability of Hilfer fractional stochastic pantograph equations with random impulses, Complex Anal. Oper. Theory, 19 (2025), 176. doi: 10.1007/s11785-025-01801-8.
[26]	D. Kasinathan, R. Kasinathan, R. Kasinathan, K. Shanmugasundaram and D. Chalishajar, Trajectory controllability of fractional neutral stochastic dynamical systems of order $\alpha \in (1, 2]$ with deviating argument, J. Appl. Math., (2025), 2539363. doi: 10.1155/jama/2539363.
[27]	J.-P. Lemor, E. Gobet and X. Warin, Rate of convergence of an empirical regression method for solving generalized backward stochastic differential equations, Bernoulli, 12 (2006), 889-916. doi: 10.3150/bj/1161614951.
[28]	M. Min and R. Hu, Signatured deep fictitious play for mean field games with common noise, Proc. Mach. Learn. Res., 139 (2021), 7799-7809.
[29]	S. Peng, Backward stochastic differential equations and applications to optimal control, Appl. Math. Optim., 27 (1993), 125-144. doi: 10.1007/BF01195978.
[30]	S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 28 (1990), 966-979. doi: 10.1137/0328054.
[31]	S. Peng and E. Pardoux, Backward stochastic differential equations and quasilinear parabolic partial differential equations, in Stochastic Partial Differential Equations and Their Applications, Lecture Notes in Control and Information Sci., 176, Springer, Berlin, Heidelberg, 1992,200-217.
[32]	H. Pham, Continuous-time Stochastic Control and Optimization with Financial Applications, Springer, 2009.
[33]	H. Pham and X. Warin, Mean-field neural networks-based algorithms for McKean-Vlasov control problems, preprint, 2022, arXiv: 2212.11518.
[34]	H. Pham and X. Warin, Actor-critic learning algorithms for mean-field control with moment neural networks, preprint, 2023, arXiv: 2309.04317.
[35]	H. M. Soner, J. Teichmann and Q. Yan, Learning algorithms for mean field optimal control, preprint, 2025, arXiv: 2503.17869.
[36]	H. Sun, Meshfree approximation for stochastic optimal control problems, Commun. Math. Res., 37 (2021), 387-420. doi: 10.4208/cmr.2021-0022.
[37]	J. Yong and X. Y. Zhou, Stochastic Controls: Hamiltonian Systems and HJB Equations, Springer, New York, 1999.
[38]	J. Zhang, Backward Stochastic Differential Equations. From Linear to Fully Nonlinear Theory, Probability Theory and Stochastic Modelling, 86, Springer, 2017.
[39]	W. Zhao, L. Chen and S. Peng, A new kind of accurate numerical method for backward stochastic differential equations, SIAM J. Sci. Comput., 28 (2006), 1563-1581. doi: 10.1137/05063341X.