Assimilating partial observation to enhance feedback control of stochastic dynamical systems

Siming Liang; Ruoyu Hu; Feng Bao; Richard Archibald; Guannan Zhang

doi:10.3934/fods.2025010

Article Contents

2026, Volume 9: 1-33. Doi: 10.3934/fods.2025010

This volume Previous Article Confidence regions for a persistence diagram of a single image with one or more loops Next Article Nested smoothing algorithms for inference and tracking of heterogeneous multi-scale state-space systems

Assimilating partial observation to enhance feedback control of stochastic dynamical systems

1.
Department of Mathematics, Florida State University, Tallahassee, 32306 Florida, USA
2.
Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, 37831 Tennessee, USA

^*Corresponding author: Siming Liang
^*Corresponding author: Siming Liang

Received: February 2025

Revised: May 2025

Early access: June 19, 2025

Published online: June 19, 2025

Abstract / Introduction Full Text(HTML) Figure(11) Related Papers Cited by

Abstract

In this paper, we present a novel methodology to tackle feedback optimal control problems in scenarios where the exact state of the controlled process is unknown. It integrates data assimilation techniques and optimal control solvers to manage partial observation of the state process, a common occurrence in practical scenarios. Traditional stochastic optimal control methods assume full state observation, which is often not feasible in real-world fluid dynamics control problems. Our approach underscores the significance of utilizing observational data to inform control policy design. Specifically, we introduce a kernel learning backward stochastic differential equation (SDE) filter to enhance data assimilation efficiency and propose a sample-wise stochastic optimization method within the stochastic maximum principle framework. We demonstrate the efficacy and accuracy of our method in the control of advection-diffusion-reaction flow problem and the Dubins airplane maneuvering problem with model uncertainty.

Keywords:

Mathematics Subject Classification: Primary: 93E11, 49N35.

Citation:

Full Text(HTML)

Figure 1. Comparison of the estimated control and the analytical optimal control. The x-axis represents time, and the y-axis represents the control action value.

Download: Full-size image PowerPoint slide

Figure 2. Comparison of the estimated trajectory and the true trajectory. The x-axis represents time, and the y-axis represents the position of the state variable.

Download: Full-size image PowerPoint slide

Figure 3. Log Error of BSDE-SGD Estimation on Control u and State X.

Download: Full-size image PowerPoint slide

Figure 4. Comparison of the estimated control with the analytical optimal control. The x-axis shows the discrete spatial partitions, and the y-axis displays the corresponding control action values.

Download: Full-size image PowerPoint slide

Figure 5. Comparison of the estimated heat distribution with the analytical solution. The x-axis represents time, while the y-axis indicates the corresponding heat energy values.

Download: Full-size image PowerPoint slide

Figure 6. BSDE-SGD Estimate VS Designed trajectory.

Download: Full-size image PowerPoint slide

Figure 7. BSDE-SGD Estimate VS Designed trajectory in Each Dimension (X, Y, Z from left to right).

Download: Full-size image PowerPoint slide

Figure 8. PF-SGD Estimate VS Designed trajectory in Each Dimension (X, Y, Z from left to right).

Download: Full-size image PowerPoint slide

Figure 9. BSDE-SGD Estimate VS Designed trajectory in Each Dimension (X, Y, Z from left to right).

Download: Full-size image PowerPoint slide

Figure 10. BSDE-SGD Estimate VS DP Estimate.

Download: Full-size image PowerPoint slide

Figure 11. DP Estimate VS BSDE-SGD Estimate in Each Dimension (X, Y, Z from left to right).

Download: Full-size image PowerPoint slide

Related Papers

Cited by

References

[1]	C. Andrieu, A. Doucet and R. Holenstein, Particle markov chain monte carlo methods, J. R. Statist. Soc. B, 72 (2010), 269-342. doi: 10.1111/j.1467-9868.2009.00736.x.
[2]	R. Archibald, F. Bao, Y. Cao and H. Sun, Numerical analysis for convergence of a sample-wise backpropagation method for training stochastic neural networks, SIAM Journal on Numerical Analysis, 62 (2024), 593-621. doi: 10.1137/22M1523765.
[3]	R. Archibald, F. Bao, J. Yong and T. Zhou, An efficient numerical algorithm for solving data driven feedback control problems, Journal of Scientific Computing, 85 (2020). doi: 10.1007/s10915-020-01358-y.
[4]	R. Archibald, F. Bao and J. Yong, A stochastic maximum principle approach for reinforcement learning with parameterized environment, Journal of Computational Physics, 488 (2023), 212-238. doi: 10.1016/j.jcp.2023.112238.
[5]	R. Archibald, F. Bao, J. Yong and T. Zhou, An efficient numerical algorithm for solving data driven feedback control problems, Journal of Scientific Computing, 85 (2020), 1-27. doi: 10.1007/s10915-020-01358-y.
[6]	F. Bao, Y. Cao, A. Meir and W. Zhao, A first order scheme for backward doubly stochastic differential equations, SIAM/ASA Journal on Uncertainty Quantification, 4 (2016), 413-445. doi: 10.1137/14095546X.
[7]	F. Bao, Y. Cao, C. Webster and G. Zhang, A hybrid sparse-grid approach for nonlinear filtering problems based on adaptive-domain of the Zakai equation approximations, SIAM/ASA J. Uncertain. Quantif., 2 (2014), 784-804. doi: 10.1137/140952910.
[8]	F. Bao, Y. Cao and W. Zhao, Numerical solutions for forward backward doubly stochastic differential equations and zakai equations, Visualization of Mechanical Processes: An International Online Journal, 1.
[9]	F. Bao and V. Maroulas, Adaptive meshfree backward SDE filter, SIAM J. Sci. Comput0, 39 (2017), A2664-A2683. doi: 10.1137/16M1100277.
[10]	J. S. Baras, R. J. Elliott and M. Kohlmann, The partially observed stochastic minimum principle, SIAM Journal on Control and Optimization, 27 (1989), 1279-1292. doi: 10.1137/0327065.
[11]	R. Bellman, Dynamic Programming, Princeton Univ. Press, 1957.
[12]	A. Bensoussan, Maximum principle and dynamic programming approaches of the optimal control of partially observed diffusions, Stochastics: An International Journal of Probability and Stochastic Processes, 9 (1983), 169-222. doi: 10.1080/17442508308833253.
[13]	A. Bensoussan, Stochastic control of partially observable systems, Cambridge University Press, Cambridge, .
[14]	A. Bensoussan and S. C. P. Yam, Mean field approach to stochastic control with partial information, ESAIM: Control, Optimisation and Calculus of Variations, 27 (2021), 89. doi: 10.1051/cocv/2021085.
[15]	L. Bottou and O. Bousquet, The tradeoffs of large scale learning, Advances in neural information processing systems, 20.
[16]	C. D. Charalambous and R. J. Elliott, Classes of nonlinear partially observable stochastic optimal control problems with explicit optimal control laws, SIAM J. Control Optim., 36 (1998), 542-578. doi: 10.1137/S0363012995287326.
[17]	A. J. Chorin and X. Tu, Implicit sampling for particle filters, Proc. Nat. Acad. Sc. USA, 106 (2009), 17249-17254. doi: 10.1073/pnas.0909196106.
[18]	A. Doucet, N. de Freitas and N. Gordon, Sequential Monte Carlo Methods in Practice, Springer, New York, 2001.
[19]	G. Evensen, The ensemble Kalman filter for combined state and parameter estimation: Monte Carlo techniques for data assimilation in large systems, IEEE Control Syst. Mag., 29 (2009), 83-104. doi: 10.1109/MCS.2009.932223.
[20]	X. Feng, R. Glowinski and M. Neilan, Recent developments in numerical methods for fully nonlinear second order partial differential equations, SIAM Rev., 55 (2013), 205-267. doi: 10.1137/110825960.
[21]	W. H. Fleming, Optimal control of partially observable diffusions, SIAM Journal on Control, 6 (1968), 194-214. doi: 10.1137/0306015.
[22]	W. H. Fleming and E. Pardoux, Optimal control for partially observed diffusions, SIAM J. Control Optim., 20 (1982), 261-285. doi: 10.1137/0320021.
[23]	W. H. Fleming and É. Pardoux, Optimal control for partially observed diffusions, SIAM Journal on Control and Optimization, 20 (1982), 261-285. doi: 10.1137/0320021.
[24]	B. Gong, W. Liu, T. Tang, W. Zhao and T. Zhou, An efficient gradient projection method for stochastic optimal control problems, SIAM J. Numer. Anal., 55 (2017), 2982-3005. doi: 10.1137/17M1123559.
[25]	N. Gordon, D. Salmond and A. Smith, Novel approach to nonlinear/non-gaussian bayesian state estimation, IEE PROCEEDING-F, 140 (1993), 107-113. doi: 10.1049/ip-f-2.1993.0015.
[26]	U. G. Haussmann, On the existence of optimal controls for partially observed diffusions, SIAM J. Control Optim., 20 (1982), 385-407. doi: 10.1137/0320030.
[27]	U. Haussmann, On the existence of optimal controls for partially observed diffusions, SIAM Journal on Control and Optimization, 20 (1982), 385-407. doi: 10.1137/0320030.
[28]	U. Haussmann, The maximum principle for optimal control of diffusions with partial information, SIAM Journal on Control and Optimization, 25 (1987), 341-361. doi: 10.1137/0325021.
[29]	T. Hofmann, B. Schölkopf and A. J. Smola, Kernel methods in machine learning, The Annals of Statistics, 36 (2008), 1171-1220. doi: 10.1214/009053607000000677.
[30]	P. E. Kloeden and E. Platen, Stochastic differential equations, Springer Berlin Heidelberg, Berlin, Heidelberg, (1992), 103-160. doi: 10.1007/978-3-662-12616-5_4.
[31]	T. Kong, W. Zhao and T. Zhou, High order numerical schemes for second order fbsdes with applications to stochastic optimal control, Communications in Computational Physics, 21 (2017), 808-834. doi: 10.4208/cicp.OA-2016-0056.
[32]	H. Kwakernaak, A minimum principle for stochastic control problems with output feedback, Systems and Control Letters, 1 (1981), 74-77. doi: 10.1016/S0167-6911(81)80017-5.
[33]	Y. Li, X. Tan and S. Tang, Discrete-time approximation of stochastic optimal control with partial observation, SIAM Journal on Control and Optimization, 62 (2024), 326-350. doi: 10.1137/23M1549018.
[34]	X. Li and S. Tang, General necessary conditions for partially observed optimal stochastic controls, Journal of Applied Probability, 32 (1995), 1118-1137. doi: 10.2307/3215225.
[35]	É. Pardoux and S. Peng, Backward doubly stochastic differential equations and systems of quasilinear spdes, Probability Theory and Related Fields, 98 (1994), 209-227. doi: 10.1007/BF01192514.
[36]	E. Pardoux and S. Peng, Backward stochastic differential equations and quasilinear parabolic partial differential equations, Stochastic Partial Differential Equations and Their Applications: Proceedings of IFIP WG 7/1 International Conference University of North Carolina at Charlotte, Springer, (2005), 200-217. doi: 10.1007/BFb0007334.
[37]	S. Peng, A general stochastic maximum principle for optimal control problems, SIAM J. Control Optim., 966-979.
[38]	S. Peng, Backward stochastic differential equation, nonlinear expectation and their applications, Proceedings of the International Congress of Mathematicians 2010 (ICM 2010) (In 4 Volumes) Vol. Ⅰ: Plenary Lectures and Ceremonies Vols. Ⅱ–Ⅳ: Invited Lectures, World Scientific, (2010), 393-432.
[39]	M. K. Pitt and N. Shephard, Filtering via simulation: Auxiliary particle filters, J. Amer. Statist. Assoc., 94 (1999), 590-599. doi: 10.1080/01621459.1999.10474153.
[40]	H. Robbins and S. Monro, A stochastic approximation method, The Annals of Mathematical Statistics, 400-407. doi: 10.1214/aoms/1177729586.
[41]	I. Sato and H. Nakagawa, Approximation analysis of stochastic gradient langevin dynamics by using fokker-planck equation and ito process, International Conference on Machine Learning, PMLR, (2014), 982-990.
[42]	A. Shapiro and Y. Wardi, Convergence analysis of gradient descent stochastic algorithms, Journal of Optimization Theory and Applications, 91 (1996), 439-454. doi: 10.1007/BF02190104.
[43]	C. Snyder, T. Bengtsson, P. Bickel and J. Anderson, Obstacles to high-dimensional particle filtering, Mon. Wea. Rev., 136 (2008), 4629-4640. doi: 10.1175/2008MWR2529.1.
[44]	J. Sun and J. Xiong, Stochastic linear-quadratic optimal control with partial observation, SIAM Journal on Control and Optimization, 61 (2023), 1231-1247. doi: 10.1137/21M146586X.
[45]	S. Tang, The maximum principle for partially observed optimal control of stochastic differential equations, SIAM J. Control Optim., 36 (1998), 1596-1617. doi: 10.1137/S0363012996313100.
[46]	X. T. Tong, A. J. Majda and D. Kelly, Nonlinear stability and ergodicity of ensemble based Kalman filters, Nonlinearity, 29 (2016), 657-691. doi: 10.1088/0951-7715/29/2/657.
[47]	H. Wan, G. Wang and J. Xiong, A branching particle system approximation for solving partially observed stochastic optimal control problems via stochastic maximum principle, Stochastics and Partial Differential Equations: Analysis and Computations, 12 (2024), 675-735. doi: 10.1007/s40072-023-00294-w.
[48]	G. Wang, Z. Wu and J. Xiong, An Introduction to Optimal Control of FBSDE with Incomplete Information, SpringerBriefs in Mathematics, Springer, Cham, 2018. doi: 10.1007/978-3-319-79039-8.
[49]	G. Wang, Z. Wu, J. Xiong et al., An Introduction to Optimal Control of FBSDE with Incomplete Information, Springer, 2018.
[50]	M. Yang, G. Zhang, D. Del-Castillo-Negrete and Y. Cao, A probabilistic scheme for semilinear nonlocal diffusion equations with volume constraints, SIAM Journal on Numerical Analysis, 61 (2023), 2718-2743. doi: 10.1137/22M1494877.
[51]	J. Yong and X. Y. Zhou, Stochastic Controls, Applications of Mathematics (New York), Springer-Verlag, New York, 43 (1999). doi: 10.1007/978-1-4612-1466-3.
[52]	M. Zakai, On the optimal filtering of diffusion processes, Z. Wahrscheinlichkeitstheorie und Verw. Gebiete, 11 (1969), 230-243. doi: 10.1007/BF00536382.
[53]	J. Zhang, A numerical scheme for BSDEs, Ann. Appl. Probab., 14 (2004), 459-488. doi: 10.1214/aoap/1075828058.
[54]	W. Zhao WeidongZhang and G. Zhang, Second-order numerical schemes for decoupled forward-backward stochastic differential equations with jumps, Journal of Computational Mathematics, 35 (2017), 213-244. doi: 10.4208/jcm.1612-m2015-0245.
[55]	W. Zhao, Y. Fu and T. Zhou, New kinds of high-order multistep schemes for coupled forward backward stochastic differential equations, SIAM J. Sci. Comput., 36 (2014), A1731-A1751. doi: 10.1137/130941274.
[56]	W. Zhao, Y. Li and G. Zhang, A generalized $\theta$-scheme for solving backward stochastic differential equations, Discrete and Continuous Dynamical Systems-B, 17 (2012), 1585-1603. doi: 10.3934/dcdsb.2012.17.1583.
[57]	W. Zhao, G. Zhang and L. Ju, A stable multistep scheme for solving backward stochastic differential equations, SIAM Journal on Numerical Analysis, 48 (2010), 1369-1394. doi: 10.1137/09076979X.
[58]	W. Zhao, T. Zhou and T. Kong, High order numerical schemes for second-order fbsdes with applications to stochastic optimal control, Communications in Computational Physics, 21 (2017), 808-834. doi: 10.4208/cicp.OA-2016-0056.