We present a numerical method for learning the dynamics of slow components of unknown multiscale stochastic dynamical systems. While the governing equations of the systems are unknown, bursts of observation data of the slow variables are available. By utilizing the observation data, our proposed method is capable of constructing a generative stochastic model that can accurately capture the effective dynamics of the slow variables in distribution. We present a comprehensive set of numerical examples to demonstrate the performance of the proposed method.
Citation: |
Figure 10. Sample trajectories of Example 4.4 slow variable with an initial condition $ {\mathbf{x}}_0 = (1.5, 1.0) $. Left column: ground truth of $ x_1 $ (top) and $ x_2 $ (bottom), with $ y_0 $ sampled from its stationary distribution (conditioned on $ {\mathbf{x}}_0 $); Right column: sFML simulation of $ x_1 $ (top) and $ x_2 $ (bottom)
[1] |
C. Archambeau, D. Cornford, M. Opper and J. Shawe-Taylor, Gaussian process approximations of stochastic differential equations. In Neil D. Lawrence, Anton Schwaighofer, and Joaquin Quiñonero Candela, editors, Gaussian Processes in Practice, volume 1 of Proceedings of Machine Learning Research, Bletchley Park, UK, 1 (2007).
![]() |
[2] |
S. L. Brunton, J. L. Proctor and J. Nathan Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, Proc. Natl. Acad. Sci. USA, 113 (2016), 3932-3937.
doi: 10.1073/pnas.1517384113.![]() ![]() ![]() |
[3] |
K. P. Champion, S. L. Brunton and J. N. Kutz, Discovery of nonlinear multiscale systems: Sampling strategies and embeddings, SIAM J. Appl. Dyn. Syst., 18 (2019), 312-333.
![]() ![]() |
[4] |
M. D. Chekroun, A. Tantet, H. A Dijkstra and J. D. Neelin, Ruelle-pollicott resonances of stochastic systems in reduced state space. part i: Theory, Journal of Statistical Physics, 179 (2020), 1366-1402.
![]() ![]() |
[5] |
R. T. Q. Chen, Y. Rubanova, J. Bettencourt and D. K. Duvenaud, Neural ordinary differential equations, In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, Curran Associates, Inc., 31 (2018).
![]() |
[6] |
X. Chen, J. Duan, J. Hu and D. Li, Data-driven method to learn the most probable transition pathway and stochastic differential equation, Phys. D, 443 (2023), 133559.
![]() ![]() |
[7] |
X. Chen, L. Yang, J. Duan and G. E. Karniadakis, Solving inverse stochastic problems from discrete particle observations using the Fokker-Planck equation and physics-informed neural networks, SIAM J. Sci. Comput., 43 (2021), 811-830.
doi: 10.1137/20M1360153.![]() ![]() ![]() |
[8] |
Y. Chen and D. Xiu, Learning stochastic dynamical system via flow map operator, J. Comput. Phys., 508 (2024), 112984, 23 pp.
![]() ![]() |
[9] |
Y. Chen and D. Xiu, Modeling unknown stochastic dynamical system subject to external excitation, preprint, arXiv: 2406.15747.
![]() |
[10] |
Z. Chen, V. Churchill, K. Wu and D. Xiu, Deep neural network modeling of unknown partial differential equations in nodal space, Journal of Computational Physics, 449 (2022), 110782, 20 pp.
doi: 10.1016/j.jcp.2021.110782.![]() ![]() ![]() |
[11] |
V. Churchill and D. Xiu, Flow map learning for unknown dynamical systems: Overview, implementation and benchmarks, Journal of Machine Learning for Modeling and Computing, 4 (2023), 173-201.
![]() |
[12] |
R. R. Coifman, I. G. Kevrekidis, S. Lafon, M. Maggioni and B. Nadler, Diffusion maps, reduction coordinates and low dimensional representation of stochastic systems, Multiscale Model. Simul., 7 (2008), 842-864.
doi: 10.1137/070696325.![]() ![]() ![]() |
[13] |
M. Darcy, B. Hamzi, G. Livieri, H. Owhadi and P. Tavallali, One-shot learning of stochastic differential equations with data adapted kernels. Phys. D, 444 (2023), 133583, 18 pp.
![]() ![]() |
[14] |
T. DelSole and B. F. Farrell, The quasi-linear equilibration of a thermally maintained, stochastically excited jet in a quasigeostrophic model, Journal of Atmospheric Sciences, 53 (1996), 1781-1797.
![]() |
[15] |
F. Dietrich, A. Makeev, G. Kevrekidis, N. Evangelou, T. Bertalan, S. Reich and I. G. Kevrekidis, Learning effective stochastic differential equations from microscopic simulations: Linking stochastic numerics to deep learning, Chaos: An Interdisciplinary Journal of Nonlinear Science, 33 (2023), 19 pp.
![]() ![]() |
[16] |
L. Dinh, J. Sohl-Dickstein and S. Bengio, Density estimation using real NVP, In International Conference on Learning Representations, 2017.
![]() |
[17] |
C. J. Dsilva, R. Talmon, C. W. Gear, R. R. Coifman and I. G. Kevrekidis, Data-driven reduction for a class of multiscale fast-slow stochastic dynamical systems, SIAM J. Appl. Dyn. Syst., 15 (2016), 1327-1351.
doi: 10.1137/151004896.![]() ![]() ![]() |
[18] |
J. Duan, K. Lu and B. Schmalfuss, Invariant manifolds for stochastic partial differential equations, Ann. Probab., 31 (2003), 2109-2135.
doi: 10.1214/aop/1068646380.![]() ![]() ![]() |
[19] |
W. E, D. Liu and E. Vanden-Eijnden, Analysis of multiscale methods for stochastic differential equations, Comm. Pure Appl. Math., 58 (2005), 1544-1585.
doi: 10.1002/cpa.20088.![]() ![]() ![]() |
[20] |
W. E and E. Vanden-Eijnden, Metastability, conformation dynamics and transition pathways in complex systems, In Multiscale modelling and simulation, Lect. Notes Comput. Sci. Eng., Springer, Berlin, 39 (2004), 35-68.
doi: 10.1007/978-3-642-18756-8_3.![]() ![]() ![]() |
[21] |
L. Feng, T. Gao, M. Dai and J. Duan, Learning effective dynamics from data-driven stochastic systems, Chaos, 33 (2023), 043131, 15 pp.
![]() ![]() |
[22] |
X. Fu, L.-B. Chang and D. Xiu, Learning reduced systems via deep neural networks with memory, J. Machine Learning Model. Comput., 1 (2020), 97-118.
doi: 10.1615/.2020034232.![]() ![]() |
[23] |
R. Khasminskij, On the principle of averaging the itov's stochastic differential equations, Kybernetika, 4 (1968), 260-279.
![]() ![]() |
[24] |
Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. liu, K. B. A. Stuart and A. Anandkumar, Fourier neural operator for parametric partial differential equations, In International Conference on Learning Representations, 2021.
![]() |
[25] |
C. Ma, J. Wang and W. E, Model reduction with memory and the machine learning of dynamical systems, Commun. Comput. Phys., 25 (2019), 947-962.
doi: 10.4208/cicp.OA-2018-0269.![]() ![]() ![]() |
[26] |
A. J. Majda, I. Timofeyev and E. Vanden Eijnden, Models for stochastic climate prediction, Proceedings of the National Academy of Sciences, 96 (1999), 14687-14691.
doi: 10.1073/pnas.96.26.14687.![]() ![]() ![]() |
[27] |
B. R. Noack, K. Afanasiev, M. Morzyński, G. Tadmor and F. Thiele, A hierarchy of low-dimensional models for the transient and post-transient cylinder wake, Journal of Fluid Mechanics, 497 (2003), 335-363.
doi: 10.1017/S0022112003006694.![]() ![]() ![]() |
[28] |
B. Øksendal, Stochastic Differential Equations, 6$^{th}$ edition, Springer-Verlag, Berlin, 2003.
doi: 10.1007/978-3-642-14394-6_5.![]() ![]() ![]() |
[29] |
M. Opper, Variational inference for stochastic differential equations, Ann. Phys., 531 (2019), 1800233, 9 pp.
doi: 10.1002/andp.201800233.![]() ![]() ![]() |
[30] |
H. Owhadi, Computational graph completion, Research in the Mathematical Sciences, 9 (2022), 27.
doi: 10.1007/s40687-022-00320-8.![]() ![]() ![]() |
[31] |
G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed and B. Lakshminarayanan, Normalizing flows for probabilistic modeling and inference, The Journal of Machine Learning Research, 22 (2021), 2617-2680.
![]() ![]() |
[32] |
G. Papamakarios, T. Pavlakou and I. Murray, Masked autoregressive flow for density estimation, In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan and R. Garnett, editors, Advances in Neural Information Processing Systems, Curran Associates, Inc., 30 (2017).
![]() |
[33] |
G. A. Pavliotis and A. Stuart, Multiscale Methods: Averaging and Homogenization, Texts Appl. Math., 53 Springer, New York, 2008.
![]() ![]() |
[34] |
T. Qin, Z. Chen, J. D. Jakeman and D. Xiu, Data-driven learning of nonautonomous systems, SIAM J. Sci. Comput., 43 (2021), 1607-1624.
doi: 10.1137/20M1342859.![]() ![]() ![]() |
[35] |
T. Qin, Z. Chen, J. D. Jakeman and D. Xiu, Deep learning of parameterized equations with applications to uncertainty quantification, Int. J. Uncertain. Quantif., 11 (2021), 63-82.
doi: 10.1615/Int.J.UncertaintyQuantification.2020034123.![]() ![]() ![]() |
[36] |
T. Qin, K. Wu and D. Xiu, Data driven governing equations approximation using deep neural networks, J. Comput. Phys., 395 (2019), 620-635.
doi: 10.1016/j.jcp.2019.06.042.![]() ![]() ![]() |
[37] |
M. Raissi, P. Perdikaris and G. E. Karniadakis, Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics, 378 (2019), 686-707.
![]() ![]() |
[38] |
M. Raissi, P. Perdikaris and G. E. Karniadakis, Multistep neural networks for data-driven discovery of nonlinear dynamical systems, preprint, arXiv: 1801.01236, 2018.
![]() |
[39] |
B. Schmalfuss and K. R. Schneider, Invariant manifolds for random dynamical systems with slow and fast variables, J. Dynam. Differential Equations, 20 (2008), 133-164.
doi: 10.1007/s10884-007-9089-7.![]() ![]() ![]() |
[40] |
K. Wu and D. Xiu, Data-driven deep learning of partial differential equations in modal space, Journal of Computational Physics, 408 (2020), 109307, 22 pp.
doi: 10.1016/j.jcp.2020.109307.![]() ![]() ![]() |
[41] |
Z. Xu, Y. Chen, Q. Chen and D. Xiu, Modeling unknown stochastic dynamical system via autoencoder, preprint, arXiv: 2312.10001, 2023.
![]() |
[42] |
L. Yang, C. Daskalakis and G. E. Karniadakis, Generative ensemble regression: Learning particle dynamics from observations of ensembles with physics-informed deep generative models, SIAM Journal on Scientific Computing, 44 (2022), 80-99.
![]() ![]() |
[43] |
F. X.-F. Ye, S. Yang and M. Maggioni, Nonlinear model reduction for slow-fast stochastic systems near unknown invariant manifolds, J. Nonlinear Sci., 34 (2024), 22, 54 pp.
![]() ![]() |
[44] |
C. Yildiz, M. Heinonen, J. Intosalmi, H. Mannerstrom and H. Lahdesmaki, Learning stochastic differential equations with gaussian processes without gradient matching, In 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP), (2018), 1-6.
![]() |
[45] |
J. Zhang, S. Zhang and G. Lin, Multiauto-deeponet: A multi-resolution autoencoder deeponet for nonlinear dimension reduction, uncertainty quantification and operator learning of forward and inverse stochastic problems, preprint, arXiv: 2204.03193, 2022.
![]() |
[46] |
P. Zieliński and J. S. Hesthaven, Discovery of slow variables in a class of multiscale stochastic systems via neural networks, J. Nonlinear Sci., 32 (2022), 51, 34 pp.
![]() ![]() |
Illustration of how the fast variables
The DNN model structure for the proposed normalizing flow sFML method (14)
Sample trajectories of Example 4.1 slow variable with initial condition
Left: One simulated sample of full system of Example 4.1 with initial condition
Sample trajectories of Example 4.2 slow variable with initial condition
Left: One simulated sample of full system of Example 4.2 with initial condition
Left: One simulated sample of full system of Example 4.4 with initial condition
Sample trajectories of Example 4.3 slow variable with initial condition
One simulated sample of full system of Example 4.4 with initial condition
Sample trajectories of Example 4.4 slow variable with an initial condition
Example 4.4: Mean and standard deviation (STD) for the slow variables
Example 4.4: Comparison of the distribution of the slow variables at
One simulated sample of full system of Example 4.5 with initial condition
Example 4.5: Samples of phase portrait of the slow variables
Example 4.5: Comparison of the mean and standard deviation (STD) for slow variables with initial condition
Example 4.5: Comparasion of the distribution of the slow variables at