Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

Marc Bocquet; Julien Brajard; Alberto Carrassi; Laurent Bertino

doi:10.3934/fods.2020004

Previous Article
Corrigendum to "Cluster, classify, regress: A general method for learning discontinuous functions [1]"
FoDS Home
This Issue
Next Article
Bayesian inference for latent chain graphs

March 2020, 2(1): 55-80. doi: 10.3934/fods.2020004

Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

Marc Bocquet ^1,, , Julien Brajard ^2, , Alberto Carrassi ^3, and Laurent Bertino ^4,

1.	CEREA, joint laboratory École des Ponts ParisTech and EDF R & D, Université Paris-Est, Champs-sur-Marne, France
2.	Nansen Environmental and Remote Sensing Center, Bergen, Norway, and Sorbonne University, CNRS-IRD-MNHN, LOCEAN, Paris, France
3.	Departement of Meteorology, University of Reading and NCEO, United-Kingdom, and Mathematical Institute, Utrecht University, The Netherlands
4.	Nansen Environmental and Remote Sensing Center, Bergen, Norway

^* Corresponding author: Marc Bocquet

Published March 2020

The reconstruction from observations of high-dimensional chaotic dynamics such as geophysical flows is hampered by (ⅰ) the partial and noisy observations that can realistically be obtained, (ⅱ) the need to learn from long time series of data, and (ⅲ) the unstable nature of the dynamics. To achieve such inference from the observations over long time series, it has been suggested to combine data assimilation and machine learning in several ways. We show how to unify these approaches from a Bayesian perspective using expectation-maximization and coordinate descents. In doing so, the model, the state trajectory and model error statistics are estimated all together. Implementations and approximations of these methods are discussed. Finally, we numerically and successfully test the approach on two relevant low-order chaotic models with distinct identifiability.

Keywords: Data assimilation, machine learning, neural networks, chaotic dynamical systems, expectation-maximization, coordinate descent.

Mathematics Subject Classification: Primary: 49J15, 65C60; Secondary: 86-08.

Citation: Marc Bocquet, Julien Brajard, Alberto Carrassi, Laurent Bertino. Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization. Foundations of Data Science, 2020, 2 (1) : 55-80. doi: 10.3934/fods.2020004

References:

[1]	H. D. I. Abarbanel, P. J. Rozdeba and S. Shirman, Machine learning: Deepest learning as statistical data assimilation problems, Neural Computation, 30 (2018), 2025-2055. doi: 10.1162/neco_a_01094.
[2]	M. Asch, M. Bocquet and M. Nodet, Data Assimilation: Methods, Algorithms, and Applications, Fundamentals of Algorithms, SIAM, Philadelphia, 2016. doi: 10.1137/1.9781611974546.pt1.
[3]	C. M. Bishop (ed.), Pattern Recognition and Machine Learning, Springer-Verlag New-York Inc, 2006.
[4]	C. H. Bishop, B. J. Etherton and S. J. Majumdar, Adaptive sampling with the ensemble transform Kalman filter. Part Ⅰ: Theoretical aspects, Mon. Wea. Rev., 129 (2001), 420-436. doi: 10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2.
[5]	M. Bocquet, Ensemble Kalman filtering without the intrinsic need for inflation, Nonlin. Processes Geophys., 18 (2011), 735-750. doi: 10.5194/npg-18-735-2011.
[6]	M. Bocquet, J. Brajard, A. Carrassi and L. Bertino, Data assimilation as a learning tool to infer ordinary differential equation representations of dynamical models, Nonlin. Processes Geophys., 26 (2019), 143-162. doi: 10.5194/npg-26-143-2019.
[7]	M. Bocquet and P. Sakov, Joint state and parameter estimation with an iterative ensemble Kalman smoother, Nonlin. Processes Geophys., 20 (2013), 803-818. doi: 10.5194/npg-20-803-2013.
[8]	J. Brajard, A. Carrassi, M. Bocquet and L. Bertino, Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model, J. Comput. Sci., 2020, http://arXiv.org/pdf/2001.01520.pdf. doi: 10.5194/gmd-2019-136.
[9]	S. L. Brunton, J. L. Proctor and J. N. Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, PNAS, 113 (2016), 3932-3937. doi: 10.1073/pnas.1517384113.
[10]	R. H. Byrd, P. Lu and J. Nocedal, A limited memory algorithm for bound constrained optimization, SIAM Journal on Scientific and Statistical Computing, 16 (1995), 1190-1208. doi: 10.1137/0916069.
[11]	M. Carlu, F. Ginelli, V. Lucarini and A. Politi, Lyapunov analysis of multiscale dynamics: The slow bundle of the two-scale Lorenz 96 model, Nonlin. Processes Geophys., 26 (2019), 73-89. doi: 10.5194/npg-26-73-2019.
[12]	A. Carrassi, M. Bocquet, L. Bertino and G. Evensen, Data assimilation in the geosciences: An overview on methods, issues, and perspectives, WIREs Climate Change, 9 (2018), e535. doi: 10.1002/wcc.535.
[13]	B. Chang, L. Meng, E. Haber, F. Tung and D. Begert, Multi-level residual networks from dynamical systems view, in Proceedings of ICLR 2018, 2018.
[14]	F. Chollet, Deep Learning with Python, Manning Publications Company, 2017.
[15]	A. P. Dempster, N. M. Laird and D. B. Rubin, Maximum likelihood from incomplete data via the EM algorithm, Journal of the Royal Statistical Society. Series B, 39 (1977), 1-38.
[16]	P. D. Dueben and P. Bauer, Challenges and design choices for global weather and climate models based on machine learning, Geosci. Model Dev., 11 (2018), 3999-4009. doi: 10.5194/gmd-11-3999-2018.
[17]	W. E, A proposal on machine learning via dynamical systems, Commun. Math. Stat., 5 (2017), 1-11. doi: 10.1007/s40304-017-0103-z.
[18]	G. Evensen, Data Assimilation: The Ensemble Kalman Filter, 2nd edition, Springer-Verlag Berlin Heildelberg, 2009. doi: 10.1007/978-3-642-03711-5.
[19]	R. Fablet, S. Ouala and C. Herzet, Bilinear residual neural network for the identification and forecasting of dynamical systems, in EUSIPCO 2018, European Signal Processing Conference, Rome, Italy, 2018, 1–5.
[20]	M. Fisher and S. Gürol, Parallelization in the time dimension of four-dimensional variational data assimilation, Q. J. R. Meteorol. Soc., 143 (2017), 1136-1147. doi: 10.1002/qj.2997.
[21]	Z. Ghahramani and S. T. Roweis, Learning nonlinear dynamical systems using an EM algorithm, in Advances in neural information processing systems, 1999,431–437.
[22]	W. W. Hsieh and B. Tang, Applying neural network models to prediction and data analysis in meteorology and oceanography, Bull. Amer. Meteor. Soc., 79 (1998), 1855-1870. doi: 10.1175/1520-0477(1998)079<1855:ANNMTP>2.0.CO;2.
[23]	B. R. Hunt, E. J. Kostelich and I. Szunyogh, Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter, Physica D, 230 (2007), 112-126. doi: 10.1016/j.physd.2006.11.008.
[24]	E. Kalnay, H. Li, T. Miyoshi, S.-C. Yang and J. Ballabrera-Poy, 4-D-Varor ensemble Kalman filter?, Tellus A, 59 (2007), 758-773.
[25]	Y. A. LeCun, L. Bottou, G. B. Orr and K.-R. Müller, Efficient backprop, in Neural networks: Tricks of the trade, Springer, 2012, 9–48.
[26]	R. Lguensat, P. Tandeo, P. Ailliot, M. Pulido and R. Fablet, The analog data assimilation, Mon. Wea. Rev., 145 (2017), 4093-4107. doi: 10.1175/MWR-D-16-0441.1.
[27]	Y. Liu, J.-M. Haussaire, M. Bocquet, Y. Roustan, O. Saunier and A. Mathieu, Uncertainty quantification of pollutant source retrieval: comparison of Bayesian methods with application to the Chernobyl and Fukushima-Daiichi accidental releases of radionuclides, Q. J. R. Meteorol. Soc., 143 (2017), 2886-2901. doi: 10.1002/qj.3138.
[28]	Z. Long, Y. Lu, X. Ma and B. Dong, PDE-Net: Learning PDEs from Data, , in Proceedings of the 35th International Conference on Machine Learning, 2018.
[29]	E. N. Lorenz, Designing chaotic models, J. Atmos. Sci., 62 (2005), 1574-1587. doi: 10.1175/JAS3430.1.
[30]	E. N. Lorenz and K. A. Emanuel, Optimal sites for supplementary weather observations: Simulation with a small model, J. Atmos. Sci., 55 (1998), 399-414. doi: 10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.
[31]	L. Magnusson and E. Källén, Factors influencing skill improvements in the ecmwf forecasting system, Mon. Wea. Rev., 141 (2013), 3142-3153. doi: 10.1175/MWR-D-12-00318.1.
[32]	V. D. Nguyen, S. Ouala, L. Drumetz and R. Fablet, EM-like learning chaotic dynamics from noisy and partial observations, arXiv preprint, arXiv: 1903.10335.
[33]	J. Paduart, L. Lauwers, J. Swevers, K. Smolders, J. Schoukens and R. Pintelon, Identification of nonlinear systems using polynomial nonlinear state space models, Automatica, 46 (2010), 647-656. doi: 10.1016/j.automatica.2010.01.001.
[34]	D. C. Park and Y. Zhu, Bilinear recurrent neural network, in IEEE World Congress on Computational Intelligence., 1994 IEEE International Conference on, 3 (1994), 1459–1464. doi: 10.1109/ICNN.1994.374501.
[35]	J. Pathak, B. Hunt, M. Girvan, Z. Lu and E. Ott, Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach, Phys. Rev. Lett., 120 (2018), 024102. doi: 10.1103/PhysRevLett.120.024102.
[36]	J. Pathak, Z. Lu, B. R. Hunt, M. Girvan and E. Ott, Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data, Chaos, 27 (2017), 121102, 9pp. doi: 10.1063/1.5010300.
[37]	M. Pulido, P. Tandeo, M. Bocquet, A. Carrassi and M. Lucini, Stochastic parameterization identification using ensemble Kalman filtering combined with maximum likelihood methods, Tellus A, 70 (2018), 1-17. doi: 10.1080/16000870.2018.1442099.
[38]	P. N. Raanes, A. Carrassi and L. Bertino, Extending the square root method to account for additive forecast noise in ensemble methods, Mon. Wea. Rev., 143 (2015), 3857-3873. doi: 10.1175/MWR-D-14-00375.1.
[39]	V. Rao and A. Sandu, A time-parallel approach to strong-constraint four-dimensional variational data assimilation, J. Comp. Phys., 313 (2016), 583-593. doi: 10.1016/j.jcp.2016.02.040.
[40]	M. Reichstein, G. Camps-Valls, B. Stevens, J. Denzler, N. Carvalhais and Pr abhat, Deep learning and process understanding for data-driven Earth system science, Nature, 566 (2019), 195-204. doi: 10.1038/s41586-019-0912-1.
[41]	P. Sakov, J.-M. Haussaire and M. Bocquet, An iterative ensemble Kalman filter in presence of additive model error, Q. J. R. Meteorol. Soc., 144 (2018), 1297-1309. doi: 10.1002/qj.3213.
[42]	P. Tandeo, P. Ailliot, M. Bocquet, A. Carrassi, T. Miyoshi, M. Pulido and Y. Zhen, A review of innovation based approaches to jointly estimate model and observation error covariance matrices in ensemble data assimilation, 2020, https://arXiv.org/abs/1807.11221, Submitted.
[43]	Y. Trémolet, Accounting for an imperfect model in 4D-Var, Q. J. R. Meteorol. Soc., 132 (2006), 2483-2504.
[44]	P. R. Vlachas, J. Pathak, B. R. Hunt, T. P. Sapsis, M. Girvan, E. Ott and P. Koumoutsakos, Forecasting of spatio-temporal chaotic dynamics with recurrent neural networks: a comparative study of reservoir computing and backpropagation algorithms, arXiv preprint, arXiv: 1910.05266.,
[45]	Y.-J. Wang and C.-T. Lin, Runge-Kutta neural network for identification of dynamical systems in high accuracy, IEEE Transactions on Neural Networks, 9 (1998), 294-307.
[46]	G. C. G. Wei and M. A. Tanner, A Monte Carlo implementation of the EM algorithm and the poor man's data augmentation algorithms, Journal of the American Statistical Association, 85 (1990), 699-704. doi: 10.1080/01621459.1990.10474930.
[47]	P. Welch, The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms, IEEE Transactions on Audio and Electroacoustics, 15 (1967), 70-73. doi: 10.1109/TAU.1967.1161901.
[48]	S. J. Wright, Coordinate descent algorithms, Mathematical Programming, 151 (2015), 3-34. doi: 10.1007/s10107-015-0892-3.

show all references

References:

[1]	H. D. I. Abarbanel, P. J. Rozdeba and S. Shirman, Machine learning: Deepest learning as statistical data assimilation problems, Neural Computation, 30 (2018), 2025-2055. doi: 10.1162/neco_a_01094.
[2]	M. Asch, M. Bocquet and M. Nodet, Data Assimilation: Methods, Algorithms, and Applications, Fundamentals of Algorithms, SIAM, Philadelphia, 2016. doi: 10.1137/1.9781611974546.pt1.
[3]	C. M. Bishop (ed.), Pattern Recognition and Machine Learning, Springer-Verlag New-York Inc, 2006.
[4]	C. H. Bishop, B. J. Etherton and S. J. Majumdar, Adaptive sampling with the ensemble transform Kalman filter. Part Ⅰ: Theoretical aspects, Mon. Wea. Rev., 129 (2001), 420-436. doi: 10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2.
[5]	M. Bocquet, Ensemble Kalman filtering without the intrinsic need for inflation, Nonlin. Processes Geophys., 18 (2011), 735-750. doi: 10.5194/npg-18-735-2011.
[6]	M. Bocquet, J. Brajard, A. Carrassi and L. Bertino, Data assimilation as a learning tool to infer ordinary differential equation representations of dynamical models, Nonlin. Processes Geophys., 26 (2019), 143-162. doi: 10.5194/npg-26-143-2019.
[7]	M. Bocquet and P. Sakov, Joint state and parameter estimation with an iterative ensemble Kalman smoother, Nonlin. Processes Geophys., 20 (2013), 803-818. doi: 10.5194/npg-20-803-2013.
[8]	J. Brajard, A. Carrassi, M. Bocquet and L. Bertino, Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model, J. Comput. Sci., 2020, http://arXiv.org/pdf/2001.01520.pdf. doi: 10.5194/gmd-2019-136.
[9]	S. L. Brunton, J. L. Proctor and J. N. Kutz, Discovering governing equations from data by sparse identification of nonlinear dynamical systems, PNAS, 113 (2016), 3932-3937. doi: 10.1073/pnas.1517384113.
[10]	R. H. Byrd, P. Lu and J. Nocedal, A limited memory algorithm for bound constrained optimization, SIAM Journal on Scientific and Statistical Computing, 16 (1995), 1190-1208. doi: 10.1137/0916069.
[11]	M. Carlu, F. Ginelli, V. Lucarini and A. Politi, Lyapunov analysis of multiscale dynamics: The slow bundle of the two-scale Lorenz 96 model, Nonlin. Processes Geophys., 26 (2019), 73-89. doi: 10.5194/npg-26-73-2019.
[12]	A. Carrassi, M. Bocquet, L. Bertino and G. Evensen, Data assimilation in the geosciences: An overview on methods, issues, and perspectives, WIREs Climate Change, 9 (2018), e535. doi: 10.1002/wcc.535.
[13]	B. Chang, L. Meng, E. Haber, F. Tung and D. Begert, Multi-level residual networks from dynamical systems view, in Proceedings of ICLR 2018, 2018.
[14]	F. Chollet, Deep Learning with Python, Manning Publications Company, 2017.
[15]	A. P. Dempster, N. M. Laird and D. B. Rubin, Maximum likelihood from incomplete data via the EM algorithm, Journal of the Royal Statistical Society. Series B, 39 (1977), 1-38.
[16]	P. D. Dueben and P. Bauer, Challenges and design choices for global weather and climate models based on machine learning, Geosci. Model Dev., 11 (2018), 3999-4009. doi: 10.5194/gmd-11-3999-2018.
[17]	W. E, A proposal on machine learning via dynamical systems, Commun. Math. Stat., 5 (2017), 1-11. doi: 10.1007/s40304-017-0103-z.
[18]	G. Evensen, Data Assimilation: The Ensemble Kalman Filter, 2nd edition, Springer-Verlag Berlin Heildelberg, 2009. doi: 10.1007/978-3-642-03711-5.
[19]	R. Fablet, S. Ouala and C. Herzet, Bilinear residual neural network for the identification and forecasting of dynamical systems, in EUSIPCO 2018, European Signal Processing Conference, Rome, Italy, 2018, 1–5.
[20]	M. Fisher and S. Gürol, Parallelization in the time dimension of four-dimensional variational data assimilation, Q. J. R. Meteorol. Soc., 143 (2017), 1136-1147. doi: 10.1002/qj.2997.
[21]	Z. Ghahramani and S. T. Roweis, Learning nonlinear dynamical systems using an EM algorithm, in Advances in neural information processing systems, 1999,431–437.
[22]	W. W. Hsieh and B. Tang, Applying neural network models to prediction and data analysis in meteorology and oceanography, Bull. Amer. Meteor. Soc., 79 (1998), 1855-1870. doi: 10.1175/1520-0477(1998)079<1855:ANNMTP>2.0.CO;2.
[23]	B. R. Hunt, E. J. Kostelich and I. Szunyogh, Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter, Physica D, 230 (2007), 112-126. doi: 10.1016/j.physd.2006.11.008.
[24]	E. Kalnay, H. Li, T. Miyoshi, S.-C. Yang and J. Ballabrera-Poy, 4-D-Varor ensemble Kalman filter?, Tellus A, 59 (2007), 758-773.
[25]	Y. A. LeCun, L. Bottou, G. B. Orr and K.-R. Müller, Efficient backprop, in Neural networks: Tricks of the trade, Springer, 2012, 9–48.
[26]	R. Lguensat, P. Tandeo, P. Ailliot, M. Pulido and R. Fablet, The analog data assimilation, Mon. Wea. Rev., 145 (2017), 4093-4107. doi: 10.1175/MWR-D-16-0441.1.
[27]	Y. Liu, J.-M. Haussaire, M. Bocquet, Y. Roustan, O. Saunier and A. Mathieu, Uncertainty quantification of pollutant source retrieval: comparison of Bayesian methods with application to the Chernobyl and Fukushima-Daiichi accidental releases of radionuclides, Q. J. R. Meteorol. Soc., 143 (2017), 2886-2901. doi: 10.1002/qj.3138.
[28]	Z. Long, Y. Lu, X. Ma and B. Dong, PDE-Net: Learning PDEs from Data, , in Proceedings of the 35th International Conference on Machine Learning, 2018.
[29]	E. N. Lorenz, Designing chaotic models, J. Atmos. Sci., 62 (2005), 1574-1587. doi: 10.1175/JAS3430.1.
[30]	E. N. Lorenz and K. A. Emanuel, Optimal sites for supplementary weather observations: Simulation with a small model, J. Atmos. Sci., 55 (1998), 399-414. doi: 10.1175/1520-0469(1998)055<0399:OSFSWO>2.0.CO;2.
[31]	L. Magnusson and E. Källén, Factors influencing skill improvements in the ecmwf forecasting system, Mon. Wea. Rev., 141 (2013), 3142-3153. doi: 10.1175/MWR-D-12-00318.1.
[32]	V. D. Nguyen, S. Ouala, L. Drumetz and R. Fablet, EM-like learning chaotic dynamics from noisy and partial observations, arXiv preprint, arXiv: 1903.10335.
[33]	J. Paduart, L. Lauwers, J. Swevers, K. Smolders, J. Schoukens and R. Pintelon, Identification of nonlinear systems using polynomial nonlinear state space models, Automatica, 46 (2010), 647-656. doi: 10.1016/j.automatica.2010.01.001.
[34]	D. C. Park and Y. Zhu, Bilinear recurrent neural network, in IEEE World Congress on Computational Intelligence., 1994 IEEE International Conference on, 3 (1994), 1459–1464. doi: 10.1109/ICNN.1994.374501.
[35]	J. Pathak, B. Hunt, M. Girvan, Z. Lu and E. Ott, Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach, Phys. Rev. Lett., 120 (2018), 024102. doi: 10.1103/PhysRevLett.120.024102.
[36]	J. Pathak, Z. Lu, B. R. Hunt, M. Girvan and E. Ott, Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data, Chaos, 27 (2017), 121102, 9pp. doi: 10.1063/1.5010300.
[37]	M. Pulido, P. Tandeo, M. Bocquet, A. Carrassi and M. Lucini, Stochastic parameterization identification using ensemble Kalman filtering combined with maximum likelihood methods, Tellus A, 70 (2018), 1-17. doi: 10.1080/16000870.2018.1442099.
[38]	P. N. Raanes, A. Carrassi and L. Bertino, Extending the square root method to account for additive forecast noise in ensemble methods, Mon. Wea. Rev., 143 (2015), 3857-3873. doi: 10.1175/MWR-D-14-00375.1.
[39]	V. Rao and A. Sandu, A time-parallel approach to strong-constraint four-dimensional variational data assimilation, J. Comp. Phys., 313 (2016), 583-593. doi: 10.1016/j.jcp.2016.02.040.
[40]	M. Reichstein, G. Camps-Valls, B. Stevens, J. Denzler, N. Carvalhais and Pr abhat, Deep learning and process understanding for data-driven Earth system science, Nature, 566 (2019), 195-204. doi: 10.1038/s41586-019-0912-1.
[41]	P. Sakov, J.-M. Haussaire and M. Bocquet, An iterative ensemble Kalman filter in presence of additive model error, Q. J. R. Meteorol. Soc., 144 (2018), 1297-1309. doi: 10.1002/qj.3213.
[42]	P. Tandeo, P. Ailliot, M. Bocquet, A. Carrassi, T. Miyoshi, M. Pulido and Y. Zhen, A review of innovation based approaches to jointly estimate model and observation error covariance matrices in ensemble data assimilation, 2020, https://arXiv.org/abs/1807.11221, Submitted.
[43]	Y. Trémolet, Accounting for an imperfect model in 4D-Var, Q. J. R. Meteorol. Soc., 132 (2006), 2483-2504.
[44]	P. R. Vlachas, J. Pathak, B. R. Hunt, T. P. Sapsis, M. Girvan, E. Ott and P. Koumoutsakos, Forecasting of spatio-temporal chaotic dynamics with recurrent neural networks: a comparative study of reservoir computing and backpropagation algorithms, arXiv preprint, arXiv: 1910.05266.,
[45]	Y.-J. Wang and C.-T. Lin, Runge-Kutta neural network for identification of dynamical systems in high accuracy, IEEE Transactions on Neural Networks, 9 (1998), 294-307.
[46]	G. C. G. Wei and M. A. Tanner, A Monte Carlo implementation of the EM algorithm and the poor man's data augmentation algorithms, Journal of the American Statistical Association, 85 (1990), 699-704. doi: 10.1080/01621459.1990.10474930.
[47]	P. Welch, The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms, IEEE Transactions on Audio and Electroacoustics, 15 (1967), 70-73. doi: 10.1109/TAU.1967.1161901.
[48]	S. J. Wright, Coordinate descent algorithms, Mathematical Programming, 151 (2015), 3-34. doi: 10.1007/s10107-015-0892-3.

Figure 1. From top to bottom: representation of the flow rate $ \boldsymbol \phi_ \mathbf{A} $ with a NN, integration of the flow rate into $ \mathbf{f}_ \mathbf{A} $ using an explicit integration scheme (here a second-order Runge Kutta scheme), and $ {N_\mathrm{c}}- $fold composition up to the full resolvent $ \mathbf{F}_ \mathbf{A} $. $ \delta t $ is the integration time step corresponding to the resolvent $ \mathbf{f}_ \mathbf{A} $

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 2. On the left hand side: Properties of the surrogate model obtained from full but noisy observation of the L96 model in the nominal configuration ($ L = 4 $, $ K = 5000 $, $ \sigma_y = 1 $, $ {N_\mathrm{y}} = {N_\mathrm{x}} = 40 $). On the right hand side: Properties of the surrogate model obtained from full but noisy observation of the L05Ⅲ model in the nominal configuration ($ L = 4 $, $ K = 5000 $, $ \sigma_y = 1 $, $ {N_\mathrm{y}} = {N_\mathrm{x}} = 36 $). From top to bottom, are plotted the FS (NRMSE as a function of lead time in Lyapunov time), the LS (all exponents), and the PSD (in log-log-scale). A total of $ 10 $ experiments have been performed for both configurations. The curves corresponding to each member are drawn with thin blue lines while the mean of each indicator over the ensemble are drawn in thick dashed orange line

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 3. Same as Figure 2 but for several values of the training window length $ K $. Each curve is the mean over $ 10 $ experiments with different sets of observations. The LS and PSD of the reference models are also plotted for comparison

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 4. On the left hand side: Properties of the surrogate model obtained from full but noisy observation of the L96 model in the nominal configuration ($ L = 4 $, $ K = 5000 $, $ {N_\mathrm{y}} = {N_\mathrm{x}} = 40 $ and with several $ \sigma_y $). On the right hand side: Properties of the surrogate model obtained from full but noisy observation of the L05Ⅲ model in the nominal configuration ($ L = 4 $, $ K = 5000 $, $ \sigma_y = 1 $, $ {N_\mathrm{y}} = {N_\mathrm{x}} = 36 $ and with several $ \sigma_y $). From top to bottom, are plotted the FS (NRMSE as a function of lead time in Lyapunov time) and the PSD (in log-log-scale), averaged over an ensemble of $ 10 $ samples

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 5. On the left hand side: Properties of the surrogate model obtained from partial and noisy observation of the L96 model in the nominal configuration ($ L = 4 $, $ K = 5000 $, $ \sigma_y = 1 $, $ {N_\mathrm{x}} = 40 $) where $ {N_\mathrm{y}} $ is varied. On the right hand side: Properties of the surrogate model obtained from partial and noisy observation of the L05Ⅲ model in the nominal configuration ($ L = 4 $, $ K = 5000 $, $ \sigma_y = 1 $, $ {N_\mathrm{x}} = 36 $) where $ {N_\mathrm{y}} $ is varied. From top to bottom, are plotted the mean FS (NRMSE as a function of lead time in Lyapunov time), the mean LS (all exponents), and the mean PSD (in log-log-scale). A total of $ 10 $ experiments have been performed for both configurations

Figure Options

Download full-size image

Download as PowerPoint slide

Table 1. Scalar indicators for nominal experiments based on L96 and L05Ⅲ. Key hyperparameters are recalled. The statistics of the indicators are obtained over $ 10 $ samples

Model	$ {N_\mathrm{y}} $	$ \sigma_y $	$ K $	$ L $	$ \pi_ \frac{1}{2} $	$ \sigma_q $	$ \lambda_1 $
L96	$ 40 $	$ 1 $	$ 5000 $	$ 4 $	$ 4.56 \pm 0.06 $	$ 0.08790 \pm 2\, 10^{-5} $	$ 1.66 \pm 0.02 $
L05Ⅲ	$ 36 $	$ 1 $	$ 5000 $	$ 4 $	$ 4.06 \pm 0.21 $	$ 0.07720 \pm 2\, 10^{-5} $	$ 1.03 \pm 0.05 $

Table Options

Download as excel

Model	$ {N_\mathrm{y}} $	$ \sigma_y $	$ K $	$ L $	$ \pi_ \frac{1}{2} $	$ \sigma_q $	$ \lambda_1 $
L96	$ 40 $	$ 1 $	$ 5000 $	$ 4 $	$ 4.56 \pm 0.06 $	$ 0.08790 \pm 2\, 10^{-5} $	$ 1.66 \pm 0.02 $
L05Ⅲ	$ 36 $	$ 1 $	$ 5000 $	$ 4 $	$ 4.06 \pm 0.21 $	$ 0.07720 \pm 2\, 10^{-5} $	$ 1.03 \pm 0.05 $

Table 2. Scalar indicators for L96 and L05Ⅲ in their nominal configuration, using either the full or the approximate schemes. The statistics of the indicators are obtained over $ 10 $ samples

Model	Scheme	$ \pi_ \frac{1}{2} $	$ \sigma_q $	$ \lambda_1 $
L96	Approximate	$ 4.56 \pm 0.06 $	$ 0.08790 \pm 2\, 10^{-5} $	$ 1.66 \pm 0.02 $
L96	Full	$ 4.24 \pm 0.07 $	$ 0.09152 $	$ 1.66 \pm 0.02 $
L05Ⅲ	Approximate	$ 4.06 \pm 0.21 $	$ 0.07720 \pm 2\, 10^{-5} $	$ 1.03 \pm 0.05 $
L05Ⅲ	Full	$ 3.97 \pm 0.17 $	$ 0.08024 $	$ 1.03 \pm 0.04 $

Table Options

Download as excel

Model	Scheme	$ \pi_ \frac{1}{2} $	$ \sigma_q $	$ \lambda_1 $
L96	Approximate	$ 4.56 \pm 0.06 $	$ 0.08790 \pm 2\, 10^{-5} $	$ 1.66 \pm 0.02 $
L96	Full	$ 4.24 \pm 0.07 $	$ 0.09152 $	$ 1.66 \pm 0.02 $
L05Ⅲ	Approximate	$ 4.06 \pm 0.21 $	$ 0.07720 \pm 2\, 10^{-5} $	$ 1.03 \pm 0.05 $
L05Ⅲ	Full	$ 3.97 \pm 0.17 $	$ 0.08024 $	$ 1.03 \pm 0.04 $

[1]	Truong-Vinh Hoang, Sebastian Krumscheid, Hermann G. Matthies, Raúl Tempone. Machine learning-based conditional mean filter: A generalization of the ensemble Kalman filter for nonlinear data assimilation. Foundations of Data Science, 2022 doi: 10.3934/fods.2022016
[2]	Yacine Chitour, Zhenyu Liao, Romain Couillet. A geometric approach of gradient descent algorithms in linear neural networks. Mathematical Control and Related Fields, 2022 doi: 10.3934/mcrf.2022021
[3]	Kengo Nakai, Yoshitaka Saiki. Machine-learning construction of a model for a macroscopic fluid variable using the delay-coordinate of a scalar observable. Discrete and Continuous Dynamical Systems - S, 2021, 14 (3) : 1079-1092. doi: 10.3934/dcdss.2020352
[4]	Jiang Xie, Junfu Xu, Celine Nie, Qing Nie. Machine learning of swimming data via wisdom of crowd and regression analysis. Mathematical Biosciences & Engineering, 2017, 14 (2) : 511-527. doi: 10.3934/mbe.2017031
[5]	Andreas Chirstmann, Qiang Wu, Ding-Xuan Zhou. Preface to the special issue on analysis in machine learning and data science. Communications on Pure and Applied Analysis, 2020, 19 (8) : i-iii. doi: 10.3934/cpaa.2020171
[6]	Stefano Galatolo. Global and local complexity in weakly chaotic dynamical systems. Discrete and Continuous Dynamical Systems, 2003, 9 (6) : 1607-1624. doi: 10.3934/dcds.2003.9.1607
[7]	Ming Yan, Alex A. T. Bui, Jason Cong, Luminita A. Vese. General convergent expectation maximization (EM)-type algorithms for image reconstruction. Inverse Problems and Imaging, 2013, 7 (3) : 1007-1029. doi: 10.3934/ipi.2013.7.1007
[8]	Michele La Rocca, Cira Perna. Designing neural networks for modeling biological data: A statistical perspective. Mathematical Biosciences & Engineering, 2014, 11 (2) : 331-342. doi: 10.3934/mbe.2014.11.331
[9]	Yoeri E. Boink, Markus Haltmeier, Sean Holman, Johannes Schwab. Data-consistent neural networks for solving nonlinear inverse problems. Inverse Problems and Imaging, , () : -. doi: 10.3934/ipi.2022037
[10]	Arash Mehrjou, Andrea Iannelli, Bernhard Schölkopf. Learning Dynamical Systems using Local Stability Priors. Journal of Computational Dynamics, 2022 doi: 10.3934/jcd.2022021
[11]	Jianping Zhou, Yamin Liu, Ju H. Park, Qingkai Kong, Zhen Wang. Fault-tolerant anti-synchronization control for chaotic switched neural networks with time delay and reaction diffusion. Discrete and Continuous Dynamical Systems - S, 2021, 14 (4) : 1569-1589. doi: 10.3934/dcdss.2020357
[12]	Alexandre J. Chorin, Fei Lu, Robert N. Miller, Matthias Morzfeld, Xuemin Tu. Sampling, feasibility, and priors in data assimilation. Discrete and Continuous Dynamical Systems, 2016, 36 (8) : 4227-4246. doi: 10.3934/dcds.2016.36.4227
[13]	Liqiang Zhu, Ying-Cheng Lai, Frank C. Hoppensteadt, Jiping He. Characterization of Neural Interaction During Learning and Adaptation from Spike-Train Data. Mathematical Biosciences & Engineering, 2005, 2 (1) : 1-23. doi: 10.3934/mbe.2005.2.1
[14]	King Hann Lim, Hong Hui Tan, Hendra G. Harno. Approximate greatest descent in neural network optimization. Numerical Algebra, Control and Optimization, 2018, 8 (3) : 327-336. doi: 10.3934/naco.2018021
[15]	Quan Hai, Shutang Liu. Mean-square delay-distribution-dependent exponential synchronization of chaotic neural networks with mixed random time-varying delays and restricted disturbances. Discrete and Continuous Dynamical Systems - B, 2021, 26 (6) : 3097-3118. doi: 10.3934/dcdsb.2020221
[16]	Liu Hui, Lin Zhi, Waqas Ahmad. Network(graph) data research in the coordinate system. Mathematical Foundations of Computing, 2018, 1 (1) : 1-10. doi: 10.3934/mfc.2018001
[17]	K. L. Mak, J. G. Peng, Z. B. Xu, K. F. C. Yiu. A novel neural network for associative memory via dynamical systems. Discrete and Continuous Dynamical Systems - B, 2006, 6 (3) : 573-590. doi: 10.3934/dcdsb.2006.6.573
[18]	Débora A. F. Albanez, Maicon J. Benvenutti. Continuous data assimilation algorithm for simplified Bardina model. Evolution Equations and Control Theory, 2018, 7 (1) : 33-52. doi: 10.3934/eect.2018002
[19]	Jochen Bröcker. Existence and uniqueness for variational data assimilation in continuous time. Mathematical Control and Related Fields, 2021 doi: 10.3934/mcrf.2021050
[20]	Jules Guillot, Emmanuel Frénod, Pierre Ailliot. Physics informed model error for data assimilation. Discrete and Continuous Dynamical Systems - S, 2022 doi: 10.3934/dcdss.2022059

Impact Factor:

Tools

Metrics

Tools

Article outline

Figures and Tables

Impact Factor:

Tools

Figures and Tables

Impact Factor:

American Institute of Mathematical Sciences

Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

References:

References:

Tools

Metrics

Other articles
by authors

Tools

Tools

Tools

Metrics

Other articles
by authors

American Institute of Mathematical Sciences

Bayesian inference of chaotic dynamics by merging data assimilation, machine learning and expectation-maximization

References:

References:

Tools

Metrics

Other articlesby authors

Tools

Tools

Tools

Metrics

Other articlesby authors

Other articles
by authors

Other articles
by authors