Article Contents
Article Contents

# Machine learning-based conditional mean filter: A generalization of the ensemble Kalman filter for nonlinear data assimilation

• *Corresponding author: Truong-Vinh Hoang
• This paper presents the machine learning-based ensemble conditional mean filter (ML-EnCMF) — a filtering method based on the conditional mean filter (CMF) previously introduced in the literature. The updated mean of the CMF matches that of the posterior, obtained by applying Bayes' rule on the filter's forecast distribution. Moreover, we show that the CMF's updated covariance coincides with the expected conditional covariance. Implementing the EnCMF requires computing the conditional mean (CM). A likelihood-based estimator is prone to significant errors for small ensemble sizes, causing the filter divergence. We develop a systematical methodology for integrating machine learning into the EnCMF based on the CM's orthogonal projection property. First, we use a combination of an artificial neural network (ANN) and a linear function, obtained based on the ensemble Kalman filter (EnKF), to approximate the CM, enabling the ML-EnCMF to inherit EnKF's advantages. Secondly, we apply a suitable variance reduction technique to reduce statistical errors when estimating loss function. Lastly, we propose a model selection procedure for element-wisely selecting the applied filter, i.e., either the EnKF or ML-EnCMF, at each updating step. We demonstrate the ML-EnCMF performance using the Lorenz-63 and Lorenz-96 systems and show that the ML-EnCMF outperforms the EnKF and the likelihood-based EnCMF.

Mathematics Subject Classification: Primary: 62M45, 62M20, 65C20; Secondary: 86-08, 93E11.

 Citation:

• Figure 1.  CM computed using Eq. (9) and its linear approximation (Eq. (11))

Figure 2.  Empirical PDF of the conditional variance $\mathbb{V} \text{ar}\left[ {Q^{f }|Y^f}\right]$ expressed in Eq. (25b) compared with the variances of the updated ensembles obtained using the EnKF and EnCMF. The expected conditional variance, $\mathbb{E}\left[ {\mathbb{V} \text{ar}\left[ {Q^{f }|Y^f}\right]}\right]$, and the updated ensemble variance of the EnCMF show closely identical estimated values of approximately 0.17

Figure 3.  Comparison of the empirical densities of the updated ensembles and the Bayesian posterior: (a) actual value $\mathrm{{ {q}}}^{\mathrm{tr}} = -2$ (minus standard deviation of the prior distribution), (b) actual value $\mathrm{{ {q}}}^{\mathrm{tr}} = 0$ (mean of the prior distribution), (c) actual value $\mathrm{{ {q}}}^{\mathrm{tr}} = +2$ (plus standard deviation of the prior distribution)

Figure 4.  Lorenz 63: average RMSE of the CM approximations using LL-EnCMF and ML-EnCMF $\overline{rmse}_{\phi}$, a) $\Delta T_{\text{obs}} = 0.5$, b) $\Delta T_{\text{obs}} = 1$

Table 1.  L63 system: performance metrics (average RMSE - average ensemble spread (average coverage probability $f_{ \text{cv}}$)) of the ML-EnCMF compared with the LL-EnCMF and EnKF with $\Delta T_{\mathrm{obs}} = 0.5$. The LL-EnCMF uses inflation coefficients $1.25$, $1.2$, and $1.05$ for ensemble sizes 20, 30, and 60, respectively. For $N = 20$, 30, and 60 we compute the coverage probability for 90%, 93.3%, and 93.3%-confident intervals, respectively

 $N$ 20 30 60 100 200 ML-EnCMF 1.27-0.96 1.11 - 0.97 0.94 - 0.97 0.86 - 0.97 0.81 - 0.96 (0.83) (0.90) (0.92) (0.94) (0.95) EnKF 1.37-1.17 1.29 - 1.21 1.24 - 1.27 1.23 - 1.29 1.22 - 1.30 (0.86) (0.90) (0.93) (0.93) (0.93) LL-EnCMF 1.43 - 1.02 1.21-1.04 0.99-0.91 0.90-0.89 0.85 - 0.90 (0.85) (0.92) (0.93) (0.93) (0.95)

Table 2.  L63 system: performance metrics (average RMSE - average ensemble spread (average coverage probability $f_{ \text{cv}}$)) of the ML-EnCMF compared with the LL-EnCMF and EnKF with $\Delta T_{\mathrm{obs}} = 1$. The LL-EnCMF uses inflation coefficients $1.25$, $1.2$, and $1.1$ for ensemble sizes 20, 30, and 60, respectively. For $N = 20$, 30, and 60 we compute the coverage probability for 90%, 93.3%, and 93.3%-confident intervals, respectively

 $N$ 20 30 60 100 200 ML-EnCMF 1.66 - 1.19 1.50 - 1.20 1.22 - 1.21 1.14-1.18 1.06 - 1.17 (0.82) (0.90) (0.90) (0.93) (0.93) EnKF 1.67 -1.46 1.61 - 1.55 1.53 - 1.57 1.51 - 1.60 1.51 - 1.61 (0.87) (0.92) (0.91) (0.93) (0.93) LL-EnCMF 2.52 - 1.11 1.78-1.15 1.35 - 1.12 1.18 - 1.06 1.05 - 1.06 (0.72) (0.86) (0.90) (0.93) (0.93)

Table 3.  L96 system: performance metrics, i.e., average RMSE - average ensemble spread (average coverage probability $f_{ \text{cv}}$), of the ML-EnCMF compared with the LL-EnCMF and EnKF with $\Delta T_{\mathrm{obs}} = 0.4$

 $N$ 100 150 200 300 400 ML-EnCMF 0.84 - 0.70 0.79 - 0.67 0.75 - 0.67 0.72 - 0.64 0.69 - 0.63 (0.94) (0.94) (0.94) (0.94) (0.94) EnKF 0.88 - 0.71 0.85 - 0.73 0.83 - 0.74 0.84 - 0.76 0.83 - 0.78 (0.92) (0.93) (0.94) (0.94) (0.95) LL-EnCMF 1.21 - 0.74 0.92 - 0.73 0.88 - 0.76 0.74 - 0.68 0.70 - 0.69 (0.89) (0.93) (0.95) (0.95) (0.96)

Figures(4)

Tables(3)