Extended neural delay differential equations

Figure 1.  Sketchy diagrams comparing ENDDEs with NDDEs, including the initial function $ \boldsymbol{\phi}(t) $. Both ENDDEs and NDDEs act as feature extractors, with the subsequent neural network layer processing the extracted features using a predefined loss function

Figure 2.  (a) The data at time $ t = 0 $. (b) The transformed data of ENDDEs (6) at the learned terminal time $ T = 0.10 $, with both $ \tau $ and T being the parameters. (c) The transformed data of NDDEs at the fixed final time $ T = 1.00 $. In both (b) and (c), the transformed data are linearly separable

Figure 3.  Evolutions of the ENDDEs in the feature space during the training procedure on fitting the function $ \boldsymbol{G}(x) $ for $ d = 2 $

Figure 4.  The training losses (a) and the number of function evaluations (NFE) (b) of the NODEs, the NDDEs and the ENDDEs on fitting the function $ \boldsymbol{G}(x) $ for $ d = 2 $

Figure 5.  Decision boundaries and the flows of ENDDE (a-d), NDDE (e-h) and NODE (i-l) on a concentric dataset in different training epochs. The flow of the NODEs is generated by the code provided in [5]

Figure 6.  System reconstruction of the population dynamics by using the ENDDEs in the model-based case. The identified $ r $ and $ \tau $ are shown in panels respectively. Here, we normalize the learned parameter at different training epochs by dividing the true value. Specifically, these parameters are initialized in different deviation levels, i.e., $ p = (1 + DL)p $, where $ p\equiv r $, or $ \tau $, and $ DL $ is selected from the set $ \{0.2, 0.4, ..., 1.0\} $

Figure 7.  System identification of the Mackey-Glass system by using the ENDDEs. The identified $ \beta $, $ n $, $ \gamma $, and $ \tau $ are shown in panels (a), (b), (c), and (d), respectively. Here, we normalize the learned parameter at different training epochs by dividing the true value. Specifically, these parameters are initialized in different deviation levels, i.e., $ p = (1 + DL)p $, where $ p\equiv \beta, n, \gamma $, or $ \tau $, and $ DL $ is selected from the set $ \{0.2, 0.4, ..., 1.0\} $

Figure 8.  Inferring the underlying delay of the Mackey-Glass system by using the ENDDEs in a model-free manner. The learned delays are shown in (a) and (b) with 8 and 16 hidden neurons, respectively, and we use 3 hidden layers and the $ \tanh $ activation function. $ DL $ is selected from the set $ \{0.2, 0.4, 0.6, 0.8\} $

Figure 9.  Reconstruction results of the model-based ENDDEs (a-d) and model-free ENDDEs (e-h) for 2-D time series when considering the delay effect of the original system. From left to right the subplots represent: (a, e) Comparisons between the true (solid lines) and fitted (dashed lines) time series; (b, f) Comparisons between the true and fitted trajectories in the phase space; (c, g) Dynamic evolution processes of the trainable parameters in the neural network; (d, h) Dynamic variation curves of loss during training

Figure 10.  The training loss (a), the test loss (b), the accuracy (c), the normalized NFE (i.e., $ \langle NFE \rangle = NFE / T $) (d), the mean value (e) and the standard deviation (f) of $ |\boldsymbol{w}| $ over 4 realizations for NODEs and ENDDEs with different fixed terminal time $ T $ (1 or 4) on CIFAR10