Recent developments in machine learning methods for stochastic control and games

Figure 1.  A case study of the COVID-19 pandemic in three states: New York (NY), New Jersey (NJ), and Pennsylvania (PA) in [294]. Plots of optimal policies (top-left), Susceptibles (top-right), Exposed (bottom-left), and Infectious (bottom-right) for three states: New York (blue), New Jersey (orange), and Pennsylvania (green). Large $ \ell $ indicates high intensity of lockdown policy. Choices of parameters are referred to [294, Section 4.2]

Figure 2.  The illustrative linear quadratic model in Section 1.2. Panels (a) and (b) give three trajectories of $ X_t $, $ m_t = \mathbb{E}[X_t \vert \mathcal{F}_t^{W^0}] $ (solid lines) and their approximations $ \widehat X_t $ (dashed lines) using different realizations of $ (X_0, W, W^0) $ from validation data. Panel (c) shows the minimized cost computed using validation data over fictitious play iterations. Parameter choices are given in [243, Section 5]

Figure 3.  The linear-quadratic regulator problem with delay in Section 2.6.1. Left: Training curve of two models in the example of linear-quadratic problem. Right: The effect of lag time $ \bar{\delta} $ processed by the feedforward model in the example of the linear-quadratic problem. The lag time $ \delta $ in the actual system is $ 1 $

Figure 4.  The linear-quadratic regulator problem with delay in Section 2.6.1. A sample path of the first 5 dimensions of the state $ X_t $ and control $ \alpha_t $ obtained from the LSTM (top) model and FNN (top) model. Left: the optimal state process discretized from the analytical solution $ (X_t)_i $ (solid lines) and its approximation $ (\hat{X}_t)_i $ (dashed lines) provided by the approximating control, under the same realized path of Brownian motion. Right: comparisons of the optimal control $ ( \alpha_t)_i $ (solid lines) and $ (\hat{ \alpha}_t)_i $ (dashed lines)

Figure 5.  Price impact MFC example in Section 2.6.2 solved by direct method. Left: Control learnt (dots) and exact solution (lines). Right: associated empirical state distribution. Here we take $ \gamma = 0.2 $ in (24)

Figure 6.  Price impact MFC example in Section 2.6.2 solved by direct method. Left: Control learned (dots) and exact solution (lines). Right: associated empirical state distribution. Here, $ \gamma = 1 $ in (24)

Figure 7.  Comparisons of cost functions and optimal trajectories for $ N = 24 $ players in the linear quadratic systemic risk problem in Section 3.1.2. Left: the maximum relative errors of the cost functions for 24 players; Right: for the sake of clarity, the comparison of optimal trajectories is only presented for the $ 1^\text{st} $, $ 4^\text{th} $, $ 7^\text{th} $, $ 10^\text{th} $, $ 13^\text{th} $, $ 16^\text{th} $, $ 19^\text{th} $ and $ 22^\text{th} $ players, where the solid lines are given by the closed-form solution and the stars are computed by deep fictitious play

Figure 8.  Comparisons of trajectories for $ N = 24 $ players in the linear quadratic game in Section 3.1.2 using the learnt equilibrium strategy profile. For the sake of clarity, we only show the mean (blue triangles) and standard deviation (red bars) of trajectories errors for the $ 1^\text{st} $, $ 4^\text{th} $, $ 7^\text{th} $, $ 10^\text{th} $, $ 11^\text{th} $, $ 13^\text{th} $, $ 16^\text{th} $, $ 19^\text{th} $ and $ 22^\text{th} $ player, respectively. The results are based on a total of $ 65536 $ sample paths. They show that deep fictitious play provides a relatively uniformly good accuracy

Figure 9.  Comparisons of optimal controls for $ N = 24 $ players in the linear quadratic game in Section 3.1.2. For clarity, we only show two sample paths of optimal controls for the $ 1^\text{st} $, $ 4^\text{th} $, $ 7^\text{th} $, $ 10^\text{th} $, $ 11^\text{th} $, $ 13^\text{th} $, $ 16^\text{th} $, $ 19^\text{th} $ and $ 22^\text{th} $ player, respectively. The solid lines are optimal controls given by the closed-form solution, and the dotted dash lines are computed by deep fictitious play

Figure 10.  Linear-quadratic systemic risk example in Section 3.1.3. The relative squared errors of $ u^i $ (left) and $ \nabla u^i $ (right) along the training process of deep fictitious play for the inter-bank game. The relative squared errors of $ u^i(0, \check {\mathit{\boldsymbol{X}}}_0^{i}) $ and $ \{\nabla u^i(t_n, \check {\mathit{\boldsymbol{X}}}_n^{i})\}_{n = 0}^{N_{T-1}} $ are evaluated

Figure 11.  Linear-quadratic systemic risk example in Section 3.1.3. A sample path for each player of the inter-bank game with $ N = 10 $. Top: the optimal state process $ X_t^i $ (solid lines) and its approximation $ \hat{X}_t^i $ (circles) provided by the optimized neural networks, under the same realized path of Brownian motion. Bottom: comparisons of the strategies $ \alpha_t^i $ and $ \hat{\alpha}_t^i $ (dashed lines)

Figure 12.  Flowchart of one iteration in the Sig-DFP Algorithm. Input: idiosyncratic noise $ W $, common noise $ W^0 $, initial position $ X_0 $ and vector $ \hat{\nu}^{(\mathtt{{k}}-1)} $ from the last iteration. Output: vector $ \hat{\nu}^{(\mathtt{{k}})} $ for the next iteration

Figure 13.  MFG of optimal consumption and investment in Section 3.2.2. Panels (a) and (b) give three trajectories of $ X_t $ and $ m_t = \exp\bigl( \mathbb{E}(\log X_t | \mathcal{F}^0_t)\bigr) $ (solid lines) and their approximations $ \hat X_t $ and $ \hat m_t $ (dashed lines) using different $ (X_0, W, W^0) $ from validation data. Panel (c) shows the maximized utility computed using validation data over fictitious play iterations. Parameter choices are: $ \delta \sim U(2, 2.5), b \sim U(0.25, 0.35), \sigma\sim U(0.2, 0.4), \theta, \xi \sim U(0, 1), \sigma^0\sim U(0.2, 0.4) $, $ \epsilon\sim U(0.5, 1) $

Figure 14.  Systemic risk MFG example solved by the algorithm described in Section 3.2.3. Left: three sample trajectories of $ X $ using the neural network approach ('Deep Solver' with full lines, in cyan, blue, and green) or using the analytical formula ('benchmark' with dashed lines, in orange, red and purple). Right: three sample trajectories of $ Y $ (similar labels and colors). Note that the analytical formula satisfies the true terminal condition, whereas the solution computed by neural networks satisfies it only approximately since the trajectories are generated in a forward way starting from the learned initial condition

Figure 15.  Trade crowding MFG example in Section 3.2.4 solved by DGM. Evolution of the distribution $ m $. Left: surface with the horizontal axes representing time and space and the vertical axis representing the value of the density. Right: contour plot of the density with a dashed red line corresponding to the mean of the density computed by the semi-explicit formula

Figure 16.  Trade crowding MFG example in Section 3.2.4 solved by DGM. Each plot corresponds to the control at a different time step: Optimal control $ \alpha^* $ (dashed line) and learned control (full line)

Figure 17.  MFG Cybersecurity example in Section 3.2.4. Test case 1: Evolution of the distribution $ m^{m_0} $ (left) and the value function $ u^{m_0} $ and $ \mathcal{U}(\cdot, \cdot, m^{m_0}(\cdot)) $ (right) for $ m_0 = (1/4, 1/4, 1/4, 1/4) $. First published in [223] by the American Mathematical Society

Figure 18.  MFG Cybersecurity example in Section 3.2.4. Test case 2: Evolution of the distribution $ m^{m_0} $ (left) and the value function $ u^{m_0} $ and $ \mathcal{U}(\cdot, \cdot, m^{m_0}(\cdot)) $ (right) for $ m_0 = (1, 0, 0, 0) $. First published in [223] by the American Mathematical Society

Figure 19.  MFG Cybersecurity example in Section 3.2.4. Test case 3: Evolution of the distribution $ m^{m_0} $ (left) and the value function $ u^{m_0} $ and $ \mathcal{U}(\cdot, \cdot, m^{m_0}(\cdot)) $ (right) for $ m_0 = (0, 0, 0, 1) $. First published in [223] by the American Mathematical Society

Figure 20.  Cybersecurity MFC model solved with DDPG in Section 4.1.2: Evolution of the population distribution for five initial distributions

Figure 21.  MFG described in Section 4.2.2, solved with fictitious play and DDPG. Left: $ L^2 $ error on the analytical control; right: stationary distribution. Results obtained by 125 iterations of fictitious play