Master equations for finite state mean field games with nonlinear activations

Yuan Gao; Jian-Guo Liu; Wuchen Li

doi:10.3934/dcdsb.2023204

Article Contents

2024, Volume 29, Issue 7: 2837-2879. Doi: 10.3934/dcdsb.2023204

This issue Previous Article The global stability of 3-D axisymmetric solutions to compressible viscous and heat-conductive fluids Next Article Finite-dimensional global attractor for the three-dimensional viscous Camassa-Holm equations with fractional diffusion on bounded domains

Master equations for finite state mean field games with nonlinear activations

1.
Department of Mathematics, Purdue University, West Lafayette, IN, USA
2.
Department of Mathematics and Department of Physics, Duke University, Durham, NC, USA
3.
Department of Mathematics, University of South Carolina, Columbia, SC, USA

^*Corresponding author: Yuan Gao
^*Corresponding author: Yuan Gao

Received: June 2023

Revised: October 2023

Early access: December 15, 2023

Published online: December 15, 2023

YG is partially supported by NSF under award DMS-2204288. JGL is partially supported by NSF under award DMS-2106988. WL is supported by AFOSR MURI FA9550-18-1-0502, AFOSR YIP award FA9550-23-1-0087, and NSF RTG: 2038080.

Abstract / Introduction Full Text(HTML) Related Papers Cited by

Abstract

We formulate a class of mean field games on a finite state space with variational principles resembling those in continuous-state mean field games. We construct a controlled continuity equation featuring a nonlinear activation function on graphs induced by finite-state reversible continuous time Markov chains. In these graphs, each edge is weighted by the transition probability and invariant measure of the original process. Using these controlled dynamics on the graph and the dynamic programming principle for the value function, we derive several key components: the mean field game systems, the functional Hamilton-Jacobi equations, and the master equations on a finite probability space for potential mean field games. The existence and uniqueness of solutions to the potential mean field game system are ensured through a convex optimization reformulation in terms of the density-flux pair. We also derive variational principles for the master equations of both non-potential games and mixed games on a continuous state space. Finally, we offer several concrete examples of discrete mean field game dynamics on a two-point space, complete with closed-formula solutions. These examples include discrete Wasserstein distances, mean field planning, and potential mean field games.

Keywords:

Value function,
Reversible Markov chains,
Onsager principle,
Nash equilibrium,
Optimal control,
Hamilton-Jacobi equations on Wasserstein spaces.

Mathematics Subject Classification: Primary: 49N80, 49L12; Secondary: 60J20, 34K35.

Citation:

Full Text(HTML)

Related Papers

Cited by

References

[1]	L. Ambrosio, N. Gigli and G. Savaré, Gradient Flows in Metric Spaces and in the Space of Probability Measures, Birkhäuser Verlag, Basel, 2005.
[2]	M. Bardi and P. Cardaliaguet, Convergence of some mean field games systems to aggregation and flocking models, Nonlinear Analysis, 204 (2021), Paper No. 112199, 24 pp. doi: 10.1016/j.na.2020.112199.
[3]	E. Bayraktar and A. Cohen, Analysis of a finite state many player game using its master equation, SIAM Journal on Control and Optimization, 56 (2018), 3538-3568. doi: 10.1137/17M113887X.
[4]	J. D. Benamou and Y. Brenier, A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem, Numerische Mathematik, 84 (2000), 375-393. doi: 10.1007/s002110050002.
[5]	C. Bertucci, J. Lasry and P. Lions, Some remarks on mean field games, Communications in Partial Differential Equations, 44 (2019), 205-227. doi: 10.1080/03605302.2018.1542438.
[6]	P. Cardaliaguet, Notes on mean field games, Technical report, 2010.
[7]	P. Cardaliaguet, F. Delarue, J. Lasry and P. Lions, The Master Equation and the Convergence Problem in Mean Field Games, Princeton University Press, Princeton, NJ, 2019. doi: 10.2307/j.ctvckq7qf.
[8]	P. Cardaliaguet and A. Porretta, An introduction to mean field game theory, Mean Field Games, Springer, Cham, 2281 (2020), 1-158. doi: 10.1007/978-3-030-59837-2_1.
[9]	R. Carmona and F. Delarue, Probabilistic Theory of Mean Field Games with Applications Ⅰ-Ⅱ, Probab. Theory Stoch. Model., 84, Springer, Cham, 2018.
[10]	A. Cecchin and M. Fischer, Probabilistic approach to finite state mean field games, Appl Math Optim, 81 (2020), 253-300. doi: 10.1007/s00245-018-9488-7.
[11]	A. Cecchin and G. Pelino, Convergence, fluctuations and large deviations for finite state mean field games via the master equation, Stochastic Processes and their Applications, 129 (2019), 4510-4555. doi: 10.1016/j.spa.2018.12.002.
[12]	S. N. Chow, W. Huang, Y. Li and H. Zhou, Fokker–Planck equations for a free energy functional or Markov process on a graph, Archive for Rational Mechanics and Analysis, 203 (2012), 969-1008. doi: 10.1007/s00205-011-0471-6.
[13]	S. N. Chow, W. Li, J. Lu and H. Zhou, Population games and discrete optimal transport, J Nonlinear Sci., 29 (2019), 871-896. doi: 10.1007/s00332-018-9507-5.
[14]	S. N. Chow, W. Li and H. Zhou, Entropy dissipation of Fokker-Planck equations on finite graphs, Discrete and Continuous Dynamical Systems-A, 38 (2018), 4929-4950. doi: 10.3934/dcds.2018215.
[15]	Y.-L. Chuang, Y. R. Huang, M. R. D'Orsogna and A. L. Bertozzi, Multi-vehicle flocking: Scalability of cooperative control algorithms using pairwise potentials, IEEE International Conference on Robotics and Automation, (2007), 2292-2299.
[16]	P. Degond, J. G. Liu and C. Ringhofer, Large-scale dynamics of mean field games driven by local Nash equilibria, Journal of Nonlinear Science, 24 (2014), 93-115. doi: 10.1007/s00332-013-9185-2.
[17]	P. Degond, H. Michael and J. G. Liu, Mean field games and model predictive control, Communications in Mathematical Sciences, 15 (2017), 1403-1422. doi: 10.4310/CMS.2017.v15.n5.a9.
[18]	F. Delarue, Master equation for finite state mean field games with additive common noise, Mean Field Games, Lecture Notes in Mathematics, Springer, Cham, 2281 (2020), 203-248. doi: 10.1007/978-3-030-59837-2_3.
[19]	M. Erbar, Gradient flows of the entropy for jump processes, In Annales de l'IHP Probabilités et Statistiques, 50 (2014), 920-945. doi: 10.1214/12-AIHP537.
[20]	M. Erbar and J. Maas, Ricci curvature of finite Markov chains via convexity of the entropy, Archive for Rational Mechanics and Analysis, 206 (2012), 997-1038. doi: 10.1007/s00205-012-0554-z.
[21]	W. H. Fleming and H. M. Soner, Controlled Markov Processes and Viscosity Solutions, Springer Science & Business Media, New York, 2nd edition, 2006.
[22]	W. Gangbo, On some Analytical Aspects of Mean Field Games,, 2018.
[23]	W. Gangbo, W. Li and C. Mou, Geodesic of minimal length in the set of probability measures on graphs, ESAIM: COCV, 25 (2019), Paper No. 78, 36 pp.. doi: 10.1051/cocv/2018052.
[24]	W. Gangbo and A. Mészáros, Global well-posedness of master equations for deterministic displacement convex potential mean field games, Communications on Pure and Applied Mathematics, 75 (2022), 2685-2801.
[25]	W. Gangbo, A. Mészáros, C. Mou and J. Zhang, Mean field games master equations with nonseparable Hamiltonians and displacement monotonicity, Ann. Probab., 50 (2022), 2178-2217. doi: 10.1214/22-aop1580.
[26]	W. Gangbo, C. Mou and A. Swiech, Well-posedness for Hamilton-Jacobi equations on the Wasserstein space on graphs, Preprint, 2023.
[27]	W. Gangbo, T. Nguyen and A. Tudorascu, Hamilton-Jacobi equations in the Wasserstein space, Meth. Appl. Anal., 15 (2008), 155-183. doi: 10.4310/MAA.2008.v15.n2.a4.
[28]	W. Gangbo and A. Swiech, Existence of a solution to an equation arising from the theory of mean field games, Journal of Differential Equations, 259 (2015), 6573-6643. doi: 10.1016/j.jde.2015.08.001.
[29]	Y. Gao, T. Li, X. Li and J.-G. Liu, Transition path theory for Langevin dynamics on manifold: optimal control and data-driven solver, Multiscale Modeling and Simulation, 21 (2023), 1-33. doi: 10.1137/21M1437883.
[30]	Y. Gao and J.-G. Liu, Revisit of macroscopic dynamics for non-equilibrium chemical reactions from a Hamiltonian viewpoint, Journal of Statistical Physics, 189 (2022), 1-57. doi: 10.1007/s10955-022-02985-5.
[31]	Y. Gao, J.-G. Liu and O. Tse, Optimal control formulation of transition path problems for Markov Jump Processes, Preprint arXiv: 2311.07795, 2023.
[32]	Y. Gao, J.-G. Liu and N. Wu, Data-driven efficient solvers for Langevin dynamics on manifold in high dimensions, Applied and Computational Harmonic Analysis, 62 (2023), 261-309. doi: 10.1016/j.acha.2022.09.003.
[33]	D. A. Gomes, J. Mohr and R. R. Souza, Discrete time, finite state space mean field games, Journal de Mathématiques Pures et Appliquées, 93 (2010), 308-328. doi: 10.1016/j.matpur.2009.10.010.
[34]	D. A. Gomes, J. Mohr and R. R. Souza, Continuous time finite state mean field games, Applied Mathematics & Optimization, 68 (2013), 99-143. doi: 10.1007/s00245-013-9202-8.
[35]	D. A. Gomes, L. Nurbekyan and M. Prazeres, One-dimensional stationary mean-field games with local coupling, Dyn Games Appl, 8 (2018), 315-351. doi: 10.1007/s13235-017-0223-9.
[36]	O. Guéant, Mean field games on graphs, NETCO, 2014.
[37]	O. Guéant, Existence and uniqueness result for mean field games with congestion effect on graphs, Applied Mathematics & Optimization, 72 (2015), 291-303. doi: 10.1007/s00245-014-9280-2.
[38]	O. Guéant, J. Lasry and P. L. Lions, Mean field games and applications, Paris-Princeton Lectures on Mathematical Finance, 2003 (2010), 205-266. doi: 10.1007/978-3-642-14660-2_3.
[39]	P. Hanggi, H. Grabert, P. Talkner and H. Thomas, Bistable systems: Master equation versus Fokker-Planck modeling, Physical Review A, 29 (1984), 371-378. doi: 10.1103/PhysRevA.29.371.
[40]	J. Hofbauer and K. Sigmund, The Theory of Evolution and Dynamical Systems: Mathematical Aspects of Selection, Cambridge University Press, Cambridge, 1988.
[41]	M. Huang, R. P. Malhame and P. Caines, Large population stochastic dynamic games: Closed-loop Mckean-Vlasov systems and the Nash certainty equivalence principle, Communications in Information & Systems, 6 (2006), 221-252.
[42]	J. Lasry and P. Lions, Mean field games, Japanese Journal of Mathematics, 2 (2007), 229-260. doi: 10.1007/s11537-007-0657-8.
[43]	W. Li, Transport information geometry: Riemannian calculus on probability simplex, Information Geometry, 5 (2020), 161-207. doi: 10.1007/s41884-021-00059-1.
[44]	W. Li, W. Lee and S. Osher, Computational mean field information dynamics associated with reaction-diffusion equations, Journal of Computational Physics, 466 (2022), Paper No. 111409, 30 pp. doi: 10.1016/j.jcp.2022.111409.
[45]	W. Li, S. Liu and S. Osher, Controlling conservation laws Ⅰ: Entropy-entropy flux pairs, Journal of Computational Physics, 480 (2023), Paper No. 112019, 21 pp. doi: 10.1016/j.jcp.2023.112019.
[46]	W. Li, S. Liu and S. Osher, Controlling conservation laws Ⅱ: Compressible Navier-Stokes equations, Journal of Computational Physics, 463 (2022), Paper No. 111264, 21 pp. doi: 10.1016/j.jcp.2022.111264.
[47]	P. Lions, Cours au College de France, http:www.college-de-france.fr.
[48]	J. Maas, Gradient flows of the entropy for finite Markov chains, Journal of Functional Analysis, 261 (2011), 2250-2292. doi: 10.1016/j.jfa.2011.06.009.
[49]	J. Maas and A. Mielke, Modeling of chemical reaction systems with detailed balance using gradient structures, Journal of statistical physics, 181 (2020), 2257-2303. doi: 10.1007/s10955-020-02663-4.
[50]	A. Mielke, A gradient structure for reaction-diffusion systems and for energy-drift-diffusion, Nonlinearity, 24 (2011), 1329-1346. doi: 10.1088/0951-7715/24/4/016.
[51]	A. Mielke, Geodesic convexity of the relative entropy in reversible Markov chains, Calculus of Variations and Partial Differential Equations, 48 (2013), 1-31. doi: 10.1007/s00526-012-0538-8.
[52]	A. Mielke, A. P. Mark and D. R. M. Renger, On the relation between gradient flows and the large-deviation principle, with applications to Markov chains and diffusion, Potential Analysis, 41 (2014), 1293-1327. doi: 10.1007/s11118-014-9418-5.
[53]	D. Monderer and L. Shapley, Potential games, Games and Economic Behavior, 14 (1996), 124-143. doi: 10.1006/game.1996.0044.
[54]	J. F. Nash, Equilibrium points in n-person games, Proceedings of the National Academy of Sciences, 36 (1950), 48-49. doi: 10.1073/pnas.36.1.48.
[55]	L. Onsager, Reciprocal relations in irreversible processes, Ⅰ+Ⅱ. Physical Review, 37 (1931), 405-426.
[56]	C. Orrieri, A. Porretta and G. Savaré, A variational approach to the mean field planning problem, Journal of Functional Analysis, 277 (2019), 1868-1957. doi: 10.1016/j.jfa.2019.04.011.
[57]	M. A. Peletier, R. Rossi, G. Savare and O. Tse, Jump processes as generalized gradient flows, Calculus of Variations and Partial Differential Equations, 61 (2022), 1-85. doi: 10.1007/s00526-021-02130-2.
[58]	D. Slepcev and A. Warren, Nonlocal wasserstein distance: Metric and asymptotic properties, Calc. Var. Partial Differential Equations, 62 (2023), Paper No. 238, 66 pp. doi: 10.1007/s00526-023-02576-6.
[59]	C. Villani, Optimal Transport: Old and New, Number 338 in Grundlehren Der Mathematischen Wissenschaften. Springer, Berlin, 2009. doi: 10.1007/978-3-540-71050-9.
[60]	B. Zhang and M. Katsoulakis, A mean-field games laboratory for generative modeling, arXiv: 2304.13534, 2023.