Reproducing kernel Hilbert spaces in the mean field limit

Christian Fiedler; Michael Herty; Michael Rom; Chiara Segala; Sebastian Trimpe

doi:10.3934/krm.2023010

Article Contents

2023, Volume 16, Issue 6: 850-870. Doi: 10.3934/krm.2023010

This issue Previous Article Linearized Boltzmann collision operator: II. Polyatomic molecules modeled by a continuous internal energy variable Next Article Solutions of the 2D radially symmetric Vlasov-Maxwell system with an initial focusing phase

Reproducing kernel Hilbert spaces in the mean field limit

1.
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Germany
2.
Institute of Geometry and Practical Mathematics, RWTH Aachen University, Germany

^*Corresponding author: Christian Fiedler
^*Corresponding author: Christian Fiedler

Received: July 2022

Revised: February 2023

Early access: March 2023

Published: December 2023

Abstract / Introduction Full Text(HTML) Figure(3) Related Papers Cited by

Abstract

Kernel methods, being supported by a well-developed theory and coming with efficient algorithms, are among the most popular and successful machine learning techniques. From a mathematical point of view, these methods rest on the concept of kernels and function spaces generated by kernels, so–called reproducing kernel Hilbert spaces. Motivated by recent developments of learning approaches in the context of interacting particle systems, we investigate kernel methods acting on data with many measurement variables. We show the rigorous mean field limit of kernels and provide a detailed analysis of the limiting reproducing kernel Hilbert space. Furthermore, several examples of kernels, that allow a rigorous mean field limit, are presented.

Keywords:

Mathematics Subject Classification: 46E22, 82B40, 74A25, 82C40.

Citation:

Full Text(HTML)

Figure 1. Commutative diagram summarizing the relation between mean field limit (MFL) of a sequence of kernels $ (k_M)_M $ and their corresponding reproducing kernel Hilbert spaces. Here, $ f_M $ denotes an element of the space $ H_M $, and $ k,H_k $ indicate the MFL of $ (k_M)_M $ and $ (H_M)_M $, respectively. The mean field limits are given in Theorem 3.2 and Theorem 4.4., respectively

Download: Full-size image PowerPoint slide

Figure 2. Commutative diagram on the relation of canonical feature map of $ k^{[M]} $ and KMEs

Download: Full-size image PowerPoint slide

Figure 3. Diagram illustration of the relations of double sum kernel, KME and MFL

Download: Full-size image PowerPoint slide

Related Papers

Cited by

References

[1]	G. Albi, N. Bellomo, L. Fermo, S.-Y. Ha, J. Kim, L. Pareschi, D. Poyato and J. Soler, Vehicular traffic, crowds, and swarms: From kinetic theory and multiscale methods to applications and research perspectives, Math. Models Methods Appl. Sci., 29 (2019), 1901-2005. doi: 10.1142/S0218202519500374.
[2]	N. Bellomo, P. Degond and E. Tadmor, Active Particles, Volume 1: Advances in Theory, Models, and Applications, Birkhäuser, 2017.
[3]	N. Bellomo and J. Soler, On the mathematical theory of the dynamics of swarms viewed as complex systems, Math. Models Methods Appl. Sci., 22 (2012), 1140006, 29 pp. doi: 10.1142/S0218202511400069.
[4]	A. Benfenati, G. Borghi and L. Pareschi, Binary interaction methods for high dimensional global optimization and machine learning, Appl. Math. Optim., 86 (2022), Paper No. 9, 41 pp. doi: 10.1007/s00245-022-09836-5.
[5]	A. Blanchet and G. Carlier, From nash to cournot–nash equilibria via the monge–kantorovich problem, Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci., 372 (2014), 20130398, 11 pp. doi: 10.1098/rsta.2013.0398.
[6]	M. Bongini, M. Fornasier, M. Hansen and M. Maggioni, Inferring interaction rules from observations of evolutive systems Ⅰ: The variational approach, Mathematical Models and Methods in Applied Sciences, 27 (2017), 909-951. doi: 10.1142/S0218202517500208.
[7]	L. Boudin and F. Salvarani, A kinetic approach to the study of opinion formation, M2AN Math. Model. Numer. Anal., 43 (2009), 507-522. doi: 10.1051/m2an/2009004.
[8]	P. Buathong, D. Ginsbourger and T. Krityakierne, Kernels over sets of finite sets using rkhs embeddings, with application to bayesian (combinatorial) optimization, International Conference on Artificial Intelligence and Statistics, PMLR, (2020), 2731-2741.
[9]	J. A. Canizo, J. A. Carrillo and J. Rosado, A well-posedness theory in measures for some kinetic models of collective motion, Math. Models Methods Appl. Sci., 21 (2011), 515-539. doi: 10.1142/S0218202511005131.
[10]	P. Cardaliaguet, Notes on Mean Field Games, tech. rep., Technical report, 2010.
[11]	R. Carmona and F. Delarue, Probabilistic Theory of Mean Field Games with Applications Ⅰ-Ⅱ, Probability Theory and Stochastic Modelling, 84. Springer, Cham, 2018
[12]	J. A. Carrillo, Y.-P. Choi and M. Hauray, The derivation of swarming models: Mean-field limit and Wasserstein distances, Collective Dynamics from Bacteria to Crowds, CISM Courses and Lect., Springer, Vienna, 553 (2014), 1-46. doi: 10.1007/978-3-7091-1785-9_1.
[13]	J. A. Carrillo, M. Fornasier, G. Toscani and F. Vecil, Particle, kinetic, and hydrodynamic models of swarming, Mathematical Mmodeling of Ccollective Behavior in Socio-Economic and Life Sciences, Springer, (2010), 297-336. doi: 10.1007/978-0-8176-4946-3_12.
[14]	J. A. Carrillo, S. Jin, L. Li and Y. Zhu, A consensus-based global optimization method for high dimensional machine learning problems, ESAIM Control Optim. Calc. Var., 27 (2021), Paper No. S5, 22 pp. doi: 10.1051/cocv/2020046.
[15]	C. Cercignani, R. Illner and M. Pulvirenti, The Mathematical Theory of Dilute Gases, vol. 106, Springer Science & Business Media, 2013.
[16]	S. Cordier, L. Pareschi and G. Toscani, On a kinetic model for a simple market economy, J. Stat. Phys., 120 (2005), 253-277. doi: 10.1007/s10955-005-5456-0.
[17]	E. Cristiani, B. Piccoli and A. Tosin, Multiscale Modeling of Pedestrian Dynamics, MS & A. Modeling, Simulation and Applications, 12. Springer, Cham, 2014. doi: 10.1007/978-3-319-06620-2.
[18]	F. Cucker and S. Smale, Emergent behavior in flocks, IEEE Trans. Automat. Control, 52 (2007), 852-862. doi: 10.1109/TAC.2007.895842.
[19]	P. Degond, M. Herty and J.-G. Liu, Flow on sweeping networks, Multiscale Model. Simul., 12 (2014), 538-565. doi: 10.1137/130927061.
[20]	P. Degond, J.-G. Liu, S. Motsch and V. Panferov, Hydrodynamic models of self-organized dynamics: Derivation and existence theory, Methods Appl. Anal., 20 (2013), 89-114. doi: 10.4310/MAA.2013.v20.n2.a1.
[21]	P. Degond and S. Motsch, Continuum limit of self-driven particles with orientation interaction, Math. Models Methods Appl. Sci., 18 (2008), 1193-1215. doi: 10.1142/S0218202508003005.
[22]	M. R. D'Orsogna, Y.-L. Chuang, A. L. Bertozzi and L. S. Chayes, Self-propelled particles with soft-core interactions: Patterns, stability, and collapse, Phys. Rev. Lett., 96 (2006), 104302, 33-47.
[23]	G. Estrada-Rodriguez and H. Gimperlein, Interacting particles with Lévy strategies: Limits of transport equations for swarm robotic systems, SIAM J. Appl. Math., 80 (2020), 476-498. doi: 10.1137/18M1205327.
[24]	M. Fornasier, J. Haskovec and G. Toscani, Fluid dynamic description of flocking via the Povzner-Boltzmann equation, Phys. D, 240 (2011), 21-31. doi: 10.1016/j.physd.2010.08.003.
[25]	M. Fornasier, H. Huang, L. Pareschi and P. Sünnen, Consensus-based optimization on the sphere: Convergence to global minimizers and machine learning, J. Mach. Learn. Res., 22 (2021), Paper No. 237, 55 PP.
[26]	T. Gärtner, P. A. Flach, A. Kowalczyk and A. J. Smola, Multi-instance kernels, ICML, 2 (2002), 7 pp.
[27]	L. Gibelli, Crowd Dynamics, Volume 2: Theory, Models, and Applications, Modeling and Simulation in Science, Engineering and Technology, Birkhäuser/Springer, Cham, 2020. doi: 10.1007/978-3-030-50450-2.
[28]	L. Gibelli and N. Bellomo, Crowd Dynamics, Volume 1: Theory, Models, and Safety Problems, Springer, 2019.
[29]	J. Gómez-Serrano, C. Graham and J.-Y. Le Boudec, The bounded confidence model of opinion dynamics, Math. Models Methods Appl. Sci., 22 (2012), 1150007, 46 pp. doi: 10.1142/S0218202511500072.
[30]	S.-Y. Ha and E. Tadmor, From particle to kinetic and hydrodynamic descriptions of flocking, Kinet. Relat. Models, 1 (2008), 415-435. doi: 10.3934/krm.2008.1.415.
[31]	M. Herty and L. Pareschi, Fokker-Planck asymptotics for traffic flow models, Kinet. Relat. Models, 3 (2010), 165-179. doi: 10.3934/krm.2010.3.165.
[32]	M. Herty, L. Pareschi and G. Visconti, Mean field models for large data-clustering problems, Netw. Heterog. Media, 15 (2020), 463-487. doi: 10.3934/nhm.2020027.
[33]	M. Herty and C. Ringhofer, Averaged kinetic models for flows on unstructured networks, Kinet. Relat. Models, 4 (2011), 1081-1096. doi: 10.3934/krm.2011.4.1081.
[34]	M. Herty, T. Trimborn and G. Visconti, Mean-field and kinetic descriptions of neural differential equations, Foundations of Data Science, 4 (2022), 271-298.
[35]	M. Herty and G. Visconti, Continuous limits for constrained ensemble Kalman filter, Inverse Problems, 36 (2020), 075006, 28 pp. doi: 10.1088/1361-6420/ab8bc5.
[36]	M. Herty and M. Zanella, Performance bounds for the mean-field limit of constrained dynamics, Discrete Contin. Dyn. Syst., 37 (2017), 2023-2043. doi: 10.3934/dcds.2017086.
[37]	M. Kanagawa, P. Hennig, D. Sejdinovic and B. K. Sriperumbudur, Gaussian processes and kernel methods: A review on connections and equivalences, arXiv preprint, (2018), arXiv: 1807.02582.
[38]	J. Kim, M. McCourt, T. You, S. Kim and S. Choi, Bayesian optimization with approximate set kernels, Machine Learning, 110 (2021), 857-879. doi: 10.1007/s10994-021-05949-0.
[39]	F. Lu, M. Maggioni and S. Tang, Learning interaction kernels in heterogeneous systems of agents from multiple trajectories, J. Mach. Learn. Res., 22 (2021), Paper No. 32, 67 pp.
[40]	———————, Learning interaction kernels in stochastic systems of interacting particles from multiple trajectories, Foundations of Computational Mathematics, (2021), 1-55.
[41]	F. Lu, M. Zhong, S. Tang and M. Maggioni, Nonparametric inference of interaction laws in systems of agents from trajectory data, Proceedings of the National Academy of Sciences, 116 (2019), 14424-14433. doi: 10.1073/pnas.1822012116.
[42]	S. Mei, T. Misiakiewicz and A. Montanari, Mean-field theory of two-layers neural networks: Dimension-free bounds and kernel limit, Conference on Learning Theory, PMLR, (2019), 2388-2464.
[43]	S. Motsch and E. Tadmor, Heterophilious dynamics enhances consensus, SIAM Review, 56 (2014), 577-621. doi: 10.1137/120901866.
[44]	K. Muandet, K. Fukumizu, B. Sriperumbudur, B. Schölkopf, et al., Kernel mean embedding of distributions: A review and beyond, Foundations and TrendsⓇ in Machine Learning, 10 (2017), 1-141.
[45]	L. Pareschi and G. Toscani, Interacting Mmulti-Agent Systems. Kinetic Equations & Monte Carlo Methods, Oxford University Press, USA, 2013.
[46]	L. Pareschi and G. Toscani, Interacting Multiagent Systems: Kinetic Equations and Monte Carlo Methods, OUP Oxford, 2013.
[47]	V. I. Paulsen and M. Raghupathi, An Introduction to the Theory of Reproducing Kernel Hilbert Spaces, Cambridge Studies in Advanced Mathematics, 152. Cambridge University Press, Cambridge, 2016. doi: 10.1017/CBO9781316219232.
[48]	R. Pinnau, C. Totzeck, O. Tse and S. Martin, A consensus-based model for global optimization and its mean-field limit, Math. Models Methods Appl. Sci., 27 (2017), 183-204. doi: 10.1142/S0218202517400061.
[49]	B. Schölkopf, R. Herbrich and A. J. Smola, A generalized representer theorem, Computational Learning Theory (Amsterdam, 2001), Lecture Notes in Comput. Sci., Lecture Notes in Artificial Intelligence, Springer, Berlin, 2111 (2001), 416–426. doi: 10.1007/3-540-44581-1_27.
[50]	B. Scholkopf and A. J. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization, and BAeyond, MIT Press, 2002.
[51]	J. Shawe-Taylor and N. Cristianini, Kernel Methods for Pattern Analysis, Cambridge University Press, 2004.
[52]	C.-J. Simon-Gabriel and B. Schölkopf, Kernel Distribution Embeddings: Universal Kernels, Characteristic Kernels and Kernel Metrics on Distributions, The Journal of Machine Learning Research, 19 (2018), Paper No. 44, 29 pp.
[53]	B. K. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf and G. R. Lanckriet, Hilbert space embeddings and metrics on probability measures, The Journal of Machine Learning Research, 11 (2010), 1517-1561.
[54]	I. Steinwart and A. Christmann, Support Vector Machines, Springer Science & Business Media, 2008.
[55]	G. Toscani, Kinetic models of opinion formation, Commun. Math. Sci., 4 (2006), 481-496. doi: 10.4310/CMS.2006.v4.n3.a1.
[56]	C. K. Williams and C. E. Rasmussen, Gaussian processes for machine learning, Adaptive Computation and Machine Learning, MIT Press, Cambridge, MA, 2006.