Deep Learning approximation of diffeomorphisms via linear-control systems

Alessandro Scagliotti

doi:10.3934/mcrf.2022036

Article Contents

2023, Volume 13, Issue 3: 1226-1257. Doi: 10.3934/mcrf.2022036

This issue Previous Article Input-to-state stability of non-autonomous infinite-dimensional control systems Next Article

Deep Learning approximation of diffeomorphisms via linear-control systems

Alessandro Scagliotti^,

Scuola Internazionale Superiore di Studi Avanzati, Trieste, Italy

^*Corresponding author: Alessandro Scagliotti
^*Corresponding author: Alessandro Scagliotti

Received: November 2021

Revised: July 2022

Early access: August 2022

Published: September 2023

The first author is partially supported by INdAM–GNAMPA.

Abstract / Introduction Full Text(HTML) Figure(3) / Table(6) Related Papers Cited by

Abstract

In this paper we propose a Deep Learning architecture to approximate diffeomorphisms diffeotopic to the identity. We consider a control system of the form $ \dot x = \sum_{i = 1}^lF_i(x)u_i $, with linear dependence in the controls, and we use the corresponding flow to approximate the action of a diffeomorphism on a compact ensemble of points. Despite the simplicity of the control system, it has been recently shown that a Universal Approximation Property holds. The problem of minimizing the sum of the training error and of a regularizing term induces a gradient flow in the space of admissible controls. A possible training procedure for the discrete-time neural network consists in projecting the gradient flow onto a finite-dimensional subspace of the admissible controls. An alternative approach relies on an iterative method based on Pontryagin Maximum Principle for the numerical resolution of Optimal Control problems. Here the maximization of the Hamiltonian can be carried out with an extremely low computational effort, owing to the linear dependence of the system in the control variables. Finally, we use tools from $ \Gamma $-convergence to provide an estimate of the expected generalization error.

Keywords:

Mathematics Subject Classification: Primary: 49M05, 49M25; Secondary: 68T07, 49J15.

Citation:

Full Text(HTML)

Figure 1. On the left we report the grid of points $ \{ x^1_0,\ldots,x^M_0 \} $ where we have evaluated the diffeomorphism $ \Psi: \mathbb{R}^2\to \mathbb{R}^2 $ defined as in (53). The picture on the right represents the transformation of the training dataset through the diffeomorphism $ \Psi $

Download: Full-size image PowerPoint slide

Figure 2. ResNet 52, 16 layers, Algorithm 1, $ \beta = 10^{-4} $. On the top-left we reported the transformation of the initial grid through the approximating diffeomorphism (red circles) and through the original one (blue circles). On the top-right, we plotted the prediction on the testing data-set provided by the approximating diffeomorphism (red crosses) and the correct values obtained through the original transformation (blue crosses). In both cases, the approximation obtained is unsatisfactory. At bottom we plotted the decrease of the training error and the testing error versus the number of iterations. Finally, the curve in magenta represents the estimate of the generalization error provided by (35)

Download: Full-size image PowerPoint slide

Figure 3. ResNet 55, 16 layers, Algorithm 1, $ \beta = 10^{-3} $. On the top-left we reported the transformation of the initial grid through the approximating diffeomorphism (red circles) and through the original one (blue circles). On the top-right, we plotted the prediction on the testing data-set provided by the approximating diffeomorphism (red crosses) and the correct values obtained through the original transformation (blue crosses). In both cases, the approximation obtained is good, and we observe that it is better where we have more data density. At bottom we plotted the decrease of the training error and the testing error versus the number of iterations. Finally, the curve in magenta represents the estimate of the generalization error provided by (35)

Download: Full-size image PowerPoint slide

Table 1. ResNet 52, $ 16 $ layers, $ 128 $ parameters, Algorithm 1. Running time $ \sim 160 $ s

$ \beta $	$ L_{\Phi_u} $	Training error	Testing error
$ 10^0 $	$ 1.19 $	$ 3.8785 $	$ 3.8173 $
$ 10^{-1} $	$ 8.40 $	$ 1.3143 $	$ 1.2476 $
$ 10^{-2} $	$ 9.32 $	$ 1.1991 $	$ 1.1451 $
$ 10^{-3} $	$ 9.37 $	$ 1.1852 $	$ 1.1330 $
$ 10^{-4} $	$ 9.37 $	$ 1.1839 $	$ 1.1318 $

| Show Table

DownLoad: CSV

Table 2. ResNet 52, $ 16 $ layers, $ 128 $ parameters, Algorithm 2. Running time $ \sim 130 $ s

$ \beta $	$ L_{\Phi_u} $	Training error	Testing error
$ 10^0 $	$ 1.19 $	$ 3.8749 $	$ 3.8157 $
$ 10^{-1} $	$ 8.40 $	$ 1.3084 $	$ 1.2455 $
$ 10^{-2} $	$ 9.32 $	$ 1.2014 $	$ 1.1486 $
$ 10^{-3} $	$ 9.33 $	$ 1.1898 $	$ 1.1387 $
$ 10^{-4} $	$ 9.33 $	$ 1.1898 $	$ 1.1379 $

| Show Table

DownLoad: CSV

Table 3. ResNet 52, $ 32 $ layers, $ 256 $ parameters, Algorithm 1. Running time $ \sim 320 $ s

$ \beta $	$ L_{\Phi_u} $	Training error	Testing error
$ 10^0 $	$ 1.19 $	$ 3.8779 $	$ 3.8168 $
$ 10^{-1} $	$ 8.40 $	$ 1.3074 $	$ 1.2425 $
$ 10^{-2} $	$ 9.26 $	$ 1.2015 $	$ 1.1477 $
$ 10^{-3} $	$ 9.34 $	$ 1.1860 $	$ 1.1352 $
$ 10^{-4} $	$ 9.34 $	$ 1.1842 $	$ 1.1332 $

| Show Table

DownLoad: CSV

Table 4. ResNet 52, $ 32 $ layers, $ 256 $ parameters, Algorithm 2. Running time $ \sim 260 $ s

$ \beta $	$ L_{\Phi_u} $	Training error	Testing error
$ 10^0 $	$ 1.19 $	$ 3.8739 $	$ 3.8148 $
$ 10^{-1} $	$ 8.35 $	$ 1.3085 $	$ 1.2449 $
$ 10^{-2} $	$ 9.23 $	$ 1.2075 $	$ 1.1538 $
$ 10^{-3} $	$ 9.26 $	$ 1.1931 $	$ 1.1416 $
$ 10^{-4} $	$ 9.26 $	$ 1.1918 $	$ 1.1404 $

| Show Table

DownLoad: CSV

Table 5. ResNet 55, $ 16 $ layers, $ 224 $ parameters, Algorithm 1. Running time $ \sim 320 $ s

$ \beta $	$ L_{\Phi_u} $	Training error	Testing error
$ 10^0 $	$ 10.14 $	$ 2.3791 $	$ 2.3036 $
$ 10^{-1} $	$ 13.84 $	$ 0.1809 $	$ 0.2314 $
$ 10^{-2} $	$ 15.64 $	$ 0.1290 $	$ 0.1784 $
$ 10^{-3} $	$ 15.83 $	$ 0.1254 $	$ 0.1747 $
$ 10^{-4} $	$ 15.86 $	$ 0.1257 $	$ 0.1751 $

| Show Table

DownLoad: CSV

Table 6. ResNet 55, $ 16 $ layers, $ 224 $ parameters, Algorithm 2. Running time $ \sim 310 $ s

$ \beta $	$ L_{\Phi_u} $	Training error	Testing error
$ 10^0 $	$ 10.78 $	$ 2.3638 $	$ 2.3910 $
$ 10^{-1} $	$ 14.32 $	$ 0.1921 $	$ 0.2422 $
$ 10^{-2} $	$ 15.43 $	$ 0.1887 $	$ 0.2347 $
$ 10^{-3} $	$ 15.56 $	$ 0.2260 $	$ 0.2719 $
$ 10^{-4} $	$ 15.59 $	$ 0.2127 $	$ 0.2564 $

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	A. Agrachev, D. Barilari and U. Boscain, A Comprehensive Introduction to Sub-Riemannian Geometry, Cambridge Studies in Advanced Mathematics, 181. Cambridge University Press, Cambridge, 2020.
[2]	A. Agrachev, Y. Baryshnikov and A. Sarychev, Ensemble controllability by Lie algebraic methods, ESAIM Control Optim. Calc. Var., 22 (2016), 921-938. doi: 10.1051/cocv/2016029.
[3]	A. A. Agrachev and Y. L. Sachkov, Control Theory from the Geometric Viewpoint, Encyclopaedia of Mathematical Sciences, 87. Control Theory and Optimization, Ⅱ. Springer-Verlag, Berlin, 2004. doi: 10.1007/978-3-662-06404-7.
[4]	A. Agrachev and A. Sarychev, Control in the spaces of ensembles of points, SIAM J. Control Optim., 58 (2020), 1579-1596. doi: 10.1137/19M1273049.
[5]	A. A. Agrachev and A. V. Sarychev, Control on the manifolds of mappings with a view to the deep learning, J. Dyn. Control Syst., (2021).
[6]	L. Ambrosio, N. Gigli and G. Savaré, Gradient Flows in Metric Spaces and in the Space of Probability Measures, Second edition, Lectures in Mathematics ETH Zürich, Birkhäuser Verlag, Basel, 2008.
[7]	Y. Bengio, P. Simard and P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE Trans. Neural Netw., 5 (1994), 157-166.
[8]	M. Benning, E. Celledoni, M. J. Erhardt, B. Owren and C. B. Schönlieb, Deep learning as optimal control problems: Models and numerical methods, J. Comput. Dyn., 6 (2019), 171-198. doi: 10.3934/jcd.2019009.
[9]	M. Bongini, M. Fornasier, F. Rossi and F. Solombrino, Mean-field pontryagin maximum principle, J. Optim. Theory Appl., 175 (2017), 1-38. doi: 10.1007/s10957-017-1149-5.
[10]	B. Bonnet, C. Cipriani, M. Fornasier and H. Huang, A measure theoretical approach to the Mean-field Maximum Principle for training NeurODEs, preprint, 2021, arXiv: 2107.08707.
[11]	F. L. Chernousko and A. A. Lyubushin, Method of successive approximations for solution of optimal control problems, Optim. Control Appl. Methods, 3 (1982), 101-114. doi: 10.1002/oca.4660030201.
[12]	E. Çinlar, Probability and Stochastics, Graduate Texts in Mathematics, 261. Springer, New York, 2011. doi: 10.1007/978-0-387-87859-1.
[13]	G. Dal Maso, An Introduction to $\Gamma$-Convergence, Progress in Nonlinear Differential Equations and their Applications, 8. Birkhäuser Boston, Inc., Boston, MA, 1993. doi: 10.1007/978-1-4612-0327-8.
[14]	W. E, A proposal on machine learning via dynamical systems, Commun. Math. Stat., 5 (2017), 1-11. doi: 10.1007/s40304-017-0103-z.
[15]	I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, Cambridge MA, 2016.
[16]	E. Haber and L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2018), 014004, 22 pp. doi: 10.1088/1361-6420/aa9a90.
[17]	K. He and J. Sun, Convolutional neural networks at constrained time cost, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2015), 5353-5360.
[18]	K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016), 770-778.
[19]	Q. Li, L. Chen, C. Tai and W. E, Maximum principle based algorithms for deep learning, J. Mach. Learn. Res., 18 (2017), Paper No. 165, 29 pp.
[20]	M. Marchi, B. Gharesifard and P. Tabuada, Training deep residual networks for uniform approximation guarantees, PMLR, 144 (2021), 677-688.
[21]	A. V. Pukhlikov, Optimal control of distributions, Comput. Math. Model., 15 (2004), 223-256. doi: 10.1023/B:COMI.0000035820.49408.56.
[22]	Y. Sakawa and Y. Shindo, On global convergence of an algorithm for optimal control, IEEE Trans. Automat. Contr., 25 (1980), 1149-1153. doi: 10.1109/TAC.1980.1102517.
[23]	A. Scagliotti, A gradient flow equation for optimal control problems with end-point cost, J. Dyn. Control Syst., (2022).
[24]	P. Tabuada, B. Gharesifard, Universal approximation power of deep neural networks via nonlinear control theory, preprint, 2020, arXiv: 2007.06007.
[25]	M. Thorpe and Y. van Gennip, Deep limits of residual neural networks, preprint, 2018, arXiv: 1810.11741.