\`x^2+y_1+z_12^34\`
Advanced Search
Article Contents
Article Contents

A practical existence theorem for reduced order models based on convolutional autoencoders

  • *Corresponding author: Nicola Rares Franco

    *Corresponding author: Nicola Rares Franco 
Abstract / Introduction Full Text(HTML) Figure(2) Related Papers Cited by
  • In recent years, deep learning has gained increasing popularity in the fields of Partial Differential Equations (PDEs) and Reduced Order Modeling (ROM), providing domain practitioners with new powerful data-driven techniques such as Physics-Informed Neural Networks (PINNs), Neural Operators, Deep Operator Networks (DeepONets) and Deep-Learning based ROMs (DL-ROMs). In this context, deep autoencoders based on Convolutional Neural Networks (CNNs) have proven extremely effective, outperforming established techniques, such as the reduced basis method, when dealing with complex nonlinear problems. However, despite the empirical success of CNN-based autoencoders, there are only a few theoretical results supporting these architectures, usually stated in the form of universal approximation theorems. In particular, although the existing literature provides users with guidelines for designing convolutional autoencoders, the subsequent challenge of learning the latent features has been barely investigated. Furthermore, many practical questions remain unanswered, e.g., the number of snapshots needed for convergence or the neural network training strategy. In this work, using recent techniques from sparse high-dimensional function approximation, we fill some of these gaps by providing a new practical existence theorem for CNN-based autoencoders when the parameter-to-solution map is holomorphic. This regularity assumption arises in many relevant classes of parametric PDEs, such as the parametric diffusion equation, for which we discuss an explicit application of our general theory.

    Mathematics Subject Classification: Primary: 35A35, 65D40; Secondary: 68T07.

    Citation:

    \begin{equation} \\ \end{equation}
  • 加载中
  • Figure 1.  A 2D convolutional layer acting on a given input (simplified setting: 1 channel at input/output, no activation nor bias). The action of the convolutional layer can be visualized either in terms of a moving filter (A) or using the equivalent matrix representation (B): in both cases, despite mapping from $ \mathbb{R}^{9} $ onto $ \mathbb{R}^{4} $, the layer only comes with 4 learnable parameters, instead of $ 9 \cdot 4 = 36 $.

    Figure 2.  Visualization of the transformation $ f\mapsto \tilde{f} $ used in Lemma A.2. The signal $ f $ is duplicated and a polynomial perturbation is added to ensure (smooth) periodicity.

  • [1] B. Adcock, S. Brugiapaglia, N. Dexter and S. Moraga, Deep neural networks are effective at learning high-dimensional Hilbert-valued functions from limited data, In Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference, Proceedings of Machine Learning Research, 145 (2022), 1-36.
    [2] B. Adcock, S. Brugiapaglia, N. Dexter and S. Moraga, Near-optimal learning of Banach-valued, high-dimensional functions via deep neural networks, preprint, arXiv: 2211.12633, 2022. doi: 10.48550/arXiv.2211.12633.
    [3] B. Adcock, S. Brugiapaglia, N. Dexter and S. Moraga, Learning smooth functions in high dimensions: From sparse polynomials to deep neural networks, In Handbook of Numerical Analysis, In press, 2024. doi: 10.1016/bs.hna.2024.05.001.
    [4] B. Adcock, S. Brugiapaglia and C. G. Webster, Sparse Polynomial Approximation of High-Dimensional Functions, Society for Industrial and Applied Mathematics, Philadelphia, PA, 2022 doi: 10.1137/1.9781611976885.
    [5] B. Adcock and N. Dexter, The gap between theory and practice in function approximation with deep neural networks, SIAM Journal on Mathematics of Data Science, 3 (2021), 624-655.  doi: 10.1137/20M131309X.
    [6] S. E. Ahmed, S. Pawar, O. San and A. Rasheed, Reduced order modeling of fluid flows: Machine learning, Kolmogorov barrier, closure modeling, and partitioning, In AIAA Aviation 2020 Forum, (2020), 2946. doi: 10.2514/6.2020-2946.
    [7] J. Barnett and C. Farhat, Quadratic approximation manifold for mitigating the Kolmogorov barrier in nonlinear projection-based model order reduction, Journal of Computational Physics, 464 (2022), 111348.  doi: 10.1016/j.jcp.2022.111348.
    [8] W. M. Boon, N. R. Franco, A. Fumagalli and P. Zunino, Deep learning based reduced order modeling of Darcy flow systems with local mass conservation, preprint, arXiv: 2311.14554, 2023. doi: 10.48550/arXiv.2311.14554.
    [9] S. Brivio, S. Fresca, N. R. Franco and A. Manzoni, Error estimates for POD-DL-ROMs: A deep learning framework for reduced order modeling of nonlinear parametrized PDEs enhanced by proper orthogonal decomposition, Adv. Comput. Math., 50 (2024), 46 pp. doi: 10.1007/s10444-024-10110-1.
    [10] Z. Cheng, H. Sun, M. Takeuchi and J. Katto, Deep convolutional autoencoder-based lossy image compression, In 2018 Picture Coding Symposium (PCS), (2018), 253-257. doi: 10.1109/PCS.2018.8456308.
    [11] A. Cohen and R. DeVore, Approximation of high-dimensional parametric PDEs, Acta Numerica, 24 (2015), 1-159.  doi: 10.1017/S0962492915000033.
    [12] A. Cohen, C. Schwab and J. Zech, Shape holomorphy of the stationary Navier–Stokes equations, SIAM Journal on Mathematical Analysis, 50 (2018), 1720-1752.  doi: 10.1137/16M1099406.
    [13] I. Daubechies, R. DeVore, S. Foucart, B. Hanin and G. Petrova, Nonlinear approximation and (deep) ReLU networks, Constructive Approximation, 55 (2022), 127-172.  doi: 10.1007/s00365-021-09548-z.
    [14] C. De Boor and R. E. Lynch, On splines and their minimum properties, Journal of Mathematics and Mechanics, 15 (1966), 953-969.  doi: 10.1512/iumj.1966.15.15063.
    [15] J. Dugundji, An extension of Tietze's theorem, Pacific J. Math., 1 (1951), 353-367.  doi: 10.2140/pjm.1951.1.353.
    [16] D. Elbrächter, D. Perekrestenko, P. Grohs and H. Bölcskei, Deep neural network approximation theory, IEEE Transactions on Information Theory, 67 (2021), 2581-2623.  doi: 10.1109/TIT.2021.3062161.
    [17] L. C. Evans, Partial Differential Equations, 2$^{nd}$ edition, vol. 19 of Grad. Stud. Math., American Mathematical Society, Providence, RI, 2010. doi: 10.1090/gsm/019.
    [18] C. Fefferman, A. Israel and G. Luli, Sobolev extension by linear operators, Journal of the American Mathematical Society, 27 (2014), 69-145.  doi: 10.1090/S0894-0347-2013-00763-8.
    [19] N. R. Franco, D. Fraulin, A. Manzoni and P. Zunino, On the latent dimension of deep autoencoders for reduced order modeling of PDEs parametrized by random fields, preprint, arXiv: 2310.12095, (2023). doi: 10.48550/arXiv.2310.12095.
    [20] N. R. Franco, S. Fresca, A. Manzoni and P. Zunino, Approximation bounds for convolutional neural networks in operator learning, Neural Networks, 161 (2023), 129-141.  doi: 10.1016/j.neunet.2023.01.029.
    [21] N. R. Franco, A. Manzoni and P. Zunino, A deep learning approach to reduced order modelling of parameter dependent partial differential equations, Mathematics of Computation, 92 (2023), 483-524.  doi: 10.1090/mcom/3781.
    [22] N. R. Franco, A. Manzoni and P. Zunino, Mesh-informed neural networks for operator learning in finite element spaces, Journal of Scientific Computing, 97 (2023), 35.  doi: 10.1007/s10915-023-02331-1.
    [23] J. Frankle and M. Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks, In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
    [24] S. Fresca, L. Dede' and A. Manzoni, A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs, Journal of Scientific Computing, 87 (2021), 1-36.  doi: 10.1007/s10915-021-01462-7.
    [25] S. Fresca and A. Manzoni, POD-DL-ROM: Enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition, Computer Methods in Applied Mechanics and Engineering, 388 (2022), 114181.  doi: 10.1016/j.cma.2021.114181.
    [26] Q. Hernandez, A. Badias, D. Gonzalez, F. Chinesta and E. Cueto, Deep learning of thermodynamics-aware reduced-order models from data, Computer Methods in Applied Mechanics and Engineering, 379 (2021), 113763.  doi: 10.1016/j.cma.2021.113763.
    [27] L. Herrmann, C. Schwab and J. Zech, Neural and GPC operator surrogates: Construction and expression rate bounds, preprint, arXiv: 2207.04950, 2022. doi: 10.48550/arXiv.2207.04950.
    [28] J. S. Hesthaven and S. Ubbiali, Non-intrusive reduced order modeling of nonlinear problems using neural networks, Journal of Computational Physics, 363 (2018), 55-78.  doi: 10.1016/j.jcp.2018.02.037.
    [29] K. Hornik, Approximation capabilities of multilayer feedforward networks, Appl. Math. J. Chinese Univ. Ser. B, 16 (2001), 45-57.  doi: 10.1016/0893-6080(91)90009-T.
    [30] N. Kovachki, S. Lanthaler and S. Mishra, On universal approximation and error bounds for Fourier neural operators, The Journal of Machine Learning Research, 22 (2021), 13237-13312. 
    [31] N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart and A. Anandkumar, Neural operator: Learning maps between function spaces with applications to PDEs, J. Mach. Learn. Res., 24 (2023), 97 pp.
    [32] G. Kutyniok, P. Petersen, M. Raslan and R. Schneider, A theoretical analysis of deep neural networks and parametric PDEs, Constructive Approximation, 55 (2022), 73-125.  doi: 10.1007/s00365-021-09551-4.
    [33] S. Lanthaler, Operator learning with PCA-Net: Upper and lower complexity bounds, Journal of Machine Learning Research, 24 (2023), 1-67. 
    [34] S. Lanthaler, S. Mishra and G. E. Karniadakis, Error estimates for DeepONets: a deep learning framework in infinite dimensions, Transactions of Mathematics and Its Applications, 6 (2022), 141 pp. doi: 10.1093/imatrm/tnac001.
    [35] Y. LeCun and Y. Bengio, Convolutional networks for images, speech, and time series, In The Handbook of Brain Theory and Neural Networks, MIT Press, 3361 (1995). doi: 10.7551/mitpress/3413.001.0001.
    [36] K. Lee and K. T. Carlberg, Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders, Journal of Computational Physics, 404 (2020), 108973.  doi: 10.1016/j.jcp.2019.108973.
    [37] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart and A. Anandkumar, Fourier neural operator for parametric partial differential equations, preprin, arXiv: 2010.08895, 2020. doi: 10.48550/arXiv.2010.08895.
    [38] H. Liu, B. Dahal, R. Lai and W. Liao, Generalization error guaranteed Auto-Encoder-Based nonlinear model reduction for operator learning, preprint, arXiv: 2401.10490, 2024. doi: 10.48550/arXiv.2401.10490.
    [39] L. Lu, P. Jin, G. Pang, Z. Zhang and and G.E. Karniadakis, Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators, Nature Machine Intelligence, 3 (2021), 218-229.  doi: 10.1038/s42256-021-00302-5.
    [40] C. Marcati and C. Schwab, Exponential convergence of deep operator networks for elliptic partial differential equations, SIAM Journal on Numerical Analysis, 61 (2023), 1513-1545.  doi: 10.1137/21M1465718.
    [41] S. Masoumi-Verki, F. Haghighat, N. Bouguila and U. Eicker, The use of GANs and transfer learning in model-order reduction of turbulent wake of an isolated high-rise building, Building and Environment, 246 (2023), 110948.  doi: 10.1016/j.buildenv.2023.110948.
    [42] S. Mishra and T. K. Rusch, Enhancing Accuracy of Deep Learning Algorithms by Training with Low-Discrepancy Sequences, SIAM Journal on Numerical Analysis, 59 (2021), 1811-1834.  doi: 10.1137/20M1344883.
    [43] N. T. Mücke, S. M. Bohtè and C. W. Oosterlee, Reduced order modeling for parameterized time-dependent PDEs using spatially and memory aware deep learning, Journal of Computational Science, 53 (2021), 101408.  doi: 10.1016/j.jocs.2021.101408.
    [44] E. Oostwal, M. Straat and M. Biehl, Hidden unit specialization in layered neural networks: ReLU vs. sigmoidal activation, Physica A: Statistical Mechanics and its Applications, 564 (2021), 125517.  doi: 10.1016/j.physa.2020.125517.
    [45] B. Peherstorfer, Breaking the Kolmogorov barrier with nonlinear model reduction, Notices of the American Mathematical Society, 69 (2022), 725-733.  doi: 10.1090/noti2475.
    [46] P. Petersen and F. Voigtlaender, Equivalence of approximation by convolutional neural networks and fully-connected networks, Proceedings of the American Mathematical Society, 148 (2020), 1567-1581.  doi: 10.1090/proc/14789.
    [47] F. Pichi, F. Ballarin, G. Rozza and J. S. Hesthaven, An artificial neural network approach to bifurcating phenomena in computational fluid dynamics, Computers & Fluids, 254 (2023), 105813.  doi: 10.1016/j.compfluid.2023.105813.
    [48] F. Pichi, B. Moya and J. S. Hesthaven, A graph convolutional autoencoder approach to model order reduction for parametrized PDEs, J. Comput. Phys., 501 (2024), 24 pp. doi: 10.1016/j.jcp.2024.112762.
    [49] F. Romor, G. Stabile and G. Rozza, Non-linear manifold ROM with convolutional autoencoders and reduced over-collocation method, Journal of Scientific Computing, 94(3) (2023), pp. 74. doi: 10.1007/s10915-023-02128-2.
    [50] L. Rosafalco, M. Torzoni, A. Manzoni, S. Mariani and A. Corigliano, Online structural health monitoring by model order reduction and deep learning algorithms, Computers & Structures, 255 (2021), 106604.  doi: 10.1016/j.compstruc.2021.106604.
    [51] C. Schwab and J. Zech, Deep learning in high dimension: Neural network expression rates for analytic functions in $L^2(\mathbb{R}^d, \gamma_d)$, SIAM/ASA Journal on Uncertainty Quantification, 11 (2023), 199-234.  doi: 10.1137/21M1462738.
    [52] C. Sun, M. Ma, Z. Zhao, S. Tian, R. Yan and X. Chen, Deep transfer learning based on sparse autoencoder for remaining useful life prediction of tool in manufacturing, IEEE Transactions on Industrial Informatics, 15 (2018), 2416-2425.  doi: 10.1109/TII.2018.2881543.
    [53] P. Vitullo, A. Colombo, N. R. Franco, A. Manzoni and P. Zunino, Nonlinear model order reduction for problems with microstructure using mesh informed neural networks, Finite Elements in Analysis and Design, 229 (2024), 104068.  doi: 10.1016/j.finel.2023.104068.
    [54] D. Yarotsky, Error bounds for approximations with deep ReLU networks, Neural Networks, 94 (2017), 103-114.  doi: 10.1016/j.neunet.2017.07.002.
  • 加载中

Figures(2)

SHARE

Article Metrics

HTML views(6021) PDF downloads(333) Cited by(0)

Access History

Other Articles By Authors

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return