A mathematical explanation of UNet

Xue-Cheng Tai; Hao Liu; Raymond H. Chan; Lingfeng Li

doi:10.3934/mfc.2024040

Article Contents

2025, Volume 8, Issue 5: 874-889. Doi: 10.3934/mfc.2024040

This issue Previous Article Dynamical analysis of two discrete-time rumor propagation models in online social networks Next Article

A mathematical explanation of UNet

1.
Norwegian Research Centre (NORCE), Nygårdsgaten 112, 5008 Bergen, Norway
2.
Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong SAR
3.
Lingnan University, Tuen Mun, Hong Kong SAR
4.
Hong Kong Center for Cerebro-Cardiovascular Health Engineering, Hong Kong SAR

^*Corresponding author: Hao Liu
^*Corresponding author: Hao Liu

Received: October 2024

Early access: October 14, 2024

Published online: October 14, 2024

The first author is supported by [HKRGC-NSFC Grant N-CityU214/19, HKRGC CRF Grant C1013-21GF and NORCE Kompetanseoppbygging program]. The second author is supported by [National Natural Science Foundation of China 12201530 and HKRGC ECS 22302123]. The third author is supported by [HKRGC GRF grants CityU1101120, CityU11309922, CRF grant C1013-21GF, and HKRGC-NSFC Grant N-CityU214/19].

Abstract / Introduction Full Text(HTML) Figure(3) / Table(1) Related Papers Cited by

Abstract

The UNet architecture has transformed image segmentation. UNet's versatility and accuracy have driven its widespread adoption, significantly advancing fields reliant on machine learning problems with images. In this work, we gave a clear and concise mathematical explanation of UNet. We explained what is the meaning and function of each of the components of UNet. We showed that UNet was solving a control problem. We decomposed the control variables using multigrid methods. Then, operator-splitting techniques were used to solve the problem, whose architecture exactly recovered the UNet architecture. Our result showed that UNet was a one-step operator-splitting algorithm for the control problem.

Keywords:

Mathematics Subject Classification: Primary: 68U10, 94A08.

Citation:

Full Text(HTML)

Figure 1. An illustration of Algorithm 1.

Download: Full-size image PowerPoint slide

Figure 2. An illustration of a V-cycle of the multigrid method.

Download: Full-size image PowerPoint slide

Figure 3. An illustration of Algorithm 2.

Download: Full-size image PowerPoint slide

Table 1. Explanation of indices for kernels and variables in the left branch of 2.

For $ A^{j}_{k, s}, b^{j}_{k}, S^{j}_k $, $ A^{j, l}_{k, s}, b^{j, l}_{k}, S^{j, l}_k $	$ j $	$ l $	$ k $	$ s $
Index meaning: index of	grid levels	sequential splittings	parallel splittings	output from the previous substep
For $ u^{j}_k, v^{j}_k $, $ u^{j, l}_k, v^{j, l}_k $	$ j $	$ l $	$ k $	-
Index meaning: index of	grid levels	sequential splittings	parallel splittings	-

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	S. Anwar and N. Barnes, Real image denoising with feature attention, Proceedings of the IEEE/CVF International Conference on Computer Vision, (2019), 3155-3164. doi: 10.1109/ICCV.2019.00325.
[2]	V. Badrinarayanan, A. Kendall and R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 (2017), 2481-2495. doi: 10.1109/TPAMI.2016.2644615.
[3]	G. Bao, D. Wang and B. Zou, Wanco: Weak adversarial networks for constrained optimization problems, arXiv preprint, (2024), arXiv: 2407.03647.
[4]	A. R. Barron, Universal approximation bounds for superpositions of a sigmoidal function, IEEE Transactions on Information Theory, 39 (1993), 930-945. doi: 10.1109/18.256500.
[5]	M. Benning, E. Celledoni, M. Ehrhardt, B. Owren and C. Schhönlieb, Deep learning as optimal control problems: Models and numerical methods, Journal of Computational Dynamics, 6 (2019), 171-198. doi: 10.3934/jcd.2019009.
[6]	T. F. Chan and T. P. Mathew, Domain decomposition algorithms, Acta numerica, 3 (1994), 61-143. doi: 10.1017/S0962492900002427.
[7]	T. F. Chan and L. A. Vese, Active contours without edges, IEEE Transactions on image processing, 10 (2001), 266-277. doi: 10.1109/83.902291.
[8]	L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy and A. L. Yuille, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs, IEEE transactions on pattern analysis and machine intelligence, 40 (2017), 834-848. doi: 10.1109/TPAMI.2017.2699184.
[9]	M. Chen, H. Jiang, W. Liao and T. Zhao, Efficient approximation of deep relu networks for functions on low dimensional manifolds, Advances in Neural Information Processing Systems, 32 (2019).
[10]	Z. Cui, T. Y. Pan, G. Yang, J. Zhao and W. Wei, A trainable variational chan-vese network based on algorithm unfolding for image segmentation, Mathematical Foundations of Computing, (2024). doi: 10.3934/mfc.2024030.
[11]	L.-J. Deng, R. Glowinski and X.-C. Tai, A new operator splitting method for the euler elastica model for image smoothing, SIAM Journal on Imaging Sciences, 12 (2019), 1190-1230. doi: 10.1137/18M1226361.
[12]	Y. Duan, Q. Zhong, X.-C. Tai and R. Glowinski, A fast operator-splitting method for beltrami color image denoising, Journal of Scientific Computing, 92 (2022), Paper No. 89, 28 pp. doi: 10.1007/s10915-022-01910-y.
[13]	W. E, A proposal on machine learning via dynamical systems, Communications in Mathematics and Statistics, 5 (2017), 1-11. doi: 10.1007/s40304-017-0103-z.
[14]	W. E, C. Ma and L. Wu, Machine learning from a continuous viewpoint, I, Science China Mathematics, 63 (2020), 2233-2266. doi: 10.1007/s11425-020-1773-8.
[15]	C.-M. Fan, T.-J. Liu and K.-H. Liu, Sunet: Swin transformer unet for image denoising, 2022 IEEE International Symposium on Circuits and Systems (ISCAS), (2022), 2333-2337. doi: 10.1109/ISCAS48785.2022.9937486.
[16]	R. Glowinski, S. Leung and J. Qian, A penalization-regularization-operator splitting method for eikonal based traveltime tomography, SIAM Journal on Imaging Sciences, 8 (2015), 1263-1292. doi: 10.1137/140992072.
[17]	R. Glowinski, H. Liu, S. Leung and J. Qian, A finite element/operator-splitting method for the numerical solution of the two dimensional elliptic monge–ampère equation, Journal of Scientific Computing, 79 (2019), 1-47. doi: 10.1007/s10915-018-0839-y.
[18]	R. Glowinski, S. Luo and X.-C. Tai, Fast operator-splitting algorithms for variational imaging models: Some recent developments, Handbook of Numerical Analysis, 20 (2019), 191-232. doi: 10.1016/bs.hna.2019.08.002.
[19]	R. Glowinski, S. J. Osher and W. Yin, Splitting Methods in Communication, Imaging, Science, and Engineering, Springer, Cham, 2016. doi: 10.1007/978-3-319-41589-5.
[20]	R. Glowinski, T.-W. Pan and X.-C. Tai, Some facts about operator-splitting and alternating direction methods, Splitting Methods in Communication, Imaging, Science, and Engineering, (2016), 19-94. doi: 10.1007/978-3-319-41589-5_2.
[21]	A. Graves, A.-r. Mohamed and G. Hinton, Speech recognition with deep recurrent neural networks, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, (2013), 6645-6649. doi: 10.1109/ICASSP.2013.6638947.
[22]	J. Gurrola-Ramos, O. Dalmau and T. E. Alarcón, A residual dense u-net neural network for image denoising, IEEE Access, 9 (2021), 31742-31754. doi: 10.1109/ACCESS.2021.3061062.
[23]	E. Haber and L. Ruthotto, Stable architectures for deep neural networks, Inverse Problems, 34 (2018), 1-23. doi: 10.1088/1361-6420/aa9a90.
[24]	E. Haber, L. Ruthotto, E. Holtham and S. H. Jun, Learning across scales - Multiscale methods for convolution neural networks, 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, (2018), 3142-3148. doi: 10.1609/aaai.v32i1.11680.
[25]	J. He and J. Xu, MgNet: A unified framework of multigrid and convolutional neural network, Science China Mathematics, 62 (2019), 1331-1354. doi: 10.1007/s11425-019-9547-2.
[26]	Y. Lan, Z. Li, J. Sun and Y. Xiang, Dosnet as a non-black-box pde solver: When deep learning meets operator splitting, J. Comput. Phys., 491 (2023), Paper No. 112343, 22 pp, arXiv: 2212.05571. doi: 10.1016/j.jcp.2023.112343.
[27]	H. Liu, J. Liu, R. Chan and X.-C. Tai, Double-well net for image segmentation, arXiv preprint, (2023), arXiv: 2401.00456.
[28]	H. Liu, X.-C. Tai and R. Chan, Connections between operator-splitting methods and deep neural networks with applications in image segmentation, Ann. Appl. Math., 39 (2023), 406-428. doi: 10.4208/aam.OA-2023-0027.
[29]	H. Liu, X.-C. Tai and R. Glowinski, An operator-splitting method for the gaussian curvature regularization model with applications to surface smoothing and imaging, SIAM Journal on Scientific Computing, 44 (2022), A935-A963. doi: 10.1137/21M143772X.
[30]	H. Liu, X.-C. Tai, R. Kimmel and R. Glowinski, A color elastica model for vector-valued image regularization, SIAM Journal on Imaging Sciences, 14 (2021), 717-748. doi: 10.1137/20M1354532.
[31]	J. Liu, X. Wang and X.-C. Tai, Deep convolutional neural networks with spatial regularization, volume and star-shape priors for image segmentation, Journal of Mathematical Imaging and Vision, 64 (2022), 625-645. doi: 10.1007/s10851-022-01087-x.
[32]	J. Long, E. Shelhamer and T. Darrell, Fully convolutional networks for semantic segmentation, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2015), 3431-3440. doi: 10.1109/CVPR.2015.7298965.
[33]	O. Ronneberger, P. Fischer and T. Brox, U-net: Convolutional networks for biomedical image segmentation, International Conference on Medical Image Computing and Computer-Assisted Intervention, (2015), 234-241. doi: 10.1007/978-3-319-24574-4_28.
[34]	L. Ruthotto and E. Haber, Deep neural networks motivated by partial differential equations, Journal of Mathematical Imaging and Vision, 62 (2020), 352-364. doi: 10.1007/s10851-019-00903-1.
[35]	X.-C. Tai, Rate of convergence for some constraint decomposition methods for nonlinear variational inequalities, Numerische Mathematik, 93 (2003), 755-786. doi: 10.1007/s002110200404.
[36]	X.-C. Tai, H. Liu and R. Chan, Pottsmgnet: A mathematical explanation of encoder-decoder based neural networks, SIAM Journal on Imaging Sciences, 17 (2024), 540-594. doi: 10.1137/23M1586355.
[37]	X.-C. Tai and J. Xu, Subspace correction methods for convex optimization problems, Eleventh International Conference on Domain Decomposition Methods (London, 1998), (1999), 130-139.
[38]	X.-C. Tai and J. Xu, Global and uniform convergence of subspace correction methods for some convex optimization problems, Mathematics of Computation, 71 (2002), 105-124. doi: 10.1090/S0025-5718-01-01311-4.
[39]	L. Tan, L. Li, W.-Q. Liu, S.-J. An and K. Munyard, Unsupervised learning of multi-task deep variational model, Journal of Visual Communication and Image Representation, 87 (2022), 103588. doi: 10.1016/j.jvcir.2022.103588.
[40]	T. Wu, C. Huang, S. Jia, W. Li, R. Chan, T. Zeng and S. K. Zhou, Medical image reconstruction with multi-level deep learning denoiser and tight frame regularization, Applied Mathematics and Computation, 477 (2024), 128795, 19 pp. doi: 10.1016/j.amc.2024.128795.
[41]	J. Xu, Iterative methods by space decomposition and subspace correction, SIAM review, 34 (1992), 581-613. doi: 10.1137/1034116.
[42]	M. Xu, Q. Ma, H. Zhang, D. Kong and T. Zeng, MEF-UNet: An end-to-end ultrasound image segmentation algorithm based on multi-scale feature extraction and fusion, Computerized Medical Imaging and Graphics, 114 (2024), 102370. doi: 10.1016/j.compmedimag.2024.102370.
[43]	D. Yarotsky, Error bounds for approximations with deep ReLU networks, Neural Networks, 94 (2017), 103-114. doi: 10.1016/j.neunet.2017.07.002.
[44]	Y. Zang, G. Bao, X. Ye and H. Zhou, Weak adversarial networks for high-dimensional partial differential equations, Journal of Computational Physics, 411 (2020), 109409, 14 pp. doi: 10.1016/j.jcp.2020.109409.
[45]	K. Zhang, L. Li, H. Liu, J. Yuan and X.-C. Tai, Deep convolutional neural networks meet variational shape compactness priors for image segmentation, arXiv preprint, (2024), arXiv: 2406.19400.
[46]	K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Conference on Computer Vision and Pattern Recognition, 26 (2017), 3142-3155. doi: 10.1109/TIP.2017.2662206.
[47]	D.-X. Zhou, Universality of deep convolutional neural networks, Applied and Computational Harmonic Analysis, 48 (2020), 787-794. doi: 10.1016/j.acha.2019.06.004.
[48]	Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh and J. Liang, Unet++: Redesigning skip connections to exploit multiscale features in image segmentation, IEEE Transactions on Medical Imaging, 39 (2019), 1856-1867. doi: 10.1109/TMI.2019.2959609.