Spatially regularized Leaky ReLU in dual space for CNN based image segmentation

Xiangyue Wang; Jun Liu

doi:10.3934/ipi.2024016

Article Contents

2024, Volume 18, Issue 6: 1320-1342. Doi: 10.3934/ipi.2024016

This issue Previous Article Bayesian experimental design for linear elasticity Next Article Reconstruction of obstacles in a Stokes flow as a shape-from-moments problem

Spatially regularized Leaky ReLU in dual space for CNN based image segmentation

Xiangyue Wang and
Jun Liu^,

Laboratory of Mathematics and Complex Systems(Ministry of Education of China, School of Mathematical Sciences, Beijing Normal University, Beijing, 100875, China

Corresponding author: Jun Liu
Corresponding author: Jun Liu

Received: October 2023

Revised: March 2024

Early access: April 2024

Published: December 2024

Jun Liu was supported by the National Natural Science Foundation of China (No. 42293272, No.12371527) and the Beijing Natural Science Foundation (No. 1232011).

Abstract / Introduction Full Text(HTML) Figure(10) / Table(7) Related Papers Cited by

Abstract

The ReLU/Leaky-ReLU activation functions are widely used in convolutional neural network (CNN) architectures due to their simple implementation and outstanding performance. However, the current ReLU/Leaky-ReLU activation functions lack spatial priors among pixels during feature extraction. As a result, they may struggle to capture the critical piecewise semantic features that are essential for image segmentation. Drawing inspiration from the regional characteristics of human brain neurons, this paper presents a novel spatially regularized ReLU/Leaky-ReLU module for CNN-based image segmentation. To enhance the extraction of piecewise image features, we introduce a spatial regularization mechanism for ReLU/Leaky-ReLU activation. This is achieved by reinterpreting the activation function as a variational problem with a dual formulation. The proposed variational framework enables the integration of various successful variational priors, such as local and nonlocal spatial regularization, into the activation function design. For efficient regularization in ReLU/Leaky-ReLU, we develop a local and nonlocal threshold dynamics regularizer using difference of concave optimization. This novel activation module facilitates CNNs in extracting crucial piecewise and smooth deep features for image segmentation, while partially mitigating the influence of noise during the CNN test procedure. It provides a means of integrating traditional methods, such as the nonlocal regularization, into ReLU activation in CNNs. The effectiveness of the proposed activation module is demonstrated in the experimental results. Compared to baseline segmentation CNNs, the proposed approach yields a higher segmentation quality index (mIoU) and exhibits better generalization ability to noise.

Keywords:

Mathematics Subject Classification: Primary: 68U10; Secondary: 68T07.

Citation:

Full Text(HTML)

Figure 1. ReLU/LReLU and their derivatives

Download: Full-size image PowerPoint slide

Figure 2. The different scales features extracted from the UNet based on the standard ReLU

Download: Full-size image PowerPoint slide

Figure 3. The different scales features extracted from the UNet based on the proposed TD-ReLU

Download: Full-size image PowerPoint slide

Figure 4. The structure of our proposed (N)TD-LReLU-UNet

Download: Full-size image PowerPoint slide

Figure 5. Leranable Regularity ReLU module

Download: Full-size image PowerPoint slide

Figure 6. Comparison between the standard LReLU and the proposed TD-LReLU in the conv1 layer and NTD-LReLU in the conv8 layer of UNet

Download: Full-size image PowerPoint slide

Figure 7. Comparison of regularized and unregularized ReLU based UNets in handling adversarial noise

Download: Full-size image PowerPoint slide

Figure 8. Comparison of standard LReLU and the proposed TD-LReLU at the last layer of DeeplabV3+

Download: Full-size image PowerPoint slide

Figure 9. Comparison of standard LReLU and the proposed TD-LReLU at the last layer of DeeplabV3+

Download: Full-size image PowerPoint slide

Figure 10. Comparison of standard LReLU and the proposed TD-LReLU at the last layer of DeeplabV3+

Download: Full-size image PowerPoint slide

Table 1. The impact of the regularization parameter $ \lambda $ in the proposed TD-LReLU. The backbone network is UNet

$ \lambda $	UNet($ \lambda $=0)	with TD Regularization
$ \lambda $	UNet($ \lambda $=0)	$ \lambda $=0.1	$ \lambda $=0.3	$ \lambda $=0.4	$ \lambda $=0.5	$ \lambda $=0.7
mIoU(%)	87.01	89.74	89.65	89.52	89.32	88.92
Unsmoothness	957	923	896	883	872	856

| Show Table

DownLoad: CSV

Table 2. Results of TD-LReLU in the eight different convolution layers of UNet

	UNet	with TD Regularization
	UNet	conv1	conv2	conv3	conv4	conv5	conv6	conv7	conv8
mIoU	87.01	$ \underline{\textbf{89.83}} $	88.92	88.99	89.66	89.22	87.29	89.33	89.65
mAP	95.87	96.47	96.45	96.43	96.5	96.37	95.94	96.37	96.43
Accuracy	96.22	97.33	96.92	96.98	97.25	97.1	96.35	97.12	97.26

| Show Table

DownLoad: CSV

Table 3. Results of NTD-LReLU in the eight different convolution layers of UNet

	UNet	with Nonlocal-TD Regularization
	UNet	conv1	conv2	conv3	conv4	conv5	conv6	conv7	conv8
mIoU	87.01	88.7	89.33	89.33	89.88	89.64	$ \underline{\textbf{90.21}} $	89.86	89.95
mAP	95.87	95.94	96.36	96.38	96.53	96.41	96.71	96.5	95.79
Accuracy	96.22	97.02	97.17	97.15	97.33	97.26	96.41	97.33	$ \underline{\textbf{97.55}} $

| Show Table

DownLoad: CSV

Table 4. Comparison of (TD1+NTD8)-LReLU and standard LReLU based UNets on the WBC dataset

	mIoU	mAP	Accuracy
UNet	87.01	95.87	96.22
Ours	$ \underline{\boldsymbol{90.78}} $	95.7	97.84

| Show Table

DownLoad: CSV

Table 5. Comparison inference speed of our TD-LReLU, NTD-LReLU with standard LReLU based UNets on the WBC dataset

Models		Speed(FPS)
UNet		31.14
Ours	with two TD-ReLU Blocks	10.62
Ours	with two NTD-ReLU Blocks	1.22

| Show Table

DownLoad: CSV

Table 6. Performance comparison of regularized and non-regularized ReLU under different levels of Gaussian noise

Noise levels	$ \delta $=10	$ \delta $=20	$ \delta $=30	$ \delta $=40	$ \delta $=50	$ \delta $=100
UNet	86.21	82.07	79.23	76.61	73.37	42.76
Ours(with $ \lambda $=0.1)	87.10	86.41	82.65	81.52	80.32	64.92

| Show Table

DownLoad: CSV

Table 7. Comparison of TD-LReLU and standard LReLU based DeepLabV3+ on PASCAL 2012 dataset

	mIoU	mAP	Accuracy
LReLU	72.45	81.76	92.97
Ours(TD-LReLU)	$ \underline{\boldsymbol{74.34}} $	83.2	93.51

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	A. F. Agarap, Deep learning using rectified linear units (relu), arXiv preprint, (2018), arXiv: 1803.08375.
[2]	Atlanta and Georgia, Rectifier nonlinearities improve neural network acoustic models, 30.
[3]	R. Arora, A. Basu, P. Mianjy and A. Mukherjee, Understanding deep neural networks with rectified linear units, ICLR, 2018 (2018).
[4]	D.-A. Clevert, T. Unterthiner and S. Hochreiter, Fast and accurate deep network learning by exponential linear units (elus), arXiv preprint, (2015), arXiv: 1511.07289.
[5]	F. Croce, M. Andriushchenko and M. Hein, Provable robustness of relu networks via maximization of linear regions, the 22nd International Conference on Artificial Intelligence and Statistics, PMLR, (2019), 2057-2066.
[6]	M. Everingham, L. Van Gool, C. K. Williams, J. Winn and A. Zisserman, The pascal visual object classes (voc) challenge, International Journal of Computer Vision, 88 (2010), 303-338. doi: 10.1007/s11263-009-0275-4.
[7]	S. Esedoḡlu and F. Otto, Threshold dynamics for networks with arbitrary surface tensions, Communications on Pure and Applied Mathematics, 68 (2015), 808-864. doi: 10.1002/cpa.21527.
[8]	X. Glorot, A. Bordes and Y. Bengio, Deep sparse rectifier neural networks. inproceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, (2011), 315-323.
[9]	I. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville and Y. Bengio, Maxout networks, International Conference on Machine Learning, PMLR, (2013), 1319-1327.
[10]	K. He, X. Zhang, S. Ren and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, Proceedings of the IEEE International Conference on Computer Vision, (2015), 1026-1034. doi: 10.1109/ICCV.2015.123.
[11]	A. Heinecke, J. Ho and W.-L. Hwang, Refinement and universal approximation via sparsely connected relu convolution nets, IEEE Signal Processing Letters, 27 (2020), 1175-1179. doi: 10.1109/LSP.2020.3005051.
[12]	D. Hendrycks and K. Gimpel, Gaussian error linear units (gelus), arXiv preprint, (2016), arXiv: 1606.08415.
[13]	H. Ide and T. Kurita, Improvement of learning for cnn with relu activation by sparse regularization, 2017 International Joint Conference on Neural Networks (IJCNN), IEEE, (2017), 2684-2691. doi: 10.1109/IJCNN.2017.7966185.
[14]	P. Krähenbühl and V. Koltun, Efficient inference in fully connected crfs with gaussian edge potentials, Advances in Neural Information Processing Systems, 24 (2011), 109-117.
[15]	A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Communications of the ACM, 60 (2017), 84-90. doi: 10.1145/3065386.
[16]	Y. LeCun, L. Bottou, Y. Bengio and P. Haffner, Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86 (1998), 2278-2324. doi: 10.1109/5.726791.
[17]	X. Liang and J. Xu, Univariate relu neural network and its application in nonlinear system identification, Chinese Intelligent Systems Conference, Springer, (2020), 679-687. doi: 10.1007/978-981-15-8450-3_71.
[18]	J. Liu, X. Wang and X.-C. Tai, Deep convolutional neural networks with spatial regularization, volume and star-shape priors for image segmentation, Journal of Mathematical Imaging and Vision, 64 (2022), 625-645. doi: 10.1007/s10851-022-01087-x.
[19]	J. A. Opschoor, P. C. Petersen and C. Schwab, Deep relu networks and high-order finite element methods, Analysis and Applications, 18 (2020), 715-770. doi: 10.1142/S0219530519410136.
[20]	B. K. Sriperumbudur and G. Lanckriet, On the convergence of the concave-convex procedure, Proceedings of the 22nd International Conference on Neural Information Processing Systems, (2009), 1759-1767.
[21]	B. Xu, N. Wang, T. Chen and M. Li, Empirical evaluation of rectified activations in convolutional network, (2015), arXiv: 1505.00853.
[22]	A. L. Yuille and A. Rangarajan, The concave-convex procedure, Neural Computation, 15 (2003), 915-936. doi: 10.1162/08997660360581958.
[23]	H. Zhang, T.-W. Weng, P.-Y. Chen, C.-J. Hsieh and L. Daniel, Efficient neural network robustness certification with general activation functions, Proceedings of the 32nd International Conference on Neural Information Processing Systems, (2018), 4944-4953.
[24]	D. Zou, Y. Cao, D. Zhou and Q. Gu, Gradient descent optimizes over-parameterized deep relu networks, Machine Learning, 109 (2020), 467-492. doi: 10.1007/s10994-019-05839-6.