# American Institute of Mathematical Sciences

October  2020, 14(5): 891-911. doi: 10.3934/ipi.2020041

## Nonlocal regularized CNN for image segmentation

 1 Department of Mathematics, Hong Kong Baptist University, Hong Kong, China 2 Laboratory of Mathematics and Complex Systems (Ministry of Education of China), School of Mathematical Sciences, Beijing Normal University, Beijing, China

* Corresponding author: Jun Liu

Received  January 2020 Revised  April 2020 Published  July 2020

Non-local dependency is a very important prior for many image segmentation tasks. Generally, convolutional operations are building blocks that process one local neighborhood at a time which means the convolutional neural networks(CNNs) usually do not explicitly make use of the non-local prior on image segmentation tasks. Though the pooling and dilated convolution techniques can enlarge the receptive field to use some nonlocal information during the feature extracting step, there is no nonlocal priori for feature classification step in the current CNNs' architectures. In this paper, we present a non-local total variation (TV) regularized softmax activation function method for semantic image segmentation tasks. The proposed method can be integrated into the architecture of CNNs. To handle the difficulty of back-propagation for CNNs due to the non-smoothness of nonlocal TV, we develop a primal-dual hybrid gradient method to realize the back-propagation of nonlocal TV in CNNs. Experimental evaluations of the non-local TV regularized softmax layer on a series of image segmentation datasets showcase its good performance. Many CNNs can benefit from our proposed method on image segmentation tasks.

Citation: Fan Jia, Xue-Cheng Tai, Jun Liu. Nonlocal regularized CNN for image segmentation. Inverse Problems & Imaging, 2020, 14 (5) : 891-911. doi: 10.3934/ipi.2020041
##### References:

show all references

##### References:
An example of segmentation results by applying the algorithm of [34] and our proposed method on an image from BSD500. When using 4 geometrical nearest neighbors, the weights are set to 1. The segmentation is quite smooth and missing details (Figure 1(b)). When we use Eq. (11) to compute W, the segmentation results are with more details and better accuracy
Given an input $O$, $\lambda = 3$ and $\tau = 0.03$, we perform algorithms for regularized softmax with local operator and non-local operator, respectively. Figure 2(a) is the convergence of softmax with local operate, the primal energy curve has a peak during the iteration. While in Figure 2(b), the energy curve drops rapidly at first and finally converges smoothly
Segmentation results predicted by Unet, RUnet and NLUnet on images from testing dataset of White Blood Cell. From row 2 to row 5, The black regions are background, the gray regions are cell sap, the white regions are nucleus
An enlarged view of segmentation results from Figure 3
Segmentation results predicted by AUnet, RAUnet and NLAUnet on images from testing dataset of White Blood Cell. From row 2 to row 5, The black regions are background, the gray regions are cell sap, the white regions are nucleus
Segmentation results predicted by Segnet, RSegnet and NLSegnet trained on CamVid Dataset
An enlarged view of segmentation results from Figure 6 column 2
An enlarged view of segmentation results from Figure 6 column 1
Results of Unet, RUnet and NLUnet trained on WBC Dataset
 Method Unet [25] RUnet [10] NLUnet mIoU 89.79 90.15 90.80 Accuracy 97.04 97.13 97.42 RE 1.82 1.30 1.59
 Method Unet [25] RUnet [10] NLUnet mIoU 89.79 90.15 90.80 Accuracy 97.04 97.13 97.42 RE 1.82 1.30 1.59
Results of AUnet, RAUnet and NLAUnet trained on WBC Dataset
 Method AUnet [23] RAUnet NLAUnet mIoU 90.75 91.01 91.69 Accuracy 97.35 97.40 97.57 RE 1.43 1.41 1.43
 Method AUnet [23] RAUnet NLAUnet mIoU 90.75 91.01 91.69 Accuracy 97.35 97.40 97.57 RE 1.43 1.41 1.43
Results of Segnet, RSegnet, NLSegnet trained on CamVid Dataset
 Method Segnet [3] RSegnet[10] NLSegnet mIoU 57.35 57.79 59.84 Accuracy 87.74 88.01 88.59 RE 4.10 2.43 3.40
 Method Segnet [3] RSegnet[10] NLSegnet mIoU 57.35 57.79 59.84 Accuracy 87.74 88.01 88.59 RE 4.10 2.43 3.40
 [1] Shi Yan, Jun Liu, Haiyang Huang, Xue-Cheng Tai. A dual EM algorithm for TV regularized Gaussian mixture model in image segmentation. Inverse Problems & Imaging, 2019, 13 (3) : 653-677. doi: 10.3934/ipi.2019030 [2] Wei Wan, Haiyang Huang, Jun Liu. Local block operators and TV regularization based image inpainting. Inverse Problems & Imaging, 2018, 12 (6) : 1389-1410. doi: 10.3934/ipi.2018058 [3] Alina Toma, Bruno Sixou, Françoise Peyrin. Iterative choice of the optimal regularization parameter in TV image restoration. Inverse Problems & Imaging, 2015, 9 (4) : 1171-1191. doi: 10.3934/ipi.2015.9.1171 [4] Yuan Wang, Zhi-Feng Pang, Yuping Duan, Ke Chen. Image retinex based on the nonconvex TV-type regularization. Inverse Problems & Imaging, 2020, 0 (0) : 1-27. doi: 10.3934/ipi.2020050 [5] Jianhong (Jackie) Shen, Sung Ha Kang. Quantum TV and applications in image processing. Inverse Problems & Imaging, 2007, 1 (3) : 557-575. doi: 10.3934/ipi.2007.1.557 [6] Ye Yuan, Yan Ren, Xiaodong Liu, Jing Wang. Approach to image segmentation based on interval neutrosophic set. Numerical Algebra, Control & Optimization, 2020, 10 (1) : 1-11. doi: 10.3934/naco.2019028 [7] Dominique Zosso, Jing An, James Stevick, Nicholas Takaki, Morgan Weiss, Liane S. Slaughter, Huan H. Cao, Paul S. Weiss, Andrea L. Bertozzi. Image segmentation with dynamic artifacts detection and bias correction. Inverse Problems & Imaging, 2017, 11 (3) : 577-600. doi: 10.3934/ipi.2017027 [8] Matthew S. Keegan, Berta Sandberg, Tony F. Chan. A multiphase logic framework for multichannel image segmentation. Inverse Problems & Imaging, 2012, 6 (1) : 95-110. doi: 10.3934/ipi.2012.6.95 [9] Feishe Chen, Lixin Shen, Yuesheng Xu, Xueying Zeng. The Moreau envelope approach for the L1/TV image denoising model. Inverse Problems & Imaging, 2014, 8 (1) : 53-77. doi: 10.3934/ipi.2014.8.53 [10] Jianping Zhang, Ke Chen, Bo Yu, Derek A. Gould. A local information based variational model for selective image segmentation. Inverse Problems & Imaging, 2014, 8 (1) : 293-320. doi: 10.3934/ipi.2014.8.293 [11] Ruiliang Zhang, Xavier Bresson, Tony F. Chan, Xue-Cheng Tai. Four color theorem and convex relaxation for image segmentation with any number of regions. Inverse Problems & Imaging, 2013, 7 (3) : 1099-1113. doi: 10.3934/ipi.2013.7.1099 [12] Lu Tan, Ling Li, Senjian An, Zhenkuan Pan. Nonlinear diffusion based image segmentation using two fast algorithms. Mathematical Foundations of Computing, 2019, 2 (2) : 149-168. doi: 10.3934/mfc.2019011 [13] Balázs Kósa, Karol Mikula, Markjoe Olunna Uba, Antonia Weberling, Neophytos Christodoulou, Magdalena Zernicka-Goetz. 3D image segmentation supported by a point cloud. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020351 [14] Jie Huang, Xiaoping Yang, Yunmei Chen. A fast algorithm for global minimization of maximum likelihood based on ultrasound image segmentation. Inverse Problems & Imaging, 2011, 5 (3) : 645-657. doi: 10.3934/ipi.2011.5.645 [15] Liam Burrows, Weihong Guo, Ke Chen, Francesco Torella. Reproducible kernel Hilbert space based global and local image segmentation. Inverse Problems & Imaging, 2020, 0 (0) : 1-25. doi: 10.3934/ipi.2020048 [16] Tingting Wu, Yufei Yang, Huichao Jing. Two-step methods for image zooming using duality strategies. Numerical Algebra, Control & Optimization, 2014, 4 (3) : 209-225. doi: 10.3934/naco.2014.4.209 [17] Yun Chen, Jiasheng Huang, Si Li, Yao Lu, Yuesheng Xu. A content-adaptive unstructured grid based integral equation method with the TV regularization for SPECT reconstruction. Inverse Problems & Imaging, 2020, 14 (1) : 27-52. doi: 10.3934/ipi.2019062 [18] C.M. Elliott, S. A. Smitheman. Analysis of the TV regularization and $H^{-1}$ fidelity model for decomposing animage into cartoon plus texture. Communications on Pure & Applied Analysis, 2007, 6 (4) : 917-936. doi: 10.3934/cpaa.2007.6.917 [19] Michael Hintermüller, Monserrat Rincon-Camacho. An adaptive finite element method in $L^2$-TV-based image denoising. Inverse Problems & Imaging, 2014, 8 (3) : 685-711. doi: 10.3934/ipi.2014.8.685 [20] Bartomeu Coll, Joan Duran, Catalina Sbert. Half-linear regularization for nonconvex image restoration models. Inverse Problems & Imaging, 2015, 9 (2) : 337-370. doi: 10.3934/ipi.2015.9.337

2019 Impact Factor: 1.373