
-
Previous Article
Convexification for a 1D hyperbolic coefficient inverse problem with single measurement data
- IPI Home
- This Issue
-
Next Article
A nonconvex truncated regularization and box-constrained model for CT reconstruction
Nonlocal regularized CNN for image segmentation
1. | Department of Mathematics, Hong Kong Baptist University, Hong Kong, China |
2. | Laboratory of Mathematics and Complex Systems (Ministry of Education of China), School of Mathematical Sciences, Beijing Normal University, Beijing, China |
Non-local dependency is a very important prior for many image segmentation tasks. Generally, convolutional operations are building blocks that process one local neighborhood at a time which means the convolutional neural networks(CNNs) usually do not explicitly make use of the non-local prior on image segmentation tasks. Though the pooling and dilated convolution techniques can enlarge the receptive field to use some nonlocal information during the feature extracting step, there is no nonlocal priori for feature classification step in the current CNNs' architectures. In this paper, we present a non-local total variation (TV) regularized softmax activation function method for semantic image segmentation tasks. The proposed method can be integrated into the architecture of CNNs. To handle the difficulty of back-propagation for CNNs due to the non-smoothness of nonlocal TV, we develop a primal-dual hybrid gradient method to realize the back-propagation of nonlocal TV in CNNs. Experimental evaluations of the non-local TV regularized softmax layer on a series of image segmentation datasets showcase its good performance. Many CNNs can benefit from our proposed method on image segmentation tasks.
References:
[1] |
R. Adams and L. Bischof,
Seeded region growing, IEEE Transactions on Pattern Analysis and Machine Intelligence, 16 (1994), 641-647.
doi: 10.1109/34.295913. |
[2] |
M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha and V. K. Asari, Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation, arXiv: 1802.06955. Google Scholar |
[3] |
V. Badrinarayanan, A. Kendall and R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, arXiv: 1511.00561.
doi: 10.1109/TPAMI.2016.2644615. |
[4] |
L. Barghout and L. Lee, Perceptual information processing system, US Patent App. 10/618,543, (2004). Google Scholar |
[5] |
M. Benning, C. Brune, M. Burger and J. Müller,
Higher-order tv methods–enhancement via bregman iteration, Journal of Scientific Computing, 54 (2013), 269-310.
doi: 10.1007/s10915-012-9650-3. |
[6] |
H. Birkholz,
A unifying approach to isotropic and anisotropic total variation denoising models, Journal of Computational and Applied Mathematics, 235 (2011), 2502-2514.
doi: 10.1016/j.cam.2010.11.003. |
[7] |
J. Canny,
A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 8 (1986), 679-698.
doi: 10.1016/B978-0-08-051581-6.50024-6. |
[8] |
G. Gilboa and S. Osher,
Nonlocal operators with applications to image processing, Multiscale Modeling & Simulation, 7 (2008), 1005-1028.
doi: 10.1137/070698592. |
[9] |
K. He, X. Zhang, S. Ren and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proceedings of the IEEE International Conference on Computer Vision, IEEE, 2015, 1026–1034.
doi: 10.1109/ICCV.2015.123. |
[10] |
F. Jia, J. Liu and X. Tai, A regularized convolutional neural network for semantic image segmentation, Analysis and Applications, (2020) 1–19. Google Scholar |
[11] |
M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen and R. Vasudevan, Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?, preprint, arXiv: 1610.01983.
doi: 10.1109/ICRA.2017.7989092. |
[12] |
M. Kass, A. Witkin and D. Terzopoulos, Snakes: Active contour models, International Journal of Computer Vision, 1, (1988) 321–331.
doi: 10.1007/BF00133570. |
[13] |
P. Krähenbühl and V. Koltun, Efficient inference in fully connected crfs with gaussian edge potentials., Advances in Neural Information Processing Systems, (2011), 109–117. Google Scholar |
[14] |
A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems, (2012), 1097–1105.
doi: 10.1145/3065386. |
[15] |
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel,
Backpropagation applied to handwritten zip code recognition, Neural Computation, 1 (1989), 541-551.
doi: 10.1162/neco.1989.1.4.541. |
[16] |
G. Lin, C. Shen, A. V. D. Hengel and I. Reid, Efficient piecewise training of deep structured models for semantic segmentation, in Proceedings of the IEEE Conference on Computer Cision and Pattern Recognition, IEEE, 2016, 3194–3203.
doi: 10.1109/CVPR.2016.348. |
[17] |
J. Long, E. Shelhamer, and T. Darrell, Fully convolutional networks for semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2015, 3431–3440.
doi: 10.1109/CVPR.2015.7298965. |
[18] |
M. Lysaker, A. Lundervold and X.-C. Tai, Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time, IEEE Transactions on Image Processing, 12, (2003), 1579–1590.
doi: 10.1109/TIP.2003.819229. |
[19] |
D. R. Martin, C. C. Fowlkes and and J. Malik,
Learning to detect natural image boundaries using local brightness, color, and texture cues, IEEE Transactions on Pattern Analysis and Machine Intelligence, 26 (2004), 530-549.
doi: 10.1109/TPAMI.2004.1273918. |
[20] |
K. Mikula, A. Sarti and F. Sgallari, Co-volume level set method in subjective surface based medical image segmentation, in Handbook of Biomedical Image Analysis, Springer, (2005), 583–626.
doi: 10.1007/0-306-48551-6_11. |
[21] |
D. Mumford and J. Shah,
Optimal approximations by piecewise smooth functions and associated variational problems, Communications on Pure and Applied Mathematics, 42 (1989), 577-685.
doi: 10.1002/cpa.3160420503. |
[22] |
H. Noh, S. Hong and B. Han, Learning deconvolution network for semantic segmentation, in Proceedings of the IEEE International Conference on Computer Vision, IEEE, 2015, 1520–1528.
doi: 10.1109/ICCV.2015.178. |
[23] |
O. Oktay, et al., Attention u-net: Learning where to look for the pancreas, preprint, arXiv: 1804.03999. Google Scholar |
[24] |
N. Otsu,
A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man and Cybernetics, 9 (1979), 62-66.
doi: 10.1109/TSMC.1979.4310076. |
[25] |
O. Ronneberger, P. Fischer and T. Brox, U-net: Convolutional networks for biomedical image segmentation, in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015,234–241.
doi: 10.1007/978-3-319-24574-4_28. |
[26] |
L. I. Rudin, S. Osher and E. Fatemi,
Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268.
doi: 10.1016/0167-2789(92)90242-F. |
[27] | B. Schölkopf, K. Tsuda and J.-P. Vert, Support Vector Machine Applications in Computational Biology, MIT press, 2004. Google Scholar |
[28] |
L. Shapiro and G. C. Stockman, Computer Vision, Prentice Hall, 2001. Google Scholar |
[29] |
J. Shi and J. Malik, Normalized cuts and image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (2000), 888-908. Google Scholar |
[30] |
K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint, arXiv: 1409.1556. Google Scholar |
[31] |
M. Unger, T. Mauthner, T. Pock and H. Bischof, Tracking as segmentation of spatial-temporal volumes by anisotropic weighted tv, in International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Springer 2009,193–206.
doi: 10.1007/978-3-642-03641-5_15. |
[32] |
P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. Cottrell, Understanding convolution for semantic segmentation, in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2018, 1451–1460.
doi: 10.1109/WACV.2018.00163. |
[33] |
K. Wei, K. Yin, X.-C. Tai and T. F. Chan, New region force for variational models in image segmentation and high dimensional data clustering, preprint, arXiv: 1704.08218.
doi: 10.4310/AMSA.2018.v3.n1.a8. |
[34] |
K. Yin and X.-C. Tai,
An effective region force for some variational models for learning and clustering, Journal of Scientific Computing, 74 (2018), 175-196.
doi: 10.1007/s10915-017-0429-4. |
[35] |
F. Yu and V. Koltun, Multi-scale context aggregation by dilated convolutions, preprint, arXiv: 1511.07122. Google Scholar |
[36] |
L. Zelnik-Manor and P. Perona, Self-tuning spectral clustering, Advances in Neural Information Processing Systems, (2005), 1601–1608. Google Scholar |
[37] |
X. Zheng, Y. Wang, G. Wang and J. Liu,
Fast and robust segmentation of white blood cell images by self-supervised learning, Micron, 107 (2018), 55-71.
doi: 10.1016/j.micron.2018.01.010. |
show all references
References:
[1] |
R. Adams and L. Bischof,
Seeded region growing, IEEE Transactions on Pattern Analysis and Machine Intelligence, 16 (1994), 641-647.
doi: 10.1109/34.295913. |
[2] |
M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha and V. K. Asari, Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation, arXiv: 1802.06955. Google Scholar |
[3] |
V. Badrinarayanan, A. Kendall and R. Cipolla, Segnet: A deep convolutional encoder-decoder architecture for image segmentation, arXiv: 1511.00561.
doi: 10.1109/TPAMI.2016.2644615. |
[4] |
L. Barghout and L. Lee, Perceptual information processing system, US Patent App. 10/618,543, (2004). Google Scholar |
[5] |
M. Benning, C. Brune, M. Burger and J. Müller,
Higher-order tv methods–enhancement via bregman iteration, Journal of Scientific Computing, 54 (2013), 269-310.
doi: 10.1007/s10915-012-9650-3. |
[6] |
H. Birkholz,
A unifying approach to isotropic and anisotropic total variation denoising models, Journal of Computational and Applied Mathematics, 235 (2011), 2502-2514.
doi: 10.1016/j.cam.2010.11.003. |
[7] |
J. Canny,
A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 8 (1986), 679-698.
doi: 10.1016/B978-0-08-051581-6.50024-6. |
[8] |
G. Gilboa and S. Osher,
Nonlocal operators with applications to image processing, Multiscale Modeling & Simulation, 7 (2008), 1005-1028.
doi: 10.1137/070698592. |
[9] |
K. He, X. Zhang, S. Ren and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in Proceedings of the IEEE International Conference on Computer Vision, IEEE, 2015, 1026–1034.
doi: 10.1109/ICCV.2015.123. |
[10] |
F. Jia, J. Liu and X. Tai, A regularized convolutional neural network for semantic image segmentation, Analysis and Applications, (2020) 1–19. Google Scholar |
[11] |
M. Johnson-Roberson, C. Barto, R. Mehta, S. N. Sridhar, K. Rosaen and R. Vasudevan, Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?, preprint, arXiv: 1610.01983.
doi: 10.1109/ICRA.2017.7989092. |
[12] |
M. Kass, A. Witkin and D. Terzopoulos, Snakes: Active contour models, International Journal of Computer Vision, 1, (1988) 321–331.
doi: 10.1007/BF00133570. |
[13] |
P. Krähenbühl and V. Koltun, Efficient inference in fully connected crfs with gaussian edge potentials., Advances in Neural Information Processing Systems, (2011), 109–117. Google Scholar |
[14] |
A. Krizhevsky, I. Sutskever and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in Neural Information Processing Systems, (2012), 1097–1105.
doi: 10.1145/3065386. |
[15] |
Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard and L. D. Jackel,
Backpropagation applied to handwritten zip code recognition, Neural Computation, 1 (1989), 541-551.
doi: 10.1162/neco.1989.1.4.541. |
[16] |
G. Lin, C. Shen, A. V. D. Hengel and I. Reid, Efficient piecewise training of deep structured models for semantic segmentation, in Proceedings of the IEEE Conference on Computer Cision and Pattern Recognition, IEEE, 2016, 3194–3203.
doi: 10.1109/CVPR.2016.348. |
[17] |
J. Long, E. Shelhamer, and T. Darrell, Fully convolutional networks for semantic segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2015, 3431–3440.
doi: 10.1109/CVPR.2015.7298965. |
[18] |
M. Lysaker, A. Lundervold and X.-C. Tai, Noise removal using fourth-order partial differential equation with applications to medical magnetic resonance images in space and time, IEEE Transactions on Image Processing, 12, (2003), 1579–1590.
doi: 10.1109/TIP.2003.819229. |
[19] |
D. R. Martin, C. C. Fowlkes and and J. Malik,
Learning to detect natural image boundaries using local brightness, color, and texture cues, IEEE Transactions on Pattern Analysis and Machine Intelligence, 26 (2004), 530-549.
doi: 10.1109/TPAMI.2004.1273918. |
[20] |
K. Mikula, A. Sarti and F. Sgallari, Co-volume level set method in subjective surface based medical image segmentation, in Handbook of Biomedical Image Analysis, Springer, (2005), 583–626.
doi: 10.1007/0-306-48551-6_11. |
[21] |
D. Mumford and J. Shah,
Optimal approximations by piecewise smooth functions and associated variational problems, Communications on Pure and Applied Mathematics, 42 (1989), 577-685.
doi: 10.1002/cpa.3160420503. |
[22] |
H. Noh, S. Hong and B. Han, Learning deconvolution network for semantic segmentation, in Proceedings of the IEEE International Conference on Computer Vision, IEEE, 2015, 1520–1528.
doi: 10.1109/ICCV.2015.178. |
[23] |
O. Oktay, et al., Attention u-net: Learning where to look for the pancreas, preprint, arXiv: 1804.03999. Google Scholar |
[24] |
N. Otsu,
A threshold selection method from gray-level histograms, IEEE Transactions on Systems, Man and Cybernetics, 9 (1979), 62-66.
doi: 10.1109/TSMC.1979.4310076. |
[25] |
O. Ronneberger, P. Fischer and T. Brox, U-net: Convolutional networks for biomedical image segmentation, in International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015,234–241.
doi: 10.1007/978-3-319-24574-4_28. |
[26] |
L. I. Rudin, S. Osher and E. Fatemi,
Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268.
doi: 10.1016/0167-2789(92)90242-F. |
[27] | B. Schölkopf, K. Tsuda and J.-P. Vert, Support Vector Machine Applications in Computational Biology, MIT press, 2004. Google Scholar |
[28] |
L. Shapiro and G. C. Stockman, Computer Vision, Prentice Hall, 2001. Google Scholar |
[29] |
J. Shi and J. Malik, Normalized cuts and image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (2000), 888-908. Google Scholar |
[30] |
K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint, arXiv: 1409.1556. Google Scholar |
[31] |
M. Unger, T. Mauthner, T. Pock and H. Bischof, Tracking as segmentation of spatial-temporal volumes by anisotropic weighted tv, in International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Springer 2009,193–206.
doi: 10.1007/978-3-642-03641-5_15. |
[32] |
P. Wang, P. Chen, Y. Yuan, D. Liu, Z. Huang, X. Hou, and G. Cottrell, Understanding convolution for semantic segmentation, in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), IEEE, 2018, 1451–1460.
doi: 10.1109/WACV.2018.00163. |
[33] |
K. Wei, K. Yin, X.-C. Tai and T. F. Chan, New region force for variational models in image segmentation and high dimensional data clustering, preprint, arXiv: 1704.08218.
doi: 10.4310/AMSA.2018.v3.n1.a8. |
[34] |
K. Yin and X.-C. Tai,
An effective region force for some variational models for learning and clustering, Journal of Scientific Computing, 74 (2018), 175-196.
doi: 10.1007/s10915-017-0429-4. |
[35] |
F. Yu and V. Koltun, Multi-scale context aggregation by dilated convolutions, preprint, arXiv: 1511.07122. Google Scholar |
[36] |
L. Zelnik-Manor and P. Perona, Self-tuning spectral clustering, Advances in Neural Information Processing Systems, (2005), 1601–1608. Google Scholar |
[37] |
X. Zheng, Y. Wang, G. Wang and J. Liu,
Fast and robust segmentation of white blood cell images by self-supervised learning, Micron, 107 (2018), 55-71.
doi: 10.1016/j.micron.2018.01.010. |





[1] |
Manxue You, Shengjie Li. Perturbation of Image and conjugate duality for vector optimization. Journal of Industrial & Management Optimization, 2020 doi: 10.3934/jimo.2020176 |
[2] |
Liam Burrows, Weihong Guo, Ke Chen, Francesco Torella. Reproducible kernel Hilbert space based global and local image segmentation. Inverse Problems & Imaging, 2021, 15 (1) : 1-25. doi: 10.3934/ipi.2020048 |
[3] |
Balázs Kósa, Karol Mikula, Markjoe Olunna Uba, Antonia Weberling, Neophytos Christodoulou, Magdalena Zernicka-Goetz. 3D image segmentation supported by a point cloud. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 971-985. doi: 10.3934/dcdss.2020351 |
[4] |
Jia Cai, Guanglong Xu, Zhensheng Hu. Sketch-based image retrieval via CAT loss with elastic net regularization. Mathematical Foundations of Computing, 2020, 3 (4) : 219-227. doi: 10.3934/mfc.2020013 |
[5] |
Maika Goto, Kazunori Kuwana, Yasuhide Uegata, Shigetoshi Yazaki. A method how to determine parameters arising in a smoldering evolution equation by image segmentation for experiment's movies. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 881-891. doi: 10.3934/dcdss.2020233 |
[6] |
Ole Løseth Elvetun, Bjørn Fredrik Nielsen. A regularization operator for source identification for elliptic PDEs. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021006 |
[7] |
Mehdi Bastani, Davod Khojasteh Salkuyeh. On the GSOR iteration method for image restoration. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 27-43. doi: 10.3934/naco.2020013 |
[8] |
Yi-Hsuan Lin, Gen Nakamura, Roland Potthast, Haibing Wang. Duality between range and no-response tests and its application for inverse problems. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2020072 |
[9] |
Kha Van Huynh, Barbara Kaltenbacher. Some application examples of minimization based formulations of inverse problems and their regularization. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2020074 |
[10] |
Matúš Tibenský, Angela Handlovičová. Convergence analysis of the discrete duality finite volume scheme for the regularised Heston model. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1181-1195. doi: 10.3934/dcdss.2020226 |
[11] |
Petr Pauš, Shigetoshi Yazaki. Segmentation of color images using mean curvature flow and parametric curves. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1123-1132. doi: 10.3934/dcdss.2020389 |
[12] |
Kateřina Škardová, Tomáš Oberhuber, Jaroslav Tintěra, Radomír Chabiniok. Signed-distance function based non-rigid registration of image series with varying image intensity. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1145-1160. doi: 10.3934/dcdss.2020386 |
[13] |
Qingfang Wang, Hua Yang. Solutions of nonlocal problem with critical exponent. Communications on Pure & Applied Analysis, 2020, 19 (12) : 5591-5608. doi: 10.3934/cpaa.2020253 |
[14] |
Monia Capanna, Jean C. Nakasato, Marcone C. Pereira, Julio D. Rossi. Homogenization for nonlocal problems with smooth kernels. Discrete & Continuous Dynamical Systems - A, 2020 doi: 10.3934/dcds.2020385 |
[15] |
Elimhan N. Mahmudov. Infimal convolution and duality in convex optimal control problems with second order evolution differential inclusions. Evolution Equations & Control Theory, 2021, 10 (1) : 37-59. doi: 10.3934/eect.2020051 |
[16] |
Abdelghafour Atlas, Mostafa Bendahmane, Fahd Karami, Driss Meskine, Omar Oubbih. A nonlinear fractional reaction-diffusion system applied to image denoising and decomposition. Discrete & Continuous Dynamical Systems - B, 2020 doi: 10.3934/dcdsb.2020321 |
[17] |
Mingchao Zhao, You-Wei Wen, Michael Ng, Hongwei Li. A nonlocal low rank model for poisson noise removal. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021003 |
[18] |
P. K. Jha, R. Lipton. Finite element approximation of nonlocal dynamic fracture models. Discrete & Continuous Dynamical Systems - B, 2021, 26 (3) : 1675-1710. doi: 10.3934/dcdsb.2020178 |
[19] |
Biyue Chen, Chunxiang Zhao, Chengkui Zhong. The global attractor for the wave equation with nonlocal strong damping. Discrete & Continuous Dynamical Systems - B, 2021 doi: 10.3934/dcdsb.2021015 |
[20] |
Karol Mikula, Jozef Urbán, Michal Kollár, Martin Ambroz, Ivan Jarolímek, Jozef Šibík, Mária Šibíková. Semi-automatic segmentation of NATURA 2000 habitats in Sentinel-2 satellite images by evolving open curves. Discrete & Continuous Dynamical Systems - S, 2021, 14 (3) : 1033-1046. doi: 10.3934/dcdss.2020231 |
2019 Impact Factor: 1.373
Tools
Metrics
Other articles
by authors
[Back to Top]