Article Contents
Article Contents

# Image segmentation with dynamic artifacts detection and bias correction

1 To whom correspondence should be addressed. Current affiliation and address: Montana State University, Department of Mathematical Sciences, Wilson Hall 2-214, P.O. Box 172400, Bozeman, MT 59717-2400, USA

This work is supported by the Swiss National Science Foundation under grant P300P2-147778, the California Research Training Program in Computational and Applied Mathematics under grant NSF DMS-1045536, the W. M. Keck Foundation, ONR N00014-16-1-2119, and the Merkin Family Foundation.
• Region-based image segmentation is well-addressed by the Chan-Vese (CV) model. However, this approach fails when images are affected by artifacts (outliers) and illumination bias that outweigh the actual image contrast. Here, we introduce a model for segmenting such images. In a single energy functional, we introduce 1) a dynamic artifact class preventing intensity outliers from skewing the segmentation, and 2), in Retinex-fashion, we decompose the image into a piecewise-constant structural part and a smooth bias part. The CV-segmentation terms then only act on the structure, and only in regions not identified as artifacts. The segmentation is parameterized using a phase-field, and efficiently minimized using threshold dynamics.

We demonstrate the proposed model on a series of sample images from diverse modalities exhibiting artifacts and/or bias. Our algorithm typically converges within 10-50 iterations and takes fractions of a second on standard equipment to produce meaningful results. We expect our method to be useful for damaged images, and anticipate use in applications where artifacts and bias are actual features of interest, such as lesion detection and bias field correction in medical imaging, e.g., in magnetic resonance imaging (MRI).

Mathematics Subject Classification: Primary: 68U10, 68T45; Secondary: 49N45, 65K10, 35A15, 35Q93.

 Citation:

• Figure 1.  Example cases 1 & 2. Top: Coronal MRI slice. Input image and initial segmentation contour are shown in the top-right corner. The image is heavily affected by intensity bias, such that the classical CV model fails. The superior parts of the white-matter are undersegmented, while the inferior regions are markedly oversegmented. CV+X is not very helpful, here. In contrast, CV+B fixes the problem: the extracted structure is nearly flat (the brain is essentially two-phase piecewise constant), while CV+XB marks some non-brain pixels as outliers. Bottom: This synthethic image is a combination of piecewise constant regions affected by strong noise and oscillating bias. Again, CV fails, and artifacts detection not appropriate. Bias correction greatly improves the segmentation, but errors persist since from the simple initialization the algorithm converges to a wrong local minimum (upper part of left hand structure)

Figure 2.  Example cases 3 & 4. Top: The seemingly simple scene is not segmentable by the CV model alone, due to strong bias. CV+X wrongly classifies bright regions as artifacts. Bias correction (CV+B) results in accurate segmentation of the T-object. Bottom: The vessel structure is not accurately segmented by classical CV: superior parts are oversegmented due to brightening, inferior parts are undersegmented due to darkening. CV+X is inappropriate, while CV+B fixes the problem and leads to much improved vessel segmentation

Figure 3.  Example cases 5 & 6. Top: Compare to ex. 4 in figure 2.Bottom: Three-phase piecewise constant synthetic image. The goal is to separate the two black ellipses from the gray background, considering the white ring to be an occlusion artifact. CV, however, groups the white ring with the light background. CV+X successfully identifies the ring as artifact, and closes the black ellipses thanks to the interface regularization. The bias correction is misleading, since much of the white ring will be considered overly illuminated background (CV+B, CV+XB), the corners being captured as transient artifacts (CV+XB)

Figure 4.  Results by region-scalable-fitting (RSF) [24]. Left: Starting from generic initialization (as used in our method), RSF fails to capture the correct bias/segmentation result.Right: Starting from a tuned initialization (as provided in [24]), RSF produces the desired bias resistant region-based image segmentation result. Note that for optimally chosen number of iterations, $i$ (as provided in [24]), the computation time is about an order of magnitude slower than our proposed method, as reported in table 1

Figure 5.  Microscopy example cases. Top: This AFM image is severely affected by inhomogeneity compared to pattern contrast. As a result, classical CV segmentation fails. The proposed CV+XB model is able to capture some of the bias as such, and correctly segments some of the actual pattern. The central-square contour initialization, however, provokes an incorrect bias field estimate at early stages of the optimization, and leads to an incorrect local minimum, misclassifying the central portions. Middle: Starting from a near-optimal contour initialization of a single pattern element, the CV model still fails entirely. The proposed CV+XB model is not misled into incorrect minima, anymore, and successfully separates bias from actual pattern contrast. Bottom: The CV+XB model captures most of the inhomogeneity present in this AFM sample image and leads to reasonable segmentation of the diamond pattern. (Note: The inversion of foreground/background between CV and CV+XB models is arbitrary and triggered by domain size

Figure 6.  Further microscopy examples. Top: Seemingly "easy" AFM sample, actually affected by strong inhomogeneity at different scales. CV+XB yields a flattened structural image and correct pattern segmentation. Second set: Fluorescence microscopy image suffering from strong bias and artifacts, failing classical CV. CV+XB corrects bias and captures most artifacts. Third set: AFM sample with crack (white) and bend (darkening). The darkening is beyond recovery, but the crack is correctly identified as artifact. Bottom: Artifact detection collects bright spots and the dark line, resulting in correct stripe-pattern segmentation

Table 1.  Algorithm convergence: number of iterations and required computer time. The algorithm is deterministic. Convergence is defined as the phase-field $u$ not changing during its update. Computation time is statistical depending on CPU scheduling; here, we report average numbers over 50 repetitions. The extra cost of artifact is negligible compared to basic CV, in particular since it may speed up convergence in appropriate images. Bias correction roughly doubles the computational load per iteration, which is an acceptable price for its benefits when appropriate

 I M × N CV CV+X CV+B CV+XB i [1] t [s] i [1] t [s] i [1] t [s] i [1] t [s] 1 78 × 119 11 0.02 20 0.03 11 0.05 11 0.05 2 75 × 79 14 0.02 16 0.02 65 0.24 66 0.25 3 96 × 127 22 0.06 45 0.12 19 0.11 35 0.21 4 110 × 111 15 0.04 30 0.08 71 0.41 100 0.58 5 131 × 103 33 0.11 33 0.12 30 0.26 36 0.32 6 124 × 184 11 0.05 18 0.07 55 0.45 26 0.23
•  [1] A. Ayvaci, M. Raptis and S. Soatto, Sparse occlusion detection with optical flow, International Journal of Computer Vision, 97 (2012), 322-338.  doi: 10.1007/s11263-011-0490-7. [2] X. Bresson, S. Esedoglu, P. Vandergheynst, J.-P. Thiran and S. Osher, Fast global minimization of the active contour/snake model, Journal of Mathematical Imaging and Vision, 28 (2007), 151-167.  doi: 10.1007/s10851-007-0002-0. [3] T. Brox and D. Cremers, On local region models and a statistical interpretation of the piecewise smooth mumford-shah functional, International Journal of Computer Vision, 84 (2008), 184-193.  doi: 10.1007/s11263-008-0153-5. [4] V. Caselles, R. Kimmel and G. Sapiro, Geodesic active contours, Int. J. Comput. Vis., 22 (1995), 61-79.  doi: 10.1109/ICCV.1995.466871. [5] T. F. Chan and L. A. Vese, Active contours without edges, IEEE Transactions on Image Processing, 10 (2001), 266-277.  doi: 10.1109/83.902291. [6] T. Chan and W. Zhu, Level set based shape prior segmentation, , in CVPR 2005, IEEE, 2 (2005), 1164-1170.  doi: 10.1109/CVPR.2005.212. [7] T. F. Chan, S. Esedoglu and M. Nikolova, Algorithms for finding global minimizers of image segmentation and denoising models, SIAM Journal on Applied Mathematics, 66 (2006), 1632-1648.  doi: 10.1137/040615286. [8] T. F. Chan, B. Sandberg and L. A. Vese, Active contours without edges for vector-valued images, Journal of Visual Communication and Image Representation, 11 (2000), 130-141.  doi: 10.1006/jvci.1999.0442. [9] S. A. Claridge, W.-S. Liao, J. C. Thomas, Y. Zhao, H. H. Cao, S. Cheunkar, A. C. Serino, A. M. Andrews and P. S. Weiss, From the bottom up: Dimensional control and characterization in molecular monolayers, Chem. Soc. Rev., 42 (2013), 2725-2745.  doi: 10.1039/C2CS35365B. [10] D. Cremers, S. J. Osher and S. Soatto, Kernel density estimation and intrinsic alignment for shape priors in level set segmentation, International Journal of Computer Vision, 69 (2006), 335-351.  doi: 10.1007/s11263-006-7533-5. [11] D. Cremers, N. Sochen and C. Schnörr, A multiphase dynamic labeling model for variational recognition-driven image segmentation, International Journal of Computer Vision, 66 (2006), 67-81.  doi: 10.1007/s11263-005-3676-z. [12] S. Esedoglu and F. Otto, Threshold dynamics for networks with arbitrary surface tensions, Communications on Pure and Applied Mathematics, 68 (2015), 808-864.  doi: 10.1002/cpa.21527. [13] S. Esedoglu and Y. H. R. Tsai, Threshold dynamics for the piecewise constant Mumford-Shah functional, Journal of Computational Physics, 211 (2006), 367-384.  doi: 10.1016/j.jcp.2005.05.027. [14] V. Estellers and S. Soatto, Detecting occlusions as an inverse problem, Journal of Mathematical Imaging and Vision, 54 (2016), 181-198.  doi: 10.1007/s10851-015-0596-6. [15] V. Estellers, D. Zosso, R. Lai, S. Osher, J.-P. Thiran and X. Bresson, Efficient algorithm for level set method preserving distance function, IEEE Transactions on Image Processing, 21 (2012), 4722-4734.  doi: 10.1109/TIP.2012.2202674. [16] P. Filzmoser, R. G. Garrett and C. Reimann, Multivariate outlier detection in exploration geochemistry, Computers & Geosciences, 31 (2005), 579-587.  doi: 10.1016/j.cageo.2004.11.013. [17] B. K. Horn, Determining lightness from an image, Computer Graphics and Image Processing, 3 (1974), 277-299.  doi: 10.1016/0146-664X(74)90022-7. [18] M. Jung, M. Kang and M. Kang, Variational image segmentation models involving non-smooth data-fidelity terms, Journal of Scientific Computing, 59 (2013), 277-308.  doi: 10.1007/s10915-013-9766-0. [19] M. Kass, A. Witkin and D. Terzopoulos, Snakes: Active contour models, International Journal of Computer Vision, 1 (1988), 321-331.  doi: 10.1007/BF00133570. [20] R. Kimmel, M. Elad, D. Shaked, R. Keshet and I. Sobel, A variational framework for Retinex, International Journal of Computer Vision, 52 (2003), 7-23. [21] E. H. Land, The retinex, American Scientist, 52 (1964), 247-264. [22] E. H. Land, The Retinex theory of color vision, Scientific American, 237 (1977), 108-128. [23] E. H. Land and J. J. McCann, Lightness and Retinex theory, Journal of the Optical Society of America, 61 (1971), 1-11. [24] C. Li, C.-Y. Kao, J. C. Gore and Z. Ding, Minimization of region-scalable fitting energy for image segmentation, IEEE Transactions on Image Processing, 17 (2008), 1940-1949.  doi: 10.1109/TIP.2008.2002304. [25] C. Li, C. Xu, C. Gui and M. D. Fox, Level set evolution without re-initialization: A new variational formulation, in IEEE CVPR, IEEE, 1 (2005), 430-436. [26] C. Li, C. Xu, C. Gui and M. D. Fox, Distance regularized level set evolution and its application to image segmentation, IEEE Transactions on Image Processing, 19 (2010), 3243-3254.  doi: 10.1109/TIP.2010.2069690. [27] C. Li, C. Xu, K. M. Konwar and M. D. Fox, Fast distance preserving level set evolution for medical image segmentation, in 2006 9th International Conference on Control, Automation, Robotics and Vision, IEEE, 2006, 1–7. doi: 10.1109/ICARCV.2006.345357. [28] F. Li, S. Osher, J. Qin and M. Yan, A multiphase image segmentation based on fuzzy membership functions and L1-norm fidelity, J. Sci. Comput., 69 (2016), 82–106, URL http://arxiv.org/abs/1504.02206. doi: 10.1007/s10915-016-0183-z. [29] W. Ma and S. Osher, A TV Bregman iterative model of Retinex theory, Inverse Problems and Imaging (IPI), 6 (2012), 697-708.  doi: 10.3934/ipi.2012.6.697. [30] R. Madani, A. Bourquard and M. Unser, Image segmentation with background correction using a multiplicative smoothing-spline model, in 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI), IEEE, 2012, 186–189. doi: 10.1109/ISBI.2012.6235515. [31] B. Merriman, J. K. Bence and S. Osher, Diffusion Generated Motion by Mean Curvature, Technical report, UCLA CAM Report 92-18, 1992. [32] B. Merriman, J. K. Bence and S. J. Osher, Motion of multiple junctions: A level set approach, Journal of Computational Physics, 112 (1994), 334-363.  doi: 10.1006/jcph.1994.1105. [33] L. Modica, The gradient theory of phase transitions and the minimal interface criterion, Archive for Rational Mechanics and Analysis, 98 (1987), 123-142.  doi: 10.1007/BF00251230. [34] J.-M. Morel, A.-B. Petro and C. Sbert, A PDE formalization of Retinex theory, IEEE Transactions on Image Processing, 19 (2010), 2825-2837.  doi: 10.1109/TIP.2010.2049239. [35] D. Mumford and J. Shah, Optimal approximations by piecewise smooth functions and associated variational problems, Communications on Pure and Applied Mathematics, 42 (1989), 577-685.  doi: 10.1002/cpa.3160420503. [36] J. Nocedal and S. J. Wright, Numerical Optimization, 2nd edition, Springer, Berlin, 2006. [37] S. Osher and J. A. Sethian, Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations, Journal of Computational Physics, 79 (1988), 12-49.  doi: 10.1016/0021-9991(88)90002-2. [38] M. Prastawa, E. Bullitt, S. Ho and G. Gerig, A brain tumor segmentation framework based on outlier detection., Medical Image Analysis, 8 (2004), 275-283.  doi: 10.1016/j.media.2004.06.007. [39] M. Rousson and N. Paragios, Shape Priors for Level Set Representations, in ECCV 2002 (eds. A. Heyden, G. Sparr, M. Nielsen and P. Johansen), vol. 2351 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, Berlin, Heidelberg, 2002, 78–92. doi: 10.1007/3-540-47967-8_6. [40] S. Ruuth, Efficient algorithms for diffusion-generated motion by mean curvature, Journal of Computational Physics, 144 (1998), 603-625.  doi: 10.1006/jcph.1998.6025. [41] Y. van Gennip, N. Guillen, B. Osting and A. L. Bertozzi, Mean curvature, threshold dynamics, and phase field theory on finite graphs, Milan Journal of Mathematics, 82 (2014), 3-65.  doi: 10.1007/s00032-014-0216-8. [42] L. A. Vese and T. F. Chan, A multiphase level set framework for image segmentation using the Mumford and Shah model, International Journal of Computer Vision, 50 (2002), 271-293. [43] L. Wang, C. Li, Q. Sun, D. Xia and C.-Y. Kao, Active contours driven by local and global intensity fitting energy with application to brain MR image segmentation, Computerized medical imaging and graphics, 33 (2009), 520-531.  doi: 10.1016/j.compmedimag.2009.04.010. [44] P. S. Weiss, Functional molecules and assemblies in controlled environments: Formation and measurements, Accounts of Chemical Research, 41 (2008), 1772-1781.  doi: 10.1021/ar8001443. [45] M. Yan, Restoration of images corrupted by impulse noise and mixed Gaussian impulse noise using blind inpainting, SIAM Journal on Imaging Sciences, 6 (2013), 1227-1245.  doi: 10.1137/12087178X. [46] Y. Yang, C. Li, C. -Y. Kao and S. Osher, Split bregman method for minimization of regionscalable fitting energy for image segmentation, in Advances in Visual Computing, vol. 6454 of Lecture Notes in Computer Science, Springer Berlin Heidelberg, 2010, 117–128. [47] D. Zosso, G. Tran and S. Osher, A unifying Retinex model based on non-local differential operators in IS & T/SPIE Electronic Imaging (ed. C. A. Bouman), 8657 (2013), 865702. doi: 10.1117/12.2008839. [48] D. Zosso, G. Tran and S. J. Osher, Non-local Retinex—a unifying framework and beyond, SIAM J. Imaging Sciences, 8 (2015), 787-826.  doi: 10.1137/140972664.

Figures(6)

Tables(1)