American Institute of Mathematical Sciences

November  2016, 10(4): 1037-1055. doi: 10.3934/ipi.2016031

A coupled total variation model with curvature driven for image colorization

 1 School of Science, Nanjing University of Posts and Telecommunications, Nanjing, China, China 2 Centre for Mathematical Imaging and Vision and Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong

Received  September 2015 Revised  December 2015 Published  October 2016

In this paper, we study the problem of image colorization based on the propagation from given color pixels to the other grey-level pixels in grayscale images. We propose to use a coupled total variation model with curvature information of luminance channel to control the colorization process. There are two distinct advantages of the proposed model: (i) the involved optimization problem is convex and it is not sensitive to initial guess of colorization procedure; (ii) the proposed model makes use of curvature information to control the color diffusion process which is more effectively than that by using the gradient information. The existence of the minimizer of the proposed model can be shown, and the numerical solver based on convex programming techniques can be developed to solve the resulting model very efficiently. Experimental results are reported to demonstrate that the performance of the proposed model is better than those of the other color propagation models, especially when we deal with large regions of grayscale images for colorization.
Citation: Zhengmeng Jin, Chen Zhou, Michael K. Ng. A coupled total variation model with curvature driven for image colorization. Inverse Problems & Imaging, 2016, 10 (4) : 1037-1055. doi: 10.3934/ipi.2016031
References:
 [1] L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems,, Oxford University Press, (2000).   Google Scholar [2] X. Bresson and T. F. Chan, Fast dual minimization of the vectorical total variation norm and application to color image processing,, Inverse Problems and Imaging, 2 (2008), 455.  doi: 10.3934/ipi.2008.2.455.  Google Scholar [3] A. Bugeau, V.-T. Ta and N. Papadakis, Variational exemplar-based image colorization,, IEEE Trans. Image Process., 23 (2014), 298.  doi: 10.1109/TIP.2013.2288929.  Google Scholar [4] V. Caselles, G. Facciolo and E. Meinhardt, Anisotropic Cheeger sets and applications,, SIAM J. Imaging Sci., 2 (2009), 1211.  doi: 10.1137/08073696X.  Google Scholar [5] R. H. Chan, J. Yang and X. Yuan, Alternating Direction Method for Image Inpainting in Wavelet Domains,, SIAM J. Imaging Sci., 4 (2011), 807.  doi: 10.1137/100807247.  Google Scholar [6] Y. Chen and T. Wunderli, Adaptive total variation for image restoration in BV space,, J. Math. Anal. Appl., 272 (2002), 117.  doi: 10.1016/S0022-247X(02)00141-5.  Google Scholar [7] Z. M. Jin, F. Li and M. K. Ng, A variational approach for image decolorization by variance maximization,, SIAM J. Imaging Sci., 7 (2014), 944.  doi: 10.1137/130935197.  Google Scholar [8] S. H. Kang and R. March, Variational models for image colorization via chromaticity and brightness decomposition,, IEEE Transactions On Image Processing, 16 (2007), 2251.  doi: 10.1109/TIP.2007.903257.  Google Scholar [9] A. Levin, D. Lischinski and Y. Weiss, Colorization using optimization,, in Proc. SIGGRAPH Conf., 23 (2004), 689.  doi: 10.1145/1186562.1015780.  Google Scholar [10] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sci., 8 (2015), 536.  doi: 10.1137/140979368.  Google Scholar [11] M. H. Quang, S. H. Kang and T. M. Le, Image and video colorization using vector-valued reproducing kernel Hilbert spaces,, Journal of Mathematical Imaging and Vision, 37 (2010), 49.  doi: 10.1007/s10851-010-0192-8.  Google Scholar [12] M. L. Song, D. C. Tao and C. Chen, Color to gray: Visual cue preservation,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32 (2012), 1537.   Google Scholar [13] J. Eckstein and D. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators,, Math. Programming, 55 (1992), 293.  doi: 10.1007/BF01581204.  Google Scholar [14] D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite-element approximations,, Comp. Math. Appl., 2 (1976), 17.  doi: 10.1016/0898-1221(76)90003-1.  Google Scholar [15] T. Goldstein and S. Osher, The split Bregman method for $L^1$ regularized problems,, SIAM Journal on Imaging Sciences, 2 (2009), 323.  doi: 10.1137/080725891.  Google Scholar [16] X. D. Hou, J. Harel and C. Koch, Image signature: Highlighting sparse salient regions,, IEEE Trans. Pattern Anal. Mach. Intell., 34 (2012), 194.   Google Scholar [17] R. Hunter, Photoelectric color difference meter,, Journal of the Optical Society of America, 48 (2008), 985.  doi: 10.1364/JOSA.48.000985.  Google Scholar [18] R. Irony, D. Cohen and D. Lischinski, Colorization by example, in Eurographics conference on Rendering Techniques,, Eurographics Association, (2005), 201.   Google Scholar [19] W. K. Pratt, Ed., Digital Image Processing, Wilely-Interscience publication,, New York, (2001).   Google Scholar [20] K. Ito and K. Kunisch, An augmented Lagrangian technique for variational inequalities,, Appl. Math. Optim., 21 (1990), 223.  doi: 10.1007/BF01445164.  Google Scholar [21] L. Itti, C. Koch and E. Niebur, A model of saliency-based visual attention for rapid scene analysis,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20 (1998), 1254.  doi: 10.1109/34.730558.  Google Scholar [22] M. K. Ng, P. Weiss and X. Yuan, Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods,, SIAM J. Sci. Comput., 32 (2010), 2710.  doi: 10.1137/090774823.  Google Scholar [23] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sciences, 8 (2015), 536.  doi: 10.1137/140979368.  Google Scholar [24] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, A unified model for image colorization, computer vision - ECCV 2014 workshops,, Lecture Notes in Computer Science, 8927 (2015), 297.   Google Scholar [25] G. Sapiro, Inpainting the colors,, in Proc. IEEE Int. Conf. Image Processing, 2 (2005), 698.  doi: 10.1109/ICIP.2005.1530151.  Google Scholar [26] G. Sapiro, Geometric Partial Differential Equations and Image Analysis,, Cambridge university press, (2006).   Google Scholar [27] A. A. Shah, M. Gandhi and K. M. Shah, Medical image colorization uisng optimization technique,, International Journal of Scientific and Research Publications, 3 (2013), 1.   Google Scholar [28] T. Welsh, M. Ashikhmin and K. Mueller, Transferring color to grayscale images,, ACM Transactions on Graphics, 21 (2002), 277.   Google Scholar [29] L. Yatziv and G. Sapiro, Fast image and video colorization using chrominance blending,, IEEE Trans. Image Process., 15 (2006), 1120.  doi: 10.1109/TIP.2005.864231.  Google Scholar

show all references

References:
 [1] L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems,, Oxford University Press, (2000).   Google Scholar [2] X. Bresson and T. F. Chan, Fast dual minimization of the vectorical total variation norm and application to color image processing,, Inverse Problems and Imaging, 2 (2008), 455.  doi: 10.3934/ipi.2008.2.455.  Google Scholar [3] A. Bugeau, V.-T. Ta and N. Papadakis, Variational exemplar-based image colorization,, IEEE Trans. Image Process., 23 (2014), 298.  doi: 10.1109/TIP.2013.2288929.  Google Scholar [4] V. Caselles, G. Facciolo and E. Meinhardt, Anisotropic Cheeger sets and applications,, SIAM J. Imaging Sci., 2 (2009), 1211.  doi: 10.1137/08073696X.  Google Scholar [5] R. H. Chan, J. Yang and X. Yuan, Alternating Direction Method for Image Inpainting in Wavelet Domains,, SIAM J. Imaging Sci., 4 (2011), 807.  doi: 10.1137/100807247.  Google Scholar [6] Y. Chen and T. Wunderli, Adaptive total variation for image restoration in BV space,, J. Math. Anal. Appl., 272 (2002), 117.  doi: 10.1016/S0022-247X(02)00141-5.  Google Scholar [7] Z. M. Jin, F. Li and M. K. Ng, A variational approach for image decolorization by variance maximization,, SIAM J. Imaging Sci., 7 (2014), 944.  doi: 10.1137/130935197.  Google Scholar [8] S. H. Kang and R. March, Variational models for image colorization via chromaticity and brightness decomposition,, IEEE Transactions On Image Processing, 16 (2007), 2251.  doi: 10.1109/TIP.2007.903257.  Google Scholar [9] A. Levin, D. Lischinski and Y. Weiss, Colorization using optimization,, in Proc. SIGGRAPH Conf., 23 (2004), 689.  doi: 10.1145/1186562.1015780.  Google Scholar [10] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sci., 8 (2015), 536.  doi: 10.1137/140979368.  Google Scholar [11] M. H. Quang, S. H. Kang and T. M. Le, Image and video colorization using vector-valued reproducing kernel Hilbert spaces,, Journal of Mathematical Imaging and Vision, 37 (2010), 49.  doi: 10.1007/s10851-010-0192-8.  Google Scholar [12] M. L. Song, D. C. Tao and C. Chen, Color to gray: Visual cue preservation,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 32 (2012), 1537.   Google Scholar [13] J. Eckstein and D. Bertsekas, On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators,, Math. Programming, 55 (1992), 293.  doi: 10.1007/BF01581204.  Google Scholar [14] D. Gabay and B. Mercier, A dual algorithm for the solution of nonlinear variational problems via finite-element approximations,, Comp. Math. Appl., 2 (1976), 17.  doi: 10.1016/0898-1221(76)90003-1.  Google Scholar [15] T. Goldstein and S. Osher, The split Bregman method for $L^1$ regularized problems,, SIAM Journal on Imaging Sciences, 2 (2009), 323.  doi: 10.1137/080725891.  Google Scholar [16] X. D. Hou, J. Harel and C. Koch, Image signature: Highlighting sparse salient regions,, IEEE Trans. Pattern Anal. Mach. Intell., 34 (2012), 194.   Google Scholar [17] R. Hunter, Photoelectric color difference meter,, Journal of the Optical Society of America, 48 (2008), 985.  doi: 10.1364/JOSA.48.000985.  Google Scholar [18] R. Irony, D. Cohen and D. Lischinski, Colorization by example, in Eurographics conference on Rendering Techniques,, Eurographics Association, (2005), 201.   Google Scholar [19] W. K. Pratt, Ed., Digital Image Processing, Wilely-Interscience publication,, New York, (2001).   Google Scholar [20] K. Ito and K. Kunisch, An augmented Lagrangian technique for variational inequalities,, Appl. Math. Optim., 21 (1990), 223.  doi: 10.1007/BF01445164.  Google Scholar [21] L. Itti, C. Koch and E. Niebur, A model of saliency-based visual attention for rapid scene analysis,, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20 (1998), 1254.  doi: 10.1109/34.730558.  Google Scholar [22] M. K. Ng, P. Weiss and X. Yuan, Solving constrained total-variation image restoration and reconstruction problems via alternating direction methods,, SIAM J. Sci. Comput., 32 (2010), 2710.  doi: 10.1137/090774823.  Google Scholar [23] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, Luminance-chrominance model for image colorization,, SIAM J. Imaging Sciences, 8 (2015), 536.  doi: 10.1137/140979368.  Google Scholar [24] F. Pierre, J.-F. Aujol, A. Bugeau, N. Papadakis and V.-T. Ta, A unified model for image colorization, computer vision - ECCV 2014 workshops,, Lecture Notes in Computer Science, 8927 (2015), 297.   Google Scholar [25] G. Sapiro, Inpainting the colors,, in Proc. IEEE Int. Conf. Image Processing, 2 (2005), 698.  doi: 10.1109/ICIP.2005.1530151.  Google Scholar [26] G. Sapiro, Geometric Partial Differential Equations and Image Analysis,, Cambridge university press, (2006).   Google Scholar [27] A. A. Shah, M. Gandhi and K. M. Shah, Medical image colorization uisng optimization technique,, International Journal of Scientific and Research Publications, 3 (2013), 1.   Google Scholar [28] T. Welsh, M. Ashikhmin and K. Mueller, Transferring color to grayscale images,, ACM Transactions on Graphics, 21 (2002), 277.   Google Scholar [29] L. Yatziv and G. Sapiro, Fast image and video colorization using chrominance blending,, IEEE Trans. Image Process., 15 (2006), 1120.  doi: 10.1109/TIP.2005.864231.  Google Scholar
 [1] Huu-Quang Nguyen, Ya-Chi Chu, Ruey-Lin Sheu. On the convexity for the range set of two quadratic functions. Journal of Industrial & Management Optimization, 2020  doi: 10.3934/jimo.2020169 [2] Ilyasse Lamrani, Imad El Harraki, Ali Boutoulout, Fatima-Zahrae El Alaoui. Feedback stabilization of bilinear coupled hyperbolic systems. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020434 [3] Yichen Zhang, Meiqiang Feng. A coupled $p$-Laplacian elliptic system: Existence, uniqueness and asymptotic behavior. Electronic Research Archive, 2020, 28 (4) : 1419-1438. doi: 10.3934/era.2020075 [4] Maoding Zhen, Binlin Zhang, Vicenţiu D. Rădulescu. Normalized solutions for nonlinear coupled fractional systems: Low and high perturbations in the attractive case. Discrete & Continuous Dynamical Systems - A, 2020  doi: 10.3934/dcds.2020379 [5] Xavier Carvajal, Liliana Esquivel, Raphael Santos. On local well-posedness and ill-posedness results for a coupled system of mkdv type equations. Discrete & Continuous Dynamical Systems - A, 2020  doi: 10.3934/dcds.2020382 [6] Li-Bin Liu, Ying Liang, Jian Zhang, Xiaobing Bao. A robust adaptive grid method for singularly perturbed Burger-Huxley equations. Electronic Research Archive, 2020, 28 (4) : 1439-1457. doi: 10.3934/era.2020076 [7] Zexuan Liu, Zhiyuan Sun, Jerry Zhijian Yang. A numerical study of superconvergence of the discontinuous Galerkin method by patch reconstruction. Electronic Research Archive, 2020, 28 (4) : 1487-1501. doi: 10.3934/era.2020078 [8] Yuxia Guo, Shaolong Peng. A direct method of moving planes for fully nonlinear nonlocal operators and applications. Discrete & Continuous Dynamical Systems - S, 2020  doi: 10.3934/dcdss.2020462 [9] Noah Stevenson, Ian Tice. A truncated real interpolation method and characterizations of screened Sobolev spaces. Communications on Pure & Applied Analysis, 2020, 19 (12) : 5509-5566. doi: 10.3934/cpaa.2020250 [10] Yue Feng, Yujie Liu, Ruishu Wang, Shangyou Zhang. A conforming discontinuous Galerkin finite element method on rectangular partitions. Electronic Research Archive, , () : -. doi: 10.3934/era.2020120 [11] Leilei Wei, Yinnian He. A fully discrete local discontinuous Galerkin method with the generalized numerical flux to solve the tempered fractional reaction-diffusion equation. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020319 [12] Marion Darbas, Jérémy Heleine, Stephanie Lohrengel. Numerical resolution by the quasi-reversibility method of a data completion problem for Maxwell's equations. Inverse Problems & Imaging, 2020, 14 (6) : 1107-1133. doi: 10.3934/ipi.2020056 [13] Gang Bao, Mingming Zhang, Bin Hu, Peijun Li. An adaptive finite element DtN method for the three-dimensional acoustic scattering problem. Discrete & Continuous Dynamical Systems - B, 2020  doi: 10.3934/dcdsb.2020351 [14] Zuliang Lu, Fei Huang, Xiankui Wu, Lin Li, Shang Liu. Convergence and quasi-optimality of $L^2-$norms based an adaptive finite element method for nonlinear optimal control problems. Electronic Research Archive, 2020, 28 (4) : 1459-1486. doi: 10.3934/era.2020077

2019 Impact Factor: 1.373