Reducing spatially varying out-of-focus blur from natural image

Faming Fang; Fang Li; Tieyong Zeng

doi:10.3934/ipi.2017004

Article Contents

2017, Volume 11, Issue 1: 65-85. Doi: 10.3934/ipi.2017004

This issue Previous Article On the set of metrics without local limiting Carleman weights Next Article On the measurement operator for scattering in layered media

Reducing spatially varying out-of-focus blur from natural image

1.
Shanghai Key Laboratory of Multidimensional Information Processing, Department of Computer Science, East China Normal University, Shanghai, China
2.
Department of Mathematics, East China Normal University, Shanghai, China
3.
Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Hong Kong, China

* Corresponding author: Tieyong Zeng
* Corresponding author: Tieyong Zeng

Received: December 2015

Revised: September 2016

Published: March 2018

Abstract / Introduction Full Text(HTML) Figure(11) / Table(2) Related Papers Cited by

Abstract

In this paper, we focus on the challenging problem of removing the spatially varying out-of-focus blur from a single natural image. We first propose an effective method to estimate the blur map by the total variation refinement on Hölder coefficient, then discuss the properties of the corresponding kernel matrix. A tight-frame based energy functional, whose minimizer is related to the optimal defocus result, is thus built. For tackling functional more efficiently, we describe the numerical procedure based on an accelerated primal-dual scheme. To verify the effectiveness of our method, we compare it with some state-of-the-art schemes using both synthesized and natural images. Experimental results demonstrate that the proposed method performs better than the compared methods.

Keywords:

Mathematics Subject Classification: Primary: 68U10, 65K10, 35A15, 91-08; Secondary: 62H35.

Citation:

Full Text(HTML)

Figure 1. The spatially-varying blurring images.

Download: Full-size image PowerPoint slide

Figure 2. The distributions of the NHC for in-focus (red) and blur images (blue).

Download: Full-size image PowerPoint slide

Figure 3. The blur maps of Figure 1(a) ($800\times 600$ pixels): (a) the rough blur map $\boldsymbol{\tilde{\sigma}}$; (b) the refined blur map $\boldsymbol{{\sigma}}$.

Download: Full-size image PowerPoint slide

Figure 4. (a) a sharp image (size:100$\times$ 400); (b) a blur version of (a) (the blur levels are 1 and 4 for 171th-230th and 271th-330th columns, respectively); (c)-(e) the blur maps genrated by ZS [52], SHP [39] and proposed methods; (f) the ground truth of blur map; (g) corresponding profiles of 1-st line for (c)-(f).

Download: Full-size image PowerPoint slide

Figure 5. (a) and (b) are original images; (c) and (d) are two blur maps (range: 1-5); (e): the blurred version of (a) using (c) as blur map; (f): the blurred version of (a) using (d) as blur map; (g): the blurred version of (b) using (c) as blur map; (h): the blurred version of (b) using (d) as blur map.

Download: Full-size image PowerPoint slide

Figure 6. Synthesised experiments: (A1-A5)/(B1-B5) are the deblurred results (with zoomed regions) of Figure 5(e)/(f) by Matlab, XJ [49], CJLS [8], SHP [39], and proposed methods, respectively.

Download: Full-size image PowerPoint slide

Figure 7. Synthesised experiments: (A1-A5)/(B1-B5) are the deblurred results (with zoomed regions) of Figure 5(g)/(h) by Matlab, XJ [49], CJLS [8], SHP [39], and proposed methods, respectively.

Download: Full-size image PowerPoint slide

Figure 8. PSNR, SSIM, and SI values corresponding to each figure.

Download: Full-size image PowerPoint slide

Figure 9. Original image and its deblurred results with zoomed regions. (a) The original image (RGB, $296\times 877$ pixels); (b)-(f) the results of Matlab, XJ [49], CJLS [8], SHP [39], and the proposed methods.

Download: Full-size image PowerPoint slide

Figure 10. Original image and its deblurred results with zoomed regions. (a) The original image (RGB, $800\times 600$ pixels); (b)-(f) the results of Matlab, XJ [49], CJLS [8], SHP [39], and the proposed methods.

Download: Full-size image PowerPoint slide

Figure 11. Original image and its deblurred results with zoomed regions. (a) The original image (RGB, $550\times 760$ pixels); (b)-(f) the results of Matlab, XJ [49], CJLS [8], SHP [39], and the proposed methods.

Download: Full-size image PowerPoint slide

Table 1. The MAE value of each blur map generated by ZS [52], SHP [39] and Proposed methods.

	ZS [52]	SHP [39]	Proposed
Figure 4(b)	1.3799	0.6495	0.5308
Figure 5(e)	0.4043	0.3753	0.3751
Figure 5(f)	0.3959	0.3950	0.3684
Figure 5(g)	0.4811	0.4455	0.3064
Figure 5(h)	0.4823	0.4756	0.3418

| Show Table

DownLoad: CSV

Table 2. Time comparison with other methods (second).

	Figure 5(e)	Figure 5(f)	Figure 5(g)	Figure 5(h)	Figure 9(a)	Figure 10(a)	Figure 11(a)
image size	300 Ã— 286	300 Ã— 286	265 Ã— 300	265 Ã— 300	200 Ã— 300	800 Ã— 600	534 Ã— 800
Matlab	1.18	1.22	1.00	1.17	2.03	6.70	6.00
XJ [49](C)	6.45	6.55	6.36	6.35	11.43	18.01	16.95
CJLS [8]	86.59	83.74	88.27	89.84	194.95	780.30	693.77
SHP [39]	70.21	69.40	72.26	71.55	126.37	187.01	168.68
Proposed	38.33	37.60	34.98	36.46	93.45	137.93	141.32

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

	S. Bae and F. Durand , Defocus magnification, in Computer Graphics Forum, 26 (2007) , 571-579. doi: 10.1111/j.1467-8659.2007.01080.x.
	L. Bar , N. Sochen and N. Kiryati , Restoration of images with piecewise space-variant blur, in In Proceedings of the First International Conference on Scale Space Methods and Variational Methods in Computer Vision, 4485 (2007) , 533-544. doi: 10.1007/978-3-540-72823-8_46.
	G. Blanchet and L. Moisan , An explicit sharpness index related to global phase coherence, in 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2012) , 1065-1068. doi: 10.1109/ICASSP.2012.6288070.
	C. Boor, A Practical Guide to Splines vol. 27, 1st edition, New York: Springer, 1987.
	S. Boyd , N. Parikh , E. Chu , B. Peleato and J. Eckstein , Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends in Machine Learning, 3 (2011) , 1-122. doi: 10.1561/2200000016.
	C. Byrne , Bounds on the largest singular value of a matrix and the convergence of simultaneous and block-iterative algorithms for sparse linear systems, International Transactions in Operational Research, 16 (2009) , 465-479. doi: 10.1111/j.1475-3995.2009.00692.x.
	J. Cai , R. Chan and Z. Shen , A framelet-based image inpainting algorithm, Applied Computational Harmonic Analysis, 24 (2008) , 131-149. doi: 10.1016/j.acha.2007.10.002.
	J. Cai , H. Ji , C. Liu and Z. Shen , Framelet-based blind motion deblurring from a single image, IEEE Transactions on Image Processing, 21 (2012) , 562-572. doi: 10.1109/TIP.2011.2164413.
	J. Cai , S. Osher and Z. Shen , Linearized bregman iterations for frame-based image deblurring, SIAM Journal on Imaging Sciences, 2 (2009) , 226-252. doi: 10.1137/080733371.
	J. Cai , S. Osher and Z. Shen , Split bregman method and frame based image restoration, Notices of the American Mathematical Society, 8 (2009) , 337-369.
	A. Chai and Z. Shen , Deconvolution: A wavelet frame approach, Numerische Mathematik, 106 (2007) , 529-587. doi: 10.1007/s00211-007-0075-0.
	A. Chakrabarti , T. Zickler and W. Freeman , Analyzing spatially-varying blur, in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, (2010) , 2512-2519. doi: 10.1109/CVPR.2010.5539954.
	A. Chambolle and A. Pock , A first-order primal-dual algorithm for convex problems with applications to imaging, Journal of Mathematical Imaging and Vision, 40 (2011) , 120-145. doi: 10.1007/s10851-010-0251-1.
	R. Chan , S. Riemenschneider , L. Shen and Z. Shen , Tight frame: an efficient way for high-resolution image reconstruction, Applied Computational Harmonic Analysis, 17 (2004) , 91-115. doi: 10.1016/j.acha.2004.02.003.
	S. Chan and T. Nguyen , Single image spatially variant out-of-focus blur removal, in Image Processing (ICIP), 2011 18th IEEE International Conference on, (2011) , 677-680.
	T. Chan, Theory and Computation of Variational Image Deblurring Lecture notes series, WSPC, 2005.
	Y. Chen , G. Lan and Y. Ouyang , Optimal primal-dual methods for a class of saddle point problems, SIAM Journal on Optimization, 24 (2014) , 1779-1814. doi: 10.1137/130919362.
	I. Daubechies , B. Han , A. Ron and Z. Shen , Framelets: Mra-based constructions of wavelet frames, Applied and Computational Harmonic Analysis, 14 (2003) , 1-46. doi: 10.1016/S1063-5203(02)00511-0.
	I. Ekeland and R. Témam, Convex Analysis and Variational Problems Classics in Applied Mathematics (Book 28), Society for Industrial and Applied Mathematics, Philadelphia, 1999.
	L. Evans, Partial Differential Equations, vol. 19, chapter Holder-spaces, 456–461, American Mathematical Society, 1998.
	F. Fang , G. Zhang , F. Li and C. Shen , Framelet based pan-sharpening via a variational method, Neurocomputing, 129 (2014) , 362-377. doi: 10.1016/j.neucom.2013.09.022.
	E. Faramarzi , D. Rajan and M. Christensen , Unified blind method for multi-image super-resolution and single/multi-image blur deconvolution, IEEE Transactions on Image Processing, 22 (2013) , 2101-2114. doi: 10.1109/TIP.2013.2237915.
	T. Goldstein and S. Osher , The split bregman method for L1-regularized problems, SIAM Journal on Imaging Sciences, 2 (2009) , 323-343. doi: 10.1137/080725891.
	R. Gonzalez and R. Woods, Digital Image Processing 3rd edition, Prentice-Hall, Inc. , Upper Saddle River, NJ, USA, 2006.
	A. Gupta , N. Joshi , L. Zitnick , M. Cohen and B. Curless , Single image deblurring using motion density functions, in ECCV '10: Proceedings of the 10th European Conference on Computer Vision, 6311 (2010) , 171-184. doi: 10.1007/978-3-642-15549-9_13.
	R. Horn and C. Johnson, Matrix Analysis Cambridge University Press, Cambridge, 1985.
	Y. Huang , D. Lu and T. Zeng , A two-step approach for the restoration of images corrupted by multiplicative, SIAM Journal on Scientific Computing, 35 (2013) , 2856-2873. doi: 10.1137/120898693.
	S. Jaffard , Wavelet techniques in multifractal analysis, in In Proceedings of symposia in pure mathematics, 72 (2004) , 91-152.
	S. Kim, I. Eom and Y. Kim, Image Interpolation Based on Statistical Relationship Between Wavelet Subbands in Multimedia and Expo, 2007 IEEE International Conference on, 2007. doi: 10.1109/ICME.2007.4285002.
	S. Kindermann , S. Osher and P. Jones , Deblurring and denoising of images by nonlocal functionals, SIAM Multiscale Modeling and Simulation, 4 (2005) , 1091-1115. doi: 10.1137/050622249.
	P. Legrand and J. Vehel , Local regularity-based image denoising, in Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, 3 (2003) , 377-380.
	Y. Lou , E. Esser , H. Zhao and J. Xin , Partially blind deblurring of barcode from out-of-focus blur, SIAM Journal on Imaging Sciences, 7 (2014) , 740-760. doi: 10.1137/130931254.
	Y. Lou , X. Zhang , S. Osher and A. Bertozzi , Image recovery via nonlocal operators, Journal of Scientific Computing, 42 (2010) , 185-197. doi: 10.1007/s10915-009-9320-2.
	J. Muzy , E. Bacry and A. Arneodo , The multifractal formalism revisited with wavelets, International Journal of Bifurcation and Chaos, 4 (1994) , 245-302. doi: 10.1142/S0218127494000204.
	J. Nagy and D. O'Leary , Restoring images degraded by spatially variant blur, SIAM Journal on Scientific Computing, 19 (1998) , 1063-1082. doi: 10.1137/S106482759528507X.
	J. Oliveira , M. Figueiredo and J. Bioucas-Dias , Parametric blur estimation for blind restoration of natural images: Linear motion and out-of-focus, IEEE Transactions on Image Processing, 23 (2014) , 466-477. doi: 10.1109/TIP.2013.2286328.
	A. Ron and Z. Shen , Affine system in $l_2(\mathbb{R}^d)$: The analysis of the analysis operator, Journal of Functional Analysis, 148 (1997) , 408-447. doi: 10.1006/jfan.1996.3079.
	L. Rudin , S. Osher and E. Fatemi , Nonlinear total variation based noise removal algorithms, Physica D, 60 (1992) , 259-268. doi: 10.1016/0167-2789(92)90242-F.
	C. Shen , W. Hwang and S. Pei , Spatially-varying out-of-focus image deblurring with l1-2 optimization and a guided blur map, in Acoustics, Speech and Signal Processing (ICASSP), 2012 IEEE International Conference on, (2012) , 1069-1072. doi: 10.1109/ICASSP.2012.6288071.
	Y. Tai and M. Brown , Single image defocus map estimation using local contrast prior, in Proceedings of the 16th IEEE International Conference on Image Processing, ICIP'09, (2009) , 1777-1780.
	H. Trussell and S. Fogel , Identification and restoration of spatially variant motion blurs in sequential images, IEEE Transactions on Image Processing, 1 (1992) , 123-126. doi: 10.1109/83.128039.
	M. Unser and T. Blu , Mathematical properties of the {JPEG2000} wavelet filters, IEEE Transactions on Image Processing, 12 (2003) , 1080-1090. doi: 10.1109/TIP.2003.812329.
	J. Véhel, Fractals in Multimedia, vol. 132 of The IMA Volumes in Mathematics and its Application, chapter Signal Enhancement Based on Hölder Regularity Analysis, 197–209, Springer, New York, 2002.
	Y. Wang, W. Yin and Y. Zhang, A Fast Algorithm for Image Deblurring with Total Variation Regularization CAAM technical reports, 2007.
	Z. Wang , A. Bovik , H. Sheikh and E. Simoncelli , Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, 13 (2004) , 600-612. doi: 10.1109/TIP.2003.819861.
	O. Whyte , J. Sivic , A. Zisserman and J. Ponce , Non-uniform deblurring for shaken images, Int. J. Comput. Vis., 98 (2012) , 168-186. doi: 10.1007/s11263-011-0502-7.
	C. Wu and X. Tai , Augmented lagrangian method, dual methods and split bregman iteration for {ROF} model, in Scale Space and Variational Methods in Computer Vision, (2009) , 502-513.
	J. Xu , H. Chang and J. Qin , Domain decomposition method for image deblurring, Journal of Computational and Applied Mathematics, 271 (2014) , 401-414. doi: 10.1016/j.cam.2014.03.030.
	L. Xu and J. Jia , Two-phase kernel estimation for robust motion deblurring, Computer Vision C ECCV 2010: The series Lecture Notes in Computer Science, 6311 (2010) , 157-170. doi: 10.1007/978-3-642-15549-9_12.
	L. Xu and J. Jia, Two-phase kernel estimation for robust motion deblurring http://www.cse.cuhk.edu.hk/leojia/projects/robust_deblur/, 2010. doi: 10.1007/978-3-642-15549-9_12.
	X. Zhu , S. Cohen , S. Schiller and P. Milanfar , Estimating spatially varying defocus blur from a single image, IEEE Transactions on Image Processing, 22 (2013) , 4879-4891. doi: 10.1109/TIP.2013.2279316.
	S. Zhuo and T. Sim , Defocus map estimation from a single image, Pattern Recognition, 44 (2011) , 1852-1858. doi: 10.1016/j.patcog.2011.03.009.