Large region inpainting by re-weighted regularized methods

Yiting Chen; Jia Li; Qingyun Yu

doi:10.3934/ipi.2021015

Article Contents

2021, Volume 15, Issue 5: 827-842. Doi: 10.3934/ipi.2021015

This issue Previous Article Next Article Inbetweening auto-animation via Fokker-Planck dynamics and thresholding

Large region inpainting by re-weighted regularized methods

School of Mathematics, Sun Yat-Sen University, 135 Xin Gang Xi Lu, Guangzhou, 510275, China

^* Corresponding author: Jia Li
^* Corresponding author: Jia Li

Received: April 2020

Revised: November 2020

Early access: February 2021

Published: October 2021

The corresponding author's work is partially supported by NSFC young researchers' grant 11801594 and Guangdong-Hong Kong-Macau Applied Math Center grant 2020B1515310011

Abstract Full Text(HTML) Figure(8) Related Papers Cited by

Abstract

In the development of imaging science and image processing request in our daily life, inpainting large regions always plays an important role. However, the existing local regularized models and some patch manifold based non-local models are often not effective in restoring the features and patterns in the large missing regions. In this paper, we will apply a strategy of inpainting from outside to inside and propose a re-weighted matching algorithm by closest patch (RWCP), contributing to further enhancing the features in the missing large regions. Additionally, we propose another re-weighted matching algorithm by distance-based weighted average (RWWA), leading to a result with higher PSNR value in some cases. Numerical simulations will demonstrate that for large region inpainting, the proposed method is more applicable than most canonical methods. Moreover, combined with image denoising methods, the proposed model is also applicable for noisy image restoration with large missing regions.

Keywords:

Mathematics Subject Classification: Primary: 65D18, 68U10; Secondary: 65J22.

Citation:

Full Text(HTML)

Figure 1. Large region ipainting by some canonical and recent methods

Download: Full-size image PowerPoint slide

Figure 2. An example of similar patches in fruits image

Download: Full-size image PowerPoint slide

Figure 5. Large region inpainting for a fruits image

Download: Full-size image PowerPoint slide

Figure 3. Convergence curve for the object functions in Algorithm 2. The left image is the convergence curve for RWMC model and the right image is that for RWWA model

Download: Full-size image PowerPoint slide

Figure 4. The result of large region inpainting from noisy images with Gaussian noise $ \sigma = 10 $

Download: Full-size image PowerPoint slide

Figure 6. Numerical results for boat and bricks image inpainting from missing large region

Download: Full-size image PowerPoint slide

Figure 7. Edge detection of inpainting results via Canny operator

Download: Full-size image PowerPoint slide

Figure 8. Large region inpainting from images with Gaussian noise by Algorithm 3

Download: Full-size image PowerPoint slide

Related Papers

Cited by

References

[1]	M. Aharon, M. Elad and A. Bruckstein, An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on Signal Processing, 54 (2006). doi: 10.1109/TSP.2006.887825.
[2]	P. Arias, V. Caselles and G. Sapiro, A variational framework for non-local image inpainting, 08 (2009), 345–358.
[3]	M. Bertalmio, G. Sapiro, V. Caselles and C. Ballester, Image inpainting, Siggraph'00, (2000), 417–424. doi: 10.1145/344779.344972.
[4]	M. Bertalmio, L. Vese, G. Sapiro and S. Osher, Simultaneous structure and texture image inpainting, IEEE Transactions on Image Processing, 12 (2003), 882-889. doi: 10.1109/TIP.2003.815261.
[5]	F. Bornemann and T. äMrz, Fast image inpainting based on coherence transport, Journal of Mathematical Imaging & Vision, 28 (2007), 259-278. doi: 10.1007/s10851-007-0017-6.
[6]	A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 2 (2005), 60-65.
[7]	A. Buades, B. Coll and J.-M. Morel, Image enhancement by non-local reverse heat equation, Preprint CMLA, 22 (2006), 2006.
[8]	J.-F. Cai, R. H. Chan, L. Shen and Z. Shen, Convergence analysis of tight framelet approach for missing data recovery, Advances in Computational Mathematics, 31 (2009), 87-113. doi: 10.1007/s10444-008-9084-5.
[9]	J.-F. Cai, R. H. Chan and Z. Shen, A framelet-based image inpainting algorithm, Applied and Computational Harmonic Analysis, 24 (2008), 131-149. doi: 10.1016/j.acha.2007.10.002.
[10]	J.-F. Cai, R. H. Chan and Z. Shen, Simultaneous cartoon and texture inpainting, Inverse Probl. Imaging, 4 (2010), 379-395. doi: 10.3934/ipi.2010.4.379.
[11]	J.-F. Cai, H. Ji, Z. Shen and G.-B. Ye, Data-driven tight frame construction and image denoising, Applied and Computational Harmonic Analysis, 37 (2014), 89-105. doi: 10.1016/j.acha.2013.10.001.
[12]	R. Chan, L. Shen and Z. Shen, A framelet-based approach for image inpainting, Research Report, 4 (2005), 325.
[13]	T. F. Chan, S. H. Kang and J. Shen, Euler's elastica and curvature-based inpainting, SIAM Journal on Applied Mathematics, 63 (2002), 564-592. doi: 10.1137/S0036139901390088.
[14]	T. F. Chan and J. Shen, Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2005. doi: 10.1137/1.9780898717877.
[15]	T. F. Chan and J. Shen, Variational image inpainting, Commun. Pure Appl. Math, 58 (2005), 579-619. doi: 10.1002/cpa.20075.
[16]	T. F. Chan, J. Shen and H.-M. Zhou, Total variation wavelet inpainting, Journal of Mathematical Imaging and Vision, 25 (2006), 107-125. doi: 10.1007/s10851-006-5257-3.
[17]	A. Criminisi, P. Pérez and K. Toyama, Region filling and object removal by exemplar-based image inpainting, IEEE Transactions on Image Processing, 13 (2004), 1200-1212.
[18]	K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Image denoising with block-matching and 3D filtering, in Electronic Imaging 2006, International Society for Optics and Photonics, (2006), 606414–606414.
[19]	B. Dong, H. Ji, J. Li, Z. Shen and Y. Xu, Wavelet frame based blind image inpainting, Applied and Computational Harmonic Analysis, 32 (2012), 268-279. doi: 10.1016/j.acha.2011.06.001.
[20]	B. Dong and Z. Shen, MRA based wavelet frames and applications, IAS Lecture Notes Series, Summer Program on "Mathematics of Image Processing", IAS/Park City Math. Ser., 19, Amer. Math. Soc., Providence, RI, 2013, 9–158. doi: 10.1090/pcms/019/02.
[21]	W. Dong, G. Shi, X. Li, Y. Ma and F. Huang, Compressive sensing via nonlocal low-rank regularization, IEEE Transactions on Image Processing, 23 (2014), 3618-3632. doi: 10.1109/TIP.2014.2329449.
[22]	W. Dong, G. Shi and X. li, Nonlocal image restoration with bilateral variance estimation: A low-rank approach, IEEE Transactions on Image Processing, 22 (2013), 700-711. doi: 10.1109/TIP.2012.2221729.
[23]	M. Elad, J.-L. Starck, P. Querre and D. L. Donoho, Simultaneous cartoon and texture image inpainting using morphological component analysis (MCA), Applied and Computational Harmonic Analysis, 19 (2005), 340-358. doi: 10.1016/j.acha.2005.03.005.
[24]	M. J. Fadili, J. L. Starck and F. Murtagh, Inpainting and zooming using sparse representations, Comput. J., 52 (2009), 64-79.
[25]	G. Gilboa and S. Osher, Nonlocal operators with applications to image processing, Multiscale Model. Simul., 7 (2008), 1005-1028. doi: 10.1137/070698592.
[26]	G. Gilboa, N. Sochen and Y. Zeevi, Image enhancement and denoising by complex diffusion processes, Pattern Analysis and Machine Intelligence, IEEE Transactions on, 26 (2004), 1020–1036.
[27]	G. Gilboa, N. Sochen and Y. Y. Zeevi, Forward-and-backward diffusion processes for adaptive image enhancement and denoising, IEEE Transactions on Image Processing, 11 (2002), 689-703. doi: 10.1109/TIP.2002.800883.
[28]	R. Glowinski and P. Le Tallec, Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics, SIAM, 1989. doi: 10.1137/1.9781611970838.
[29]	H. Ji, Z. Shen and Y. Xu, Wavelet frame based image restoration with missing/damaged pixels, East Asia Journal on Applied Mathematics, 1 (2011), 108-131.
[30]	R. Lai and J. Li, Manifold based low-rank regularization for image restoration and semi-supervised learning, Journal of Scientific Computing, 74 (2018), 1241-1263. doi: 10.1007/s10915-017-0492-x.
[31]	F. Li and T. Zeng, A universal variational framework for sparsity-based image inpainting, IEEE Transactions on Image Processing, 23 (2014), 4242-4254. doi: 10.1109/TIP.2014.2346030.
[32]	T. März, Image inpainting based on coherence transport with adapted distance functions, SIAM Journal on Imaging Sciences, 4 (2011), 981-1000. doi: 10.1137/100807296.
[33]	A. Newson, A. Almansa, Y. Gousseau and P. Pérez, Non-local patch-based image inpainting, IPOL J. Image Processing On Line, 7 (2017), 373-385. doi: 10.5201/ipol.2017.189.
[34]	S. Osher, Z. Shi and W. Zhu, Low dimensional manifold model for image processing, SIAM Journal on Imaging Sciences, 10 (2017), 1669-1690. doi: 10.1137/16M1058686.
[35]	A. Ron and Z. Shen, Affine Systems in $ L_2(\mathbb{R}^d)$: The Analysis of the Analysis Operator, Journal of Functional Analysis, 148 (1997), 408-447. doi: 10.1006/jfan.1996.3079.
[36]	Z. Shen, Wavelet frames and image restorations, Proceedings of the International Congress of Mathematicians, Volume IV, Hindustan Book Agency, New Delhi, 2010, 2834–2863.
[37]	Z. Shi, S. Osher and W. Zhu, Weighted graph laplacian and image inpainting, Journal of Scientific Computing, 577 (2017).
[38]	J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu and T. Huang, Generative image inpainting with contextual attention, 06 (2018), 5505–5514.
[39]	K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155. doi: 10.1109/TIP.2017.2662206.