# American Institute of Mathematical Sciences

August  2013, 7(3): 863-884. doi: 10.3934/ipi.2013.7.863

## A conformal approach for surface inpainting

 1 Department of Mathematics, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong, China 2 Department of Mathematics, The Chinese University of Hong Kong, Shatin, New Territories, Hong Kong,, China 3 Department of Computer Sciences, Room 2425, Computer Science Building, State University of New York at Stony Brook, Stony Brook, New York 11794-4400

Received  December 2012 Revised  April 2013 Published  September 2013

We address the problem of surface inpainting, which aims to fill in holes or missing regions on a Riemann surface based on its surface geometry. In practical situation, surfaces obtained from range scanners often have holes or missing regions where the 3D models are incomplete. In order to analyze the 3D shapes effectively, restoring the incomplete shape by filling in the surface holes is necessary. In this paper, we propose a novel conformal approach to inpaint surface holes on a Riemann surface based on its surface geometry. The basic idea is to represent the Riemann surface using its conformal factor and mean curvature. According to Riemann surface theory, a Riemann surface can be uniquely determined by its conformal factor and mean curvature up to a rigid motion. Given a Riemann surface $S$, its mean curvature $H$ and conformal factor $\lambda$ can be computed easily through its conformal parameterization. Conversely, given $\lambda$ and $H$, a Riemann surface can be uniquely reconstructed by solving the Gauss-Codazzi equation on the conformal parameter domain. Hence, the conformal factor and the mean curvature are two geometric quantities fully describing the surface. With this $\lambda$-$H$ representation of the surface, the problem of surface inpainting can be reduced to the problem of image inpainting of $\lambda$ and $H$ on the conformal parameter domain. The inpainting of $\lambda$ and $H$ can be done by conventional image inpainting models. Once $\lambda$ and $H$ are inpainted, a Riemann surface can be reconstructed which effectively restores the 3D surface with missing holes. Since the inpainting model is based on the geometric quantities $\lambda$ and $H$, the restored surface follows the surface geometric pattern as much as possible. We test the proposed algorithm on synthetic data, 3D human face data and MRI-derived brain surfaces. Experimental results show that our proposed method is an effective surface inpainting algorithm to fill in surface holes on an incomplete 3D models based their surface geometry.
Citation: Lok Ming Lui, Chengfeng Wen, Xianfeng Gu. A conformal approach for surface inpainting. Inverse Problems & Imaging, 2013, 7 (3) : 863-884. doi: 10.3934/ipi.2013.7.863
##### References:
 [1] T. F. Chan and J. Shen, Variational restoration of non-flat image features: Models and algorithms,, SIAM Journal on Applied Mathematics, 61 (2001), 1338. doi: 10.1137/S003613999935799X. [2] T. F. Chan and J. Shen, Mathematical models for local nontexture inpaintings,, SIAM Journal on Applied Mathematics, 62 (): 1019. doi: 10.1137/S0036139900368844. [3] S. Esedoglu and J. Shen, Digital inpainting based on the Mumford-Shah-Euler image model,, Eureopean Journal on Applied Mathematics, 13 (2002), 353. doi: 10.1017/S0956792502004904. [4] T. F. Chan and J. Shen, Variational image inpainting,, Communications in Pure and Applied Mathematics, 58 (2005), 579. doi: 10.1002/cpa.20075. [5] T. F. Chan, S. H. Kang and J. Shen, Euler's elastica and curvature-based inpainting,, SIAM Journal on Applied Mathematics, 63 (2002), 564. doi: 10.1137/S0036139901390088. [6] S. Masnou and J. Morel, Level lines based disocclusion,, $5^{th}$ IEEE International Conference on Image Processing, 3 (1998), 259. doi: 10.1109/ICIP.1998.999016. [7] M. Bertalmio, A. L. Bertozzi and G. Sapiro, Navier-Stokes, fluid dynamics, and image and video inpainting,, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1 (2001), 355. doi: 10.1109/CVPR.2001.990497. [8] T. F. Chan, J. Shen and H.-M. Zhou, Total variation wavelet inpainting,, Journal of Mathematical Imaging and Visions, 25 (2006), 107. doi: 10.1007/s10851-006-5257-3. [9] J. A. Dobrosotskaya and A. L. Bertozzi, A wavelet-Laplace variational technique for image deconvolution and inpainting,, IEEE Transaction on Image Processing, 17 (2008), 657. doi: 10.1109/TIP.2008.919367. [10] A. Tsai, A. Yezzi, Jr. and A. S. Willsky, Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation and magnification,, IEEE Transaction on Image Processing, 10 (2001), 1169. doi: 10.1109/83.935033. [11] M. Desbrun, M. Meyer, P. Schrder and A. H. Barr, Implicit fairing of irregular meshes using diffusion and curvature flow,, in, (1999), 317. doi: 10.1145/311535.311576. [12] G. Taubin, Geometric signal processing on polygonal meshes,, Eurographics, (2000). [13] P. Smereka, Semi implicit level set methods curvature surface diffusion motion,, Journal of Scientific Computing, 19 (2003), 439. doi: 10.1023/A:1025324613450. [14] M. Desbrun, M. Meyer, P. Schröder and A. H. Barr, Anisotropic feature preserving denoising of height fields and bivariate data,, Graphics Interface, (2000). [15] M. Desbrun, M. Meyer, P. Schröder and A. H. Barr, Discrete differential geometry operators in nD,, in, (2002), 35. [16] U. Clarenz, U. Diewald and M. Rumpf, Anisotropic diffusion in surface processing,, in, (2000), 397. doi: 10.1109/VISUAL.2000.885721. [17] C. L. Bajaj and G. Xu, Anisotropic diffusion of subdivision surfaces and functions on surfaces,, ACM Transaction on Graphics, 22 (2003), 4. [18] T. Tasdizen, R. Whitaker, P. Burchard and S. Osher, Geometric surface processing via anisotropic diffusion of normals,, IEEE Visualization, (2002), 125. [19] T. Tasdizen, R. Whitaker, P. Burchard and S. Osher, Geometric surface processing via normal maps,, ACM Transactions on Graphics, 22 (2003), 1012. doi: 10.1145/944020.944024. [20] V. Caselles, G. Haro, G. Sapiro and J. Verdera, On geometric variational models for inpainting surface holes,, Computer Vision and Image Understanding, 111 (2008), 351. doi: 10.1016/j.cviu.2008.01.002. [21] J. Verdera, V. Caselles, M. Bertalmio and G. Sapiro, Inpainting surface holes,, in, (2003), 903. doi: 10.1109/ICIP.2003.1246828. [22] J. Davis, S. Marschner, M. Garr and M. Levoy, Filling holes in complex surfaces using volumetric diffusion,, in, (2002), 428. doi: 10.1109/TDPVT.2002.1024098. [23] U. Clarenz, U. Diewald, G. Dziuk, M. Rumpf and R. Rusu, A finite element method for surface restoration with smooth boundary conditions,, Computer Aided Geometric Design, 21 (2004), 427. doi: 10.1016/j.cagd.2004.02.004. [24] A. Sharf, M. Alexa and D. Cohen, Context-based surface completion,, ACM Transaction on Graphics, 23 (2004), 878. [25] V. Savchenko and N. Kojekine, An approach to blend surfaces,, Advances in Modeling, (2002), 139. doi: 10.1007/978-1-4471-0103-1_9. [26] M. K. Hurdal and K. Stephenson, Discrete conformal methods for cortical brain flattening,, NeuroImage, 45 (2009). doi: 10.1016/j.neuroimage.2008.10.045. [27] B. Fischl, M. Sereno, R. Tootell and A. Dale, High-resolution intersubject averaging and a coordinate system for the cortical surface,, Human Brain Mapping, 8 (1999), 272. [28] X. Gu, Y. Wang, T. F. Chan, P. M. Thompson and S.-T. Yau, Genus zero surface conformal mapping and its application to brain surface mapping,, IEEE Transactions on Medical Imaging, 23 (2004), 949. [29] Y. Wang, L. M. Lui, X. Gu, K. M. Hayashi, T. F. Chan, A. W. Toga, P. M. Thompson and S.-T. Yau, Brain surface conformal parameterization using Riemann surface structure,, IEEE Transactions on Medical Imaging, 26 (2007), 853. doi: 10.1109/TMI.2007.895464. [30] X. Gu and S.-T. Yau, Computing conformal structures of surfaces,, Communication in Information System, 2 (2002), 121. [31] X. Gu, Y. Wang and S.-T. Yau, Geometric compression using Riemann surface structure,, Communication in Information System, 3 (2004), 171. [32] L. M. Lui, T. W. Wong, W. Zeng, X. Gu, P. M. Thompson, T. F. Chan and S.-T. Yau, Optimization of surface registrations using Beltrami holomorphic flow,, Journal of Scientific Computing, 50 (2012), 557. doi: 10.1007/s10915-011-9506-2. [33] L. M. Lui, T. W. Wong, X. Gu, P. M. Thompson, T. F. Chan and S.-T. Yau, Hippocampal shape registration using Beltrami holomorphic flow,, Medical Image Computing and Computer Assisted Intervention(MICCAI), 6362 (2010), 323. [34] L. M. Lui, K. C. Lam, T. W. Wong and X. Gu, Texture map and video compression using Beltrami representatio,, to appear in SIAM Journal on Imaging Sciences, (2013). [35] S. Haker, S. Angenent, A. Tannenbaum, R. Kikinis, G. Sapiro and M. Halle, Conformal surface parameterization for texture mapping,, IEEE Transaction of Visualization and Computer Graphics, 6 (2000), 181. [36] L. M. Lui, T. W. Wong, X. Gu, T. F. Chan and S.-T. Yau, Compression of surface diffeomorphism using Beltrami coefficient,, IEEE Computer Vision and Pattern Recognition(CVPR), (2010), 2839. [37] W. Zeng, L. M. Lui, F. Luo, T. F. Chan, S.-T. Yau and X. Gu, Computing quasiconformal maps using an auxiliary metric and discrete curvature flow,, Numerische Mathematik, 121 (2012), 671. doi: 10.1007/s00211-012-0446-z. [38] L. M. Lui, K. C. Lam, S.-T. Yau and X. Gu, Teichmüller extremal mapping and its applications to landmark matching registration,, , (). [39] W. Zeng and X. Gu, Registration for 3D surfaces with large deformations using quasi-conformal curvature flow,, IEEE Conference on Computer Vision and Pattern Recognition (CVPR1), (2011), 20. [40] H. Zhao, S. Osher, B. Merriman and and M. Kang, Implicit and non-parametric shape reconstruction from unorganized points using variational level set method,, Computer Vision and Image Understanding, 80 (2000), 295. [41] J. Liang, F. Park and H. Zhao, Robust and efficient implicit surface reconstruction of point clouds based on convexified image segmentation,, Journal of Scientific Computing, 54 (2013), 577. doi: 10.1007/s10915-012-9674-8. [42] A. Wong and H. Zhao, Computation of quasiconformal surface maps using discrete Beltrami flow,, UCLA CAM report 12-85., (): 12.

show all references

##### References:
 [1] T. F. Chan and J. Shen, Variational restoration of non-flat image features: Models and algorithms,, SIAM Journal on Applied Mathematics, 61 (2001), 1338. doi: 10.1137/S003613999935799X. [2] T. F. Chan and J. Shen, Mathematical models for local nontexture inpaintings,, SIAM Journal on Applied Mathematics, 62 (): 1019. doi: 10.1137/S0036139900368844. [3] S. Esedoglu and J. Shen, Digital inpainting based on the Mumford-Shah-Euler image model,, Eureopean Journal on Applied Mathematics, 13 (2002), 353. doi: 10.1017/S0956792502004904. [4] T. F. Chan and J. Shen, Variational image inpainting,, Communications in Pure and Applied Mathematics, 58 (2005), 579. doi: 10.1002/cpa.20075. [5] T. F. Chan, S. H. Kang and J. Shen, Euler's elastica and curvature-based inpainting,, SIAM Journal on Applied Mathematics, 63 (2002), 564. doi: 10.1137/S0036139901390088. [6] S. Masnou and J. Morel, Level lines based disocclusion,, $5^{th}$ IEEE International Conference on Image Processing, 3 (1998), 259. doi: 10.1109/ICIP.1998.999016. [7] M. Bertalmio, A. L. Bertozzi and G. Sapiro, Navier-Stokes, fluid dynamics, and image and video inpainting,, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1 (2001), 355. doi: 10.1109/CVPR.2001.990497. [8] T. F. Chan, J. Shen and H.-M. Zhou, Total variation wavelet inpainting,, Journal of Mathematical Imaging and Visions, 25 (2006), 107. doi: 10.1007/s10851-006-5257-3. [9] J. A. Dobrosotskaya and A. L. Bertozzi, A wavelet-Laplace variational technique for image deconvolution and inpainting,, IEEE Transaction on Image Processing, 17 (2008), 657. doi: 10.1109/TIP.2008.919367. [10] A. Tsai, A. Yezzi, Jr. and A. S. Willsky, Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation and magnification,, IEEE Transaction on Image Processing, 10 (2001), 1169. doi: 10.1109/83.935033. [11] M. Desbrun, M. Meyer, P. Schrder and A. H. Barr, Implicit fairing of irregular meshes using diffusion and curvature flow,, in, (1999), 317. doi: 10.1145/311535.311576. [12] G. Taubin, Geometric signal processing on polygonal meshes,, Eurographics, (2000). [13] P. Smereka, Semi implicit level set methods curvature surface diffusion motion,, Journal of Scientific Computing, 19 (2003), 439. doi: 10.1023/A:1025324613450. [14] M. Desbrun, M. Meyer, P. Schröder and A. H. Barr, Anisotropic feature preserving denoising of height fields and bivariate data,, Graphics Interface, (2000). [15] M. Desbrun, M. Meyer, P. Schröder and A. H. Barr, Discrete differential geometry operators in nD,, in, (2002), 35. [16] U. Clarenz, U. Diewald and M. Rumpf, Anisotropic diffusion in surface processing,, in, (2000), 397. doi: 10.1109/VISUAL.2000.885721. [17] C. L. Bajaj and G. Xu, Anisotropic diffusion of subdivision surfaces and functions on surfaces,, ACM Transaction on Graphics, 22 (2003), 4. [18] T. Tasdizen, R. Whitaker, P. Burchard and S. Osher, Geometric surface processing via anisotropic diffusion of normals,, IEEE Visualization, (2002), 125. [19] T. Tasdizen, R. Whitaker, P. Burchard and S. Osher, Geometric surface processing via normal maps,, ACM Transactions on Graphics, 22 (2003), 1012. doi: 10.1145/944020.944024. [20] V. Caselles, G. Haro, G. Sapiro and J. Verdera, On geometric variational models for inpainting surface holes,, Computer Vision and Image Understanding, 111 (2008), 351. doi: 10.1016/j.cviu.2008.01.002. [21] J. Verdera, V. Caselles, M. Bertalmio and G. Sapiro, Inpainting surface holes,, in, (2003), 903. doi: 10.1109/ICIP.2003.1246828. [22] J. Davis, S. Marschner, M. Garr and M. Levoy, Filling holes in complex surfaces using volumetric diffusion,, in, (2002), 428. doi: 10.1109/TDPVT.2002.1024098. [23] U. Clarenz, U. Diewald, G. Dziuk, M. Rumpf and R. Rusu, A finite element method for surface restoration with smooth boundary conditions,, Computer Aided Geometric Design, 21 (2004), 427. doi: 10.1016/j.cagd.2004.02.004. [24] A. Sharf, M. Alexa and D. Cohen, Context-based surface completion,, ACM Transaction on Graphics, 23 (2004), 878. [25] V. Savchenko and N. Kojekine, An approach to blend surfaces,, Advances in Modeling, (2002), 139. doi: 10.1007/978-1-4471-0103-1_9. [26] M. K. Hurdal and K. Stephenson, Discrete conformal methods for cortical brain flattening,, NeuroImage, 45 (2009). doi: 10.1016/j.neuroimage.2008.10.045. [27] B. Fischl, M. Sereno, R. Tootell and A. Dale, High-resolution intersubject averaging and a coordinate system for the cortical surface,, Human Brain Mapping, 8 (1999), 272. [28] X. Gu, Y. Wang, T. F. Chan, P. M. Thompson and S.-T. Yau, Genus zero surface conformal mapping and its application to brain surface mapping,, IEEE Transactions on Medical Imaging, 23 (2004), 949. [29] Y. Wang, L. M. Lui, X. Gu, K. M. Hayashi, T. F. Chan, A. W. Toga, P. M. Thompson and S.-T. Yau, Brain surface conformal parameterization using Riemann surface structure,, IEEE Transactions on Medical Imaging, 26 (2007), 853. doi: 10.1109/TMI.2007.895464. [30] X. Gu and S.-T. Yau, Computing conformal structures of surfaces,, Communication in Information System, 2 (2002), 121. [31] X. Gu, Y. Wang and S.-T. Yau, Geometric compression using Riemann surface structure,, Communication in Information System, 3 (2004), 171. [32] L. M. Lui, T. W. Wong, W. Zeng, X. Gu, P. M. Thompson, T. F. Chan and S.-T. Yau, Optimization of surface registrations using Beltrami holomorphic flow,, Journal of Scientific Computing, 50 (2012), 557. doi: 10.1007/s10915-011-9506-2. [33] L. M. Lui, T. W. Wong, X. Gu, P. M. Thompson, T. F. Chan and S.-T. Yau, Hippocampal shape registration using Beltrami holomorphic flow,, Medical Image Computing and Computer Assisted Intervention(MICCAI), 6362 (2010), 323. [34] L. M. Lui, K. C. Lam, T. W. Wong and X. Gu, Texture map and video compression using Beltrami representatio,, to appear in SIAM Journal on Imaging Sciences, (2013). [35] S. Haker, S. Angenent, A. Tannenbaum, R. Kikinis, G. Sapiro and M. Halle, Conformal surface parameterization for texture mapping,, IEEE Transaction of Visualization and Computer Graphics, 6 (2000), 181. [36] L. M. Lui, T. W. Wong, X. Gu, T. F. Chan and S.-T. Yau, Compression of surface diffeomorphism using Beltrami coefficient,, IEEE Computer Vision and Pattern Recognition(CVPR), (2010), 2839. [37] W. Zeng, L. M. Lui, F. Luo, T. F. Chan, S.-T. Yau and X. Gu, Computing quasiconformal maps using an auxiliary metric and discrete curvature flow,, Numerische Mathematik, 121 (2012), 671. doi: 10.1007/s00211-012-0446-z. [38] L. M. Lui, K. C. Lam, S.-T. Yau and X. Gu, Teichmüller extremal mapping and its applications to landmark matching registration,, , (). [39] W. Zeng and X. Gu, Registration for 3D surfaces with large deformations using quasi-conformal curvature flow,, IEEE Conference on Computer Vision and Pattern Recognition (CVPR1), (2011), 20. [40] H. Zhao, S. Osher, B. Merriman and and M. Kang, Implicit and non-parametric shape reconstruction from unorganized points using variational level set method,, Computer Vision and Image Understanding, 80 (2000), 295. [41] J. Liang, F. Park and H. Zhao, Robust and efficient implicit surface reconstruction of point clouds based on convexified image segmentation,, Journal of Scientific Computing, 54 (2013), 577. doi: 10.1007/s10915-012-9674-8. [42] A. Wong and H. Zhao, Computation of quasiconformal surface maps using discrete Beltrami flow,, UCLA CAM report 12-85., (): 12.
 [1] Sebastian Acosta. A control approach to recover the wave speed (conformal factor) from one measurement. Inverse Problems & Imaging, 2015, 9 (2) : 301-315. doi: 10.3934/ipi.2015.9.301 [2] Georgi I. Kamberov. Recovering the shape of a surface from the mean curvature. Conference Publications, 1998, 1998 (Special) : 353-359. doi: 10.3934/proc.1998.1998.353 [3] Ali Hyder, Luca Martinazzi. Conformal metrics on $\mathbb{R}^{2m}$ with constant Q-curvature, prescribed volume and asymptotic behavior. Discrete & Continuous Dynamical Systems - A, 2015, 35 (1) : 283-299. doi: 10.3934/dcds.2015.35.283 [4] Peter Haïssinsky, Kevin M. Pilgrim. Examples of coarse expanding conformal maps. Discrete & Continuous Dynamical Systems - A, 2012, 32 (7) : 2403-2416. doi: 10.3934/dcds.2012.32.2403 [5] Zuxing Xuan. On conformal measures of parabolic meromorphic functions. Discrete & Continuous Dynamical Systems - B, 2015, 20 (1) : 249-257. doi: 10.3934/dcdsb.2015.20.249 [6] Nicholas Hoell, Guillaume Bal. Ray transforms on a conformal class of curves. Inverse Problems & Imaging, 2014, 8 (1) : 103-125. doi: 10.3934/ipi.2014.8.103 [7] Tomasz Szarek, Mariusz Urbański, Anna Zdunik. Continuity of Hausdorff measure for conformal dynamical systems. Discrete & Continuous Dynamical Systems - A, 2013, 33 (10) : 4647-4692. doi: 10.3934/dcds.2013.33.4647 [8] Hans Henrik Rugh. On dimensions of conformal repellers. Randomness and parameter dependency. Discrete & Continuous Dynamical Systems - A, 2012, 32 (7) : 2553-2564. doi: 10.3934/dcds.2012.32.2553 [9] Mario Roy, Mariusz Urbański. Multifractal analysis for conformal graph directed Markov systems. Discrete & Continuous Dynamical Systems - A, 2009, 25 (2) : 627-650. doi: 10.3934/dcds.2009.25.627 [10] Domenico Mucci. Maps into projective spaces: Liquid crystal and conformal energies. Discrete & Continuous Dynamical Systems - B, 2012, 17 (2) : 597-635. doi: 10.3934/dcdsb.2012.17.597 [11] Rossen I. Ivanov. Conformal and Geometric Properties of the Camassa-Holm Hierarchy. Discrete & Continuous Dynamical Systems - A, 2007, 19 (3) : 545-554. doi: 10.3934/dcds.2007.19.545 [12] Juan Wang, Yongluo Cao, Yun Zhao. Dimension estimates in non-conformal setting. Discrete & Continuous Dynamical Systems - A, 2014, 34 (9) : 3847-3873. doi: 10.3934/dcds.2014.34.3847 [13] Marcelo M. Disconzi. On the existence of solutions and causality for relativistic viscous conformal fluids. Communications on Pure & Applied Analysis, 2019, 18 (4) : 1567-1599. doi: 10.3934/cpaa.2019075 [14] Guido De Philippis, Antonio De Rosa, Jonas Hirsch. The area blow up set for bounded mean curvature submanifolds with respect to elliptic surface energy functionals. Discrete & Continuous Dynamical Systems - A, 2019, 0 (0) : 1-28. doi: 10.3934/dcds.2019243 [15] Robert Eymard, Angela Handlovičová, Karol Mikula. Approximation of nonlinear parabolic equations using a family of conformal and non-conformal schemes. Communications on Pure & Applied Analysis, 2012, 11 (1) : 147-172. doi: 10.3934/cpaa.2012.11.147 [16] Jérôme Bertrand. Prescription of Gauss curvature on compact hyperbolic orbifolds. Discrete & Continuous Dynamical Systems - A, 2014, 34 (4) : 1269-1284. doi: 10.3934/dcds.2014.34.1269 [17] Dimitra Antonopoulou, Georgia Karali. A nonlinear partial differential equation for the volume preserving mean curvature flow. Networks & Heterogeneous Media, 2013, 8 (1) : 9-22. doi: 10.3934/nhm.2013.8.9 [18] Chiara Corsato, Franco Obersnel, Pierpaolo Omari, Sabrina Rivetti. On the lower and upper solution method for the prescribed mean curvature equation in Minkowski space. Conference Publications, 2013, 2013 (special) : 159-169. doi: 10.3934/proc.2013.2013.159 [19] Chiara Corsato, Colette De Coster, Pierpaolo Omari. Radially symmetric solutions of an anisotropic mean curvature equation modeling the corneal shape. Conference Publications, 2015, 2015 (special) : 297-303. doi: 10.3934/proc.2015.0297 [20] Elias M. Guio, Ricardo Sa Earp. Existence and non-existence for a mean curvature equation in hyperbolic space. Communications on Pure & Applied Analysis, 2005, 4 (3) : 549-568. doi: 10.3934/cpaa.2005.4.549

2017 Impact Factor: 1.465