Quasiconformal model with CNN features for large deformation image registration

Law Ho; Choi Gary P. T.; Lam Ka Chun; Lui Lok Ming

doi:10.3934/ipi.2022010

Article Contents

2022, Volume 16, Issue 4: 1019-1046. Doi: 10.3934/ipi.2022010 open access

This issue Previous Article A non-iterative sampling method for inverse elastic wave scattering by rough surfaces Next Article The interior inverse scattering problem for a two-layered cavity using the Bayesian method

Quasiconformal model with CNN features for large deformation image registration

1.
School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332, USA
2.
Department of Mathematics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
3.
Machine Learning Team, National Institute of Mental Health, Bethesda, MD 20892, USA
4.
Department of Mathematics, The Chinese University of Hong Kong, Shatin, NT, Hong Kong, China

Received: June 2021

Revised: January 2022

Early access: March 2022

Published: August 2022

This work was supported in part by the National Science Foundation under Grant No. DMS-2002103 (to Gary P. T. Choi), and HKRGC GRF under project ID 14305919 (to Lok Ming Lui)

Abstract / Introduction Full Text(HTML) Figure(11) / Table(1) Related Papers Cited by

Abstract

Image registration has been widely studied over the past several decades, with numerous applications in science, engineering and medicine. Most of the conventional mathematical models for large deformation image registration rely on prescribed landmarks, which usually require tedious manual labeling. In recent years, there has been a surge of interest in the use of machine learning for image registration. In this paper, we develop a novel method for large deformation image registration by a fusion of quasiconformal theory and convolutional neural network (CNN). More specifically, we propose a quasiconformal energy model with a novel fidelity term that incorporates the features extracted using a pre-trained CNN, thereby allowing us to obtain meaningful registration results without any guidance of prescribed landmarks. Moreover, unlike many prior image registration methods, the bijectivity of our method is guaranteed by quasiconformal theory. Experimental results are presented to demonstrate the effectiveness of the proposed method. More broadly, our work sheds light on how rigorous mathematical theories and practical machine learning approaches can be integrated for developing computational methods with improved performance.

Keywords:

Mathematics Subject Classification: 65D18, 68U05, 68U10, 68T07.

Citation:

FullText(HTML)

Figure 1. An illustration of how the Beltrami coefficient $ \mu $ determines the conformality distortion. Under a quasiconformal map $ f $, an infinitesimal circle around a point $ p $ is mapped to an infinitesimal ellipse centered at $ f(p) $, where the major axis length and the minor axis length are given by $ |f_z(p)|(1+|\mu(p)|) $ and $ |f_z(p)|(1-|\mu(p)|) $, respectively. Therefore, the maximal dilation of $ f $ is $ K(f) = \frac{1+||\mu||_{\infty}}{1-||\mu||_{\infty}} $. Also, the orientation change of the major axis of the ellipse is given by $ \arg(\mu(p))/2 $

Download: Full-size image PowerPoint slide

Figure 2. An illustration of receptive field

Download: Full-size image PowerPoint slide

Figure 3. The process of obtaining feature vectors from the images. The two images on the left are the moving image and the fixed image. We first partition both images into smaller patches, and then feed each patch into the truncated classification network to obtain a 3D array of size $ m \times n \times d $, where $ m, n $ are the number of receptive fields along the width and height of the input image depending on stride, kernel and padding size, and $ d $ is the dimension of the feature vector depending on the architecture of the network. We can then transform this 3D array into a vector by vertically stacking along one direction. Here we partition each image into $ 3\times 3 = 9 $ patches for illustrative purposes, and hence 9 vectors in $ \mathbb{R}^{mnd} $ are produced for each of the two images as shown on the right. In practice, a finer partition is often used

Download: Full-size image PowerPoint slide

Figure 4. An illustration of the multiresolution Scheme. We first coarsen both input source and target images as shown in Fig. 4a and Fig. 4b. Then, we run our proposed algorithm on this coarsen pair. From the obtained mapping on this coarsest level, with linear interpolation, we warp the source image on second level to yield Fig. 4c and register it against Fig. 4d using our proposed algorithm. Finally, with the mapping from the last level, we interpolate it to warp the source image on the finest level as shown in Fig. 4e and register it against Fig. 4f. Refer to Fig. 6 for the final registration result

Download: Full-size image PowerPoint slide

Figure 5. The 'Z' to '2' example

Download: Full-size image PowerPoint slide

Figure 6. The eagle example

Download: Full-size image PowerPoint slide

Figure 7. The rabbit example

Download: Full-size image PowerPoint slide

Figure 8. The first hand X-ray example

Download: Full-size image PowerPoint slide

Figure 9. The second hand X-ray example

Download: Full-size image PowerPoint slide

Figure 10. The lung CT example

Download: Full-size image PowerPoint slide

Figure 11. The chest CT example

Download: Full-size image PowerPoint slide

Table 1. The performance of different image registration methods for various synthetic and real medical images. Here, $ E_{\text{sim}} $ measures the accuracy of the registration mapping as described in Equation (37), $ E_{\text{smooth}} $ measures the smoothness of the mapping as described in Equation (38), $ E_{\text{total}} $ measures the overall quality of the mapping as described in Equation (39), and the number of flips reflects the bijectivity of the mapping. For each example and each measure, the best entry among all methods is in bold

${{bf Method}}$	Results ($E_{\text{sim}}$/$E_{\text{smooth}}$/$E_{\text{total}}$/#Flips)
${{bf Method}}$	'Z' to '2' (Fig. 5)	Eagle (Fig. 6)	Rabbit (Fig. 7)	Hand 1 (Fig. 8)	Hand 2 (Fig. 9)	Lung (Fig. 10)	Chest (Fig. 11)
Our method	0.3099 0.4476 0.7575 0	0.0916 0.1558 0.2474 0	0.1953 0.1436 0.3389 0	0.1417 0.1075 0.2492 0	0.1317 0.1536 0.2853 0	0.2435 0.4774 0.7210 0	0.0368 0.1414 0.1783 0
DDemons^[47]	2.0660 0.0481 2.1141 121	0.3724 0.2470 0.6194 3697	0.1806 0.1278 0.3083 174	0.4528 0.2651 0.7180 6036	0.4273 0.2718 0.6991 8925	0.6332 0.2271 0.8602 4191	0.0725 0.3565 0.4290 16316
LDDMM^[5]	2.0488 0.2941 2.3629 105	0.3570 0.2388 0.5968 26	0.5973 0.0738 0.6711 1	0.7018 0.1842 0.8860 0	0.9563 0.1902 1.1465 6	0.6843 0.2568 0.9410 209	0.2446 0.0717 0.3164 0
Elastix^[30]	3.6377 0.9185 4.5562 68280	0.1324 0.1779 0.3103 1158	0.1808 0.1679 0.3487 0	0.1103 0.1513 0.2617 0	0.1856 0.1406 0.3263 0	0.3432 0.3203 0.6635 3579	0.0530 0.1408 0.1938 0
DROP^[19]	2.0554 0.3084 2.3638 2515	0.2509 0.1518 0.4027 0	0.1788 0.1634 0.3423 0	0.3021 0.1186 0.4207 0	1.2514 0.1406 1.3920 0	0.5065 0.6193 1.1259 3663	0.2415 0.1841 0.4256 0

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	The National Lung Screening Trial (NLST), https://cdas.cancer.gov/nlst/.
[2]	Open Access Biomedical Image Search Engine, https://openi.nlm.nih.gov/.
[3]	G. Balakrishnan, A. Zhao, M. R. Sabuncu, J. Guttag and A. V. Dalca, VoxelMorph: A learning framework for deformable medical image registration, IEEE Transactions on Medical Imaging, 38 (2019), 1788-1800. doi: 10.1109/TMI.2019.2897538.
[4]	V. Balntas, E. Johns, L. Tang and K. Mikolajczyk, PN-Net: Conjoined triple deep network for learning local image descriptors, Preprint, arXiv: 1601.05030.
[5]	M. F. Beg, M. I. Miller, A. Trouvé and L. Younes, Computing large deformation metric mappings via geodesic flows of diffeomorphisms, International Journal of Computer Vision, 61 (2005), 139-157. doi: 10.1023/B:VISI.0000043755.93987.aa.
[6]	F. L. Bookstein, Principal warps: Thin-plate splines and the decomposition of deformations, IEEE Transactions on Pattern Analysis and Machine Intelligence, 11 (1989), 567-585. doi: 10.1109/34.24792.
[7]	L. G. Brown, A survey of image registration techniques, ACM Computing Surveys (CSUR), 24 (1992), 325-376. doi: 10.1145/146370.146374.
[8]	G. P. T. Choi, H. L. Chan, R. Yong, S. Ranjitkar, A. Brook, G. Townsend, K. Chen and L. M. Lui, Tooth morphometry using quasi-conformal theory, Pattern Recognition, 99 (2020), 107064. doi: 10.1016/j.patcog.2019.107064.
[9]	G. P. T. Choi, Y. Leung-Liu, X. Gu and L. M. Lui, Parallelizable global conformal parameterization of simply-connected surfaces via partial welding, SIAM J. Imaging Sci., 13 (2020), 1049-1083. doi: 10.1137/19M125337X.
[10]	G. P. T. Choi, Y. Liu and L. M. Lui, Free-boundary conformal parameterization of point clouds, J. Sci. Comput., 90 (2022), Paper No. 14, 26 pp. doi: 10.1007/s10915-021-01641-6.
[11]	G. P.-T. Choi and L. M. Lui, A linear formulation for disk conformal parameterization of simply-connected open surfaces, Adv. Comput. Math., 44 (2018), 87-114. doi: 10.1007/s10444-017-9536-x.
[12]	G. P. T. Choi, D. Qiu and L. M. Lui, Shape analysis via inconsistent surface registration, Proc. A., 476 (2020), 20200147, 21 pp. doi: 10.1098/rspa.2020.0147.
[13]	P. T. Choi, K. C. Lam and L. M. Lui, FLASH: Fast landmark aligned spherical harmonic parameterization for genus-0 closed brain surfaces, SIAM J. Imaging Sci., 8 (2015), 67-94. doi: 10.1137/130950008.
[14]	P. T. Choi and L. M. Lui, Fast disk conformal parameterization of simply-connected open surfaces, J. Sci. Comput., 65 (2015), 1065-1090. doi: 10.1007/s10915-015-9998-2.
[15]	Z. Daoping and K. Chen, 3D orientation-preserving variational models for accurate image registration, SIAM J. Imaging Sci., 13 (2020), 1653-1691. doi: 10.1137/20M1320006.
[16]	B. D. de Vos, F. F. Berendsen, M. A. Viergever, H. Sokooti, M. Staring and I. Išgum, A deep learning framework for unsupervised affine and deformable image registration, Medical Image Analysis, 52 (2019), 128-143.
[17]	F. P. Gardiner and N. Lakic, Quasiconformal Teichmüller Theory, 76, American Mathematical Society, 2000. doi: 10.1090/surv/076.
[18]	B. Glocker, Drop - Deformable registration using discrete optimization, http://campar.in.tum.de/Main/Drop.
[19]	B. Glocker, A. Sotiras, N. Komodakis and N. Paragios, Deformable medical image registration: Setting the state of the art with discrete methods, Annual Review of Biomedical Engineering, 13 (2011), 219-244. doi: 10.1146/annurev-bioeng-071910-124649.
[20]	I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, MIT Press, 2016, http://www.deeplearningbook.org.
[21]	X. Han, T. Leung, Y. Jia, R. Sukthankar and A. C. Berg, MatchNet: Unifying feature and metric learning for patch-based matching, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2015), 3279–3286.
[22]	K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770–778. doi: 10.1109/CVPR.2016.90.
[23]	B. K. P. Horn and B. G. Schunck, Determining optical flow, Artificial Intelligence, 17 (1981), 185-203. doi: 10.1016/0004-3702(81)90024-2.
[24]	G. Huang, Z. Liu, L. Van Der Maaten and K. Q. Weinberger, Densely connected convolutional networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 4700–4708. doi: 10.1109/CVPR.2017.243.
[25]	M. Jahrer, M. Grabner and H. Bischof, Learned local descriptors for recognition and matching, In Computer Vision Winter Workshop, 2 (2008).
[26]	F. Jia, J. Liu and X.-C. Tai, A regularized convolutional neural network for semantic image segmentation, Anal. Appl., 19 (2021), 147-165. doi: 10.1142/S0219530519410148.
[27]	H. J. Johnson and G. E. Christensen, Consistent landmark and intensity-based image registration, IEEE Transactions on Medical Imaging, 21 (2002), 450-461. doi: 10.1109/TMI.2002.1009381.
[28]	S. C. Joshi and M. I. Miller, Landmark matching via large deformation diffeomorphisms, IEEE Transactions on Image Processing, 9 (2000), 1357-1370. doi: 10.1109/83.855431.
[29]	S. Klein and M. Staring, Elastix: A toolbox for rigid and nonrigid registration of images, https://elastix.lumc.nl/.
[30]	S. Klein, M. Staring, K. Murphy, M. A. Viergever and J. P. Pluim, Elastix: A toolbox for intensity-based medical image registration, IEEE Transactions on Medical Imaging, 29 (2010), 196-205. doi: 10.1109/TMI.2009.2035616.
[31]	D.-J. Kroon, Multimodality non-rigid demon algorithm image registration, MATLAB Central File Exchange, https://www.mathworks.com/matlabcentral/fileexchange/21451-multimodality-non-rigid-demon-algorithm-image-registration.
[32]	D. Kuang, Cycle-consistent training for reducing negative jacobian determinant in deep registration networks, In International Workshop on Simulation and Synthesis in Medical Imaging, Springer, 11827 (2019), 120–129. doi: 10.1007/978-3-030-32778-1_13.
[33]	D. Kuang and T. Schmah, FAIM–a ConvNet method for unsupervised 3D medical image registration, In International Workshop on Machine Learning in Medical Imaging, Springer, 11861 (2019), 646–654. doi: 10.1007/978-3-030-32692-0_74.
[34]	K. C. Lam and L. M. Lui, Landmark- and intensity-based registration with large deformations via quasi-conformal maps, SIAM J. Imaging Sci., 7 (2014), 2364-2392. doi: 10.1137/130943406.
[35]	Y. T. Lee, K. C. Lam and L. M. Lui, Landmark-matching transformation with large deformation via n-dimensional quasi-conformal maps, J. Sci. Comput., 67 (2016), 926-954. doi: 10.1007/s10915-015-0113-5.
[36]	O. Lehto and K. I. Virtanen, Quasiconformal Mappings in the Plane, vol. 126, 2$^{nd}$ edition, Springer-Verlag Berlin Heidelberg, 1973.
[37]	H. Lombaert, Diffeomorphic log demons image registration, MATLAB Central File Exchange, https://www.mathworks.com/matlabcentral/fileexchange/39194-diffeomorphic-log-demons-image-registration.
[38]	J. B. A. Maintz and M. A. Viergever, A survey of medical image registration, Medical Image Analysis, 2 (1998), 1-36. doi: 10.1016/S1361-8415(01)80026-8.
[39]	T. W. Meng, G. P.-T. Choi and L. M. Lui, TEMPO: Feature-endowed Teichmüller extremal mappings of point clouds, SIAM J. Imaging Sci., 9 (2016), 1922-1962. doi: 10.1137/15M1049117.
[40]	J. Modersitzki, FAIR: Flexible Algorithms for Image Registration, SIAM, 2009. doi: 10.1137/1.9780898718843.
[41]	I. Rocco, R. Arandjelovic and J. Sivic, Convolutional neural network architecture for geometric matching, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2017), 6148–6157. doi: 10.1109/CVPR.2017.12.
[42]	E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua and F. Moreno-Noguer, Discriminative learning of deep convolutional feature point descriptors, In Proceedings of the IEEE International Conference on Computer Vision, (2015), 118–126. doi: 10.1109/ICCV.2015.22.
[43]	K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, Preprint, arXiv: 1409.1556.
[44]	S. Sommer, Segframe image registration, https://github.com/nefan/segframe.
[45]	A. Sotiras, C. Davatzikos and N. Paragios, Deformable medical image registration: A survey, IEEE Transactions on Medical Imaging, 32 (2013), 1153-1190. doi: 10.1109/TMI.2013.2265603.
[46]	J.-P. Thirion, Image matching as a diffusion process: An analogy with Maxwell's demons, Medical Image Analysis, 2 (1998), 243-260. doi: 10.1016/S1361-8415(98)80022-4.
[47]	T. Vercauteren, X. Pennec, A. Perchant and N. Ayache, Diffeomorphic demons: Efficient non-parametric image registration, NeuroImage, 45 (2009), 61-72. doi: 10.1016/j.neuroimage.2008.10.040.
[48]	H. Wang, L. Dong, J. O'Daniel, R. Mohan, A. S. Garden, K. K. Ang, D. A. Kuban, M. Bonnen, J. Y. Chang and R. Cheung, Validation of an accelerated 'Demons' algorithm for deformable image registration in radiation therapy, Physics in Medicine & Biology, 50 (2005), 2887. doi: 10.1088/0031-9155/50/12/011.
[49]	X. Yang, R. Kwitt, M. Styner and M. Niethammer, Quicksilver: Fast predictive image registration–a deep learning approach, NeuroImage, 158 (2017), 378-396. doi: 10.1016/j.neuroimage.2017.07.008.
[50]	C. P. Yung, G. P. T. Choi, K. Chen and L. M. Lui, Efficient feature-based image registration by mapping sparsified surfaces, Journal of Visual Communication and Image Representation, 55 (2018), 561-571. doi: 10.1016/j.jvcir.2018.07.005.
[51]	S. Zagoruyko and N. Komodakis, Learning to compare image patches via convolutional neural networks, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2015), 4353–4361. doi: 10.1109/CVPR.2015.7299064.
[52]	B. Zitova and J. Flusser, Image registration methods: A survey, Image and Vision Computing, 21 (2003), 977-1000. doi: 10.1016/S0262-8856(03)00137-9.