A Wasserstein distance and total variation regularized model for image reconstruction problems

Yiming Gao

doi:10.3934/ipi.2023045

Article Contents

2024, Volume 18, Issue 3: 600-622. Doi: 10.3934/ipi.2023045

This issue Previous Article How to best combine demosaicing and denoising? Next Article Self-supervised multi-scale neural network for blind deblurring

A Wasserstein distance and total variation regularized model for image reconstruction problems

Yiming Gao

School of Mathematics, Nanjing University of Aeronautics and Astronautics, China

Received: February 2023

Revised: July 2023

Early access: November 7, 2023

Published online: November 7, 2023

The author is supported by the Natural Science Foundation of Jiangsu Province (No. BK20220864) and National Natural Science Foundation of China (No. 12301541).

Abstract / Introduction Full Text(HTML) Figure(6) / Table(3) Related Papers Cited by

Abstract

Optimal transport has gained much attention in image processing fields, such as computer vision, image interpolation and medical image registration. Recently, Bredies et al. (ESAIM: M2AN 54:2351-2382, 2020) and Schmitzer et al. (IEEE T MED IMAGING 39:1626-1635, 2019) established the framework of optimal transport regularization for dynamic inverse problems. In this paper, we incorporate Wasserstein distance, together with total variation, into static inverse problems as a prior regularization. The Wasserstein distance formulated by Benamou-Brenier energy measures the similarity between the given template and the reconstructed image. Also, we analyze the existence of solutions of such variational problems in Radon measure space. Moreover, the first-order primal-dual algorithm is constructed for solving this general imaging problem in a specific grid strategy. Finally, numerical experiments for undersampled MRI reconstruction are presented which show that our proposed model can recover images well with high quality and structure preservation.

Keywords:

Mathematics Subject Classification: Primary: 26A45, 68U10; Secondary: 92C55.

Citation:

Full Text(HTML)

Figure 1. The sampling pattern

Download: Full-size image PowerPoint slide

Figure 2. The testing images named "Phantom" (128$ \times $128), "SheppLogan" (128$ \times $128) and "Brain" (196$ \times $196) and their templates. The templates are made synthetically by performing some deformations on the ground truths

Download: Full-size image PowerPoint slide

Figure 3. The restorations of Zero-filling, TV and the proposed Wass-TV method on image "Phantom." The sampling rates of the first, third and fifth rows are 4.29%, 8.48% and 12.68%, respectively. The even rows exhibit the differences to the ground truth (blue indicates over estimation, and red indicates under estimation)

Download: Full-size image PowerPoint slide

Figure 4. The "Phantom" image reconstruction based on another modality template with 10 spokes (8.48% sampling rate). Wass-TV: PSNR is 29.94; SSIM is 0.9421

Download: Full-size image PowerPoint slide

Figure 5. The restorations of Zero-filling, TV and the proposed Wass-TV method on image "SheppLogan." The sampling rates of the first, second and third rows are 4.29%, 8.48% and 12.68%, respectively. The even rows exhibit the differences to the ground truth (blue indicates over estimation, and red indicates under estimation)

Download: Full-size image PowerPoint slide

Figure 6. The restorations of Zero-filling, TV and the proposed Wass-TV method on image "Brain." The sampling rates of the first, second and third rows are 5.58%, 11.40% and 16.37%, respectively. The even rows exhibit the differences to the ground truth (blue indicates over estimation, and red indicates under estimation)

Download: Full-size image PowerPoint slide

Table 1. PSNR and SSIM values of Zero-filling, TV and our proposed method with different sampling rates on "Phantom" image. The numbers in the brackets (the first column) are the numbers of sampling spokes

sampling rate	PSNR (dB)			SSIM
sampling rate	Zero-filling	TV	Wass-TV	Zero-filling	TV	Wass-TV
4.29% (5)	15.40	15.48	17.26	0.2497	0.3163	0.5344
8.48% (10)	16.90	22.02	30.05	0.3173	0.6918	0.9441
12.68% (15)	18.02	28.70	37.60	0.3695	0.9181	0.9924

| Show Table

DownLoad: CSV

Table 2. PSNR and SSIM values of Zero-filling, TV and our proposed method with different sampling rates on "SheppLogan" image. The numbers in the brackets (the first column) are the numbers of sampling spokes

sampling rate	PSNR (dB)			SSIM
sampling rate	Zero-filling	TV	Wass-TV	Zero-filling	TV	Wass-TV
4.29% (5)	15.40	15.68	17.55	0.2989	0.3636	0.6218
8.48% (10)	16.61	20.80	30.74	0.3158	0.6927	0.9748
12.68% (15)	17.32	28.67	43.54	0.3235	0.8658	0.9979

| Show Table

DownLoad: CSV

Table 3. PSNR and SSIM values of Zero-filling, TV and our proposed method with different sampling rates on "Brain" image. The numbers in the brackets (the first column) are the numbers of sampling spokes

sampling rate	PSNR (dB)			SSIM
sampling rate	Zero-filling	TV	Wass-TV	Zero-filling	TV	Wass-TV
5.58% (10)	16.67	20.03	22.31	0.2813	0.4571	0.7407
11.40% (20)	19.29	27.54	29.74	0.3845	0.8215	0.9451
16.37% (30)	21.14	32.29	34.92	0.4646	0.9595	0.9865

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems, Oxford Mathematical Monographs, Oxford University Press, New York, 2000.
[2]	L. Ambrosio, N. Gigli and G. Savaré, Gradient Flows: In Metric Spaces and in the Space of Probability Measures, Springer Science & Business Media, 2005.
[3]	J. Anderson, Computational Fluid Dynamics, McGraw-Hill Education, 1995.
[4]	M. Arjovsky, S. Chintala and L. Bottou, Wasserstein gan, arXiv preprint, arXiv: 1701.07875, 2017.
[5]	J.-D. Benamou and Y. Brenier, A computational fluid mechanics solution to the monge-kantorovich mass transfer problem, Numerische Mathematik, 84 (2000), 375-393. doi: 10.1007/s002110050002.
[6]	J.-D. Benamou, G. Carlier, M. Cuturi, L. Nenna and G. Peyré, Iterative bregman projections for regularized transportation problems, SIAM Journal on Scientific Computing, 37 (2015), A1111-A1138. doi: 10.1137/141000439.
[7]	K. T. Block, M. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magnetic Resonance in Medicine, 57 (2007), 1086-1098. doi: 10.1002/mrm.21236.
[8]	K. Bredies, M. Carioni and S. Fanzon, A superposition principle for the inhomogeneous continuity equation with Hellinger-Kantorovich-regular coefficients, Communications in Partial Differential Equations, 47 (2022), 2023-2069. doi: 10.1080/03605302.2022.2109172.
[9]	K. Bredies, M. Carioni, S. Fanzon and F. Romero, On the extremal points of the ball of the Benamou-Brenier energy, Bulletin of the London Mathematical Society, 53 (2021), 1436-1452. doi: 10.1112/blms.12509.
[10]	K. Bredies, M. Carioni, S. Fanzon and F. Romero, A generalized conditional gradient method for dynamic inverse problems with optimal transport regularization, Foundations of Computational Mathematics, 23 (2023), 833-898. doi: 10.1007/s10208-022-09561-z.
[11]	K. Bredies, M. Carioni, S. Fanzon and D. Walter, Asymptotic linear convergence of fully-corrective generalized conditional gradient methods, Mathematical Programming, Online First, 2023. doi: 10.1007/s10107-023-01975-z.
[12]	K. Bredies and S. Fanzon, An optimal transport approach for solving dynamic inverse problems in spaces of measures, ESAIM: Mathematical Modelling and Numerical Analysis, 54 (2020), 2351-2382. doi: 10.1051/m2an/2020056.
[13]	K. Bredies, K. Kunisch and T. Pock, Total generalized variation, SIAM Journal on Imaging Sciences, 3 (2010), 492-526. doi: 10.1137/090769521.
[14]	J.-F. Cai, B. Dong, S. Osher and Z. Shen, Image restoration: Total variation, wavelet frames, and beyond, Journal of the American Mathematical Society, 25 (2012), 1033-1089. doi: 10.1090/S0894-0347-2012-00740-1.
[15]	J.-F. Cai, S. Osher and Z. Shen, Split bregman methods and frame based image restoration, SIAM Journal on Multiscale Modeling and Simulation, 8 (2010), 337-369. doi: 10.1137/090753504.
[16]	A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, Journal of Mathematical Imaging and Vision, 40 (2011), 120-145. doi: 10.1007/s10851-010-0251-1.
[17]	C. Chen, B. Gris and O. Öktem, A new variational model for joint image reconstruction and motion estimation in spatiotemporal imaging, SIAM Journal on Imaging Sciences, 12 (2019), 1686-1719. doi: 10.1137/18M1234047.
[18]	C. Chen and O. Öktem, Indirect image registration with large diffeomorphic deformations, SIAM Journal on Imaging Sciences, 11 (2018), 575-617. doi: 10.1137/17M1134627.
[19]	L. Chizat, G. Peyré, B. Schmitzer and F.-X. Vialard, An interpolating distance between optimal transport and fisher-rao metrics, Foundations of Computational Mathematics, 18 (2018), 1-44. doi: 10.1007/s10208-016-9331-y.
[20]	M. Cuturi, Sinkhorn distances: Lightspeed computation of optimal transport, Advances in Neural Information Processing Systems, 26 (2013), 2292-2300.
[21]	V. Duval and R. Tovey, Dynamical programming for off-the-grid dynamic inverse problems, arXiv preprint, arXiv: 2112.11378, 2021.
[22]	H. W. Engl, M. Hanke and A. Neubauer, Regularization of Inverse Problems, volume 375., Springer Science & Business Media, 1996.
[23]	J. H. Fitschen, F. Laus and G. Steidl, Dynamic optimal transport with mixed boundary condition for color image processing, In 2015 International Conference on Sampling Theory and Applications (SampTA), pages 558-562. IEEE, 2015. doi: 10.1109/SAMPTA.2015.7148953.
[24]	W. Gangbo, W. Li, S. Osher and M. Puthawala, Unnormalized optimal transport, Journal of Computational Physics, 399 (2019), 108940. doi: 10.1016/j.jcp.2019.108940.
[25]	Y. Gao and K. Bredies, Infimal convolution of oscillation total generalized variation for the recovery of images with structured texture, SIAM Journal on Imaging Sciences, 11 (2018), 2021-2063. doi: 10.1137/17M1153960.
[26]	B. Gris, Incorporation of a deformation prior in image reconstruction, Journal of Mathematical Imaging and Vision, 61 (2019), 691-709. doi: 10.1007/s10851-018-0868-z.
[27]	B. Gris, C. Chen and O. Öktem, Image reconstruction through metamorphosis, Inverse Problems, 36 (2020), 025001. doi: 10.1088/1361-6420/ab5832.
[28]	I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin and A. Courville, Improved training of wasserstein gans, arXiv preprint, arXiv: 1704.00028, 2017.
[29]	S. Haker, L. Zhu, A. Tannenbaum and S. Angenent, Optimal mass transport for registration and warping, International Journal of Computer Vision, 60 (2004), 225-240. doi: 10.1023/B:VISI.0000036836.66311.97.
[30]	F. H. Harlow and J. E. Welch, Numerical calculation of time-dependent viscous incompressible flow of fluid with free surface, The Physics of Fluids, 8 (1965), 2182-2189. doi: 10.1063/1.1761178.
[31]	M. Henry, E. Maitre and V. Perrier, Primal-dual formulation of the dynamic optimal transport using helmholtz-hodge decomposition, Hal-02087017.
[32]	R. Hug, E. Maitre and N. Papadakis, Multi-physics optimal transportation and image interpolation, ESAIM: Mathematical Modelling and Numerical Analysis, 49 (2015), 1671-1692. doi: 10.1051/m2an/2015038.
[33]	L. V. Kantorovich, On a problem of monge, Uspekhi Mat. Nauk, 3 (1948), 225-226.
[34]	J. Karlsson and A. Ringh, Generalized sinkhorn iterations for regularizing inverse problems using optimal mass transport, SIAM Journal on Imaging Sciences, 10 (2017), 1935-1962. doi: 10.1137/17M111208X.
[35]	J. Liu, W. Yin, W. Li and Y. T. Chow, Multilevel optimal transport: A fast approximation of wasserstein-1 distances, SIAM Journal on Scientific Computing, 43 (2021), A193-A220. doi: 10.1137/18M1219813.
[36]	M. Lustig, D. Donoho and J. M. Pauly, Sparse MRI: The application of compressed sensing for rapid MR imaging, Magnetic Resonance in Medicine, 58 (2007), 1182-1195. doi: 10.1002/mrm.21391.
[37]	J. Maas, M. Rumpf, C. Schönlieb and S. Simon, A generalized model for optimal transport of images including dissipation and density modulation, ESAIM: Mathematical Modelling and Numerical Analysis, 49 (2015), 1745-1769. doi: 10.1051/m2an/2015043.
[38]	J. Maas, M. Rumpf and S. Simon, Generalized optimal transport with singular sources, arXiv preprint, arXiv: 1607.01186, 2016.
[39]	L. Métivier, R. Brossier, Q. Mérigot, É. Oudet and J. Virieux, An optimal transport approach for seismic tomography: Application to 3d full waveform inversion, Inverse Problems, 32 (2016), 115008. doi: 10.1088/0266-5611/32/11/115008.
[40]	G. Monge, Mémoire sur la théorie des déblais et des remblais, Histoire de l'Académie Royale des Sciences de Paris, 1781.
[41]	K. Ni, X. Bresson, T. Chan and S. Esedoglu, Local histogram based segmentation using the wasserstein distance, International Journal of Computer Vision, 84 (2009), 97-111. doi: 10.1007/s11263-009-0234-0.
[42]	O. Öektem, C. Chen, N. O. Domaniç, P. Ravikumar and C. Bajaj, Shape-based image reconstruction using linearized deformations, Inverse Problems, 33 (2017), 035004. doi: 10.1088/1361-6420/aa55af.
[43]	N. Papadakis, G. Peyré and E. Oudet, Optimal transport with proximal splitting, SIAM Journal on Imaging Sciences, 7 (2014), 212-238. doi: 10.1137/130920058.
[44]	G. Peyré, Entropic approximation of wasserstein gradient flows, SIAM Journal on Imaging Sciences, 8 (2015), 2323-2351. doi: 10.1137/15M1010087.
[45]	Y. Rubner, C. Tomasi and L. J. Guibas, The earth mover's distance as a metric for image retrieval, International Journal of Computer Vision, 40 (2000), 99-121. doi: 10.1023/A:1026543900054.
[46]	L. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D, 60 (1992), 259-268. doi: 10.1016/0167-2789(92)90242-F.
[47]	A. Sadeghian, D. Lim, J. Karlsson and J. Li, Automatic target recognition using discrimination based on optimal transport, In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2604-2608. IEEE, 2015. doi: 10.1109/ICASSP.2015.7178442.
[48]	F. Santambrogio, Optimal Transport for Applied Mathematicians, volume 87., Springer, 2015. doi: 10.1007/978-3-319-20828-2.
[49]	O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier and F. Lenzen, Variational Methods in Imaging, Springer, 2009.
[50]	B. Schmitzer, K. P. Schäfers and B. Wirth, Dynamic cell imaging in pet with optimal transport regularization, IEEE Transactions on Medical Imaging, 39 (2019), 1626-1635. doi: 10.1109/TMI.2019.2953773.
[51]	R. Sinkhorn, A relationship between arbitrary positive matrices and doubly stochastic matrices, The Annals of Mathematical Statistics, 35 (1964), 876-879. doi: 10.1214/aoms/1177703591.