Data-consistent neural networks for solving nonlinear inverse problems

Yoeri E. Boink; Markus Haltmeier; Sean Holman; Johannes Schwab

doi:10.3934/ipi.2022037

Article Contents

2023, Volume 17, Issue 1: 203-229. Doi: 10.3934/ipi.2022037

This issue Previous Article The enclosure method for the detection of variable order in fractional diffusion equations Next Article A colorization-based anisotropic variational model for vector-valued image compression

Data-consistent neural networks for solving nonlinear inverse problems

1.
Department of Applied Mathematics, University of Twente, Netherlands
2.
Department of Mathematics, University of Innsbruck, Austria
3.
Department of Mathematics, University of Manchester, United Kingdom

^*Corresponding author: Johannes Schwab
^*Corresponding author: Johannes Schwab

Received: May 2021

Revised: May 2022

Early access: July 28, 2022

Published online: July 28, 2022

Abstract / Introduction Full Text(HTML) Figure(16) / Table(3) Related Papers Cited by

Abstract

Data assisted reconstruction algorithms, incorporating trained neural networks, are a novel paradigm for solving inverse problems. One approach is to first apply a classical reconstruction method and then apply a neural network to improve its solution. Empirical evidence shows that plain two-step methods provide high-quality reconstructions, but they lack a convergence analysis as known for classical regularization methods. In this paper we formalize the use of such two-step approaches in the context of classical regularization theory. We propose data-consistent neural networks that can be combined with classical regularization methods. This yields a data-driven regularization method for which we provide a convergence analysis with respect to noise. Numerical simulations show that compared to standard two-step deep learning methods, our approach provides better stability with respect to out of distribution examples in the test set, while performing similarly on test data drawn from the distribution of the training set. Our method provides a stable solution approach to inverse problems that beneficially combines the known nonlinear forward model with available information on the desired solution manifold in training data.

Keywords:

Mathematics Subject Classification: Primary: 65J20, 68T07, 65J22; Secondary: 45F05.

Citation:

Full Text(HTML)

Figure 1. On the left, a standard post-processing network. The red box illustrates that the output in general does not reproduce the data under the forward operation $ \mathcal{F} $. On the right a data-consistent network architecture. The green box illustrates that it does reproduce the data under the forward operation $ \mathcal{F} $

Download: Full-size image PowerPoint slide

Figure 2. The figure illustrates the solution set for given data $ y $ and the output of the network $ \mathbf{U} $ acting on the solution after applying a right inverse of $ \mathcal{F} $. The generalized projection operator approximated by some iterative scheme or closed form operator is shown in green and blue respectively

Download: Full-size image PowerPoint slide

Figure 3. Two-dimensional visualization of the data-consistent network for the 'projection on convex set' problem explained in Section 4.1. The blue region indicates the convex set $ C $, the green region (that extends infinitely to the right) indicates the affine normal cone to $ C $ at $ \mathcal{P}_C(z) $. The output of the data-consistent network is indicated by $ \Phi_0(z) $; it can be seen that it is obtained by taking the input $ z $, applying a Lipschitz continuous neural network $ \mathbf{U}(z) $ and projecting it to the normal cone $ \mathcal N_C $ at $ \mathcal{P}_C(z) $. This ensures that $ \mathcal{P}_C( \Phi_0(z)) = \mathcal{P}_C(z) $, as required by the definition of a data-consistent network

Download: Full-size image PowerPoint slide

Figure 7. Reconstructions of a typical sample from the modified test set. Top: reconstructed sinograms with all 8 angles in different colors. Bottom: reconstructed images (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Figure 4. Reconstructions of a sample from the regular test set. In the bottom the horizontal central slice is shown. Both U-Net and data-consistent network provide an almost perfect reconstruction

Download: Full-size image PowerPoint slide

Figure 5. Reconstructions of a sample from the modified test set. In the bottom the horizontal central slice is shown. Data-consistency makes sure that intensity is only changed above the saturation level

Download: Full-size image PowerPoint slide

Figure 6. Reconstructions of a typical sample from the regular test set. Top: reconstructed sinograms with all 8 angles in different colors. Bottom: reconstructed images (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Figure 8.

Download: Full-size image PowerPoint slide

Figure 9.

Download: Full-size image PowerPoint slide

Figure 10.

Download: Full-size image PowerPoint slide

Figure 11. Sample for which the data-consistent PSNR value is relatively high compared to the U-Net PSNR values (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Figure 12. Sample for which the data-consistent PSNR value is approximately the same as the U-Net PSNR values (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Figure 13. Sample for which the data-invariant PSNR value is relatively low compared to the U-Net PSNR values (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Figure 14. Sample for which the data-invariant PSNR value is relatively high compared to the U-Net PSNR values (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Figure 15. Sample for which the data-invariant PSNR value is approximately the same as the U-Net PSNR values (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Figure 16. Sample for which the data-invariant PSNR value is relatively low compared to the U-Net PSNR values (grayscale from 0 to 1)

Download: Full-size image PowerPoint slide

Table 1. U-Net parameter details for all simulation experiments

	Exp. 1 $ ( \mathbf{U}/ {{\mathit{\boldsymbol{\Phi}}}}_0) $:	Exp. 2 $ ( \mathbf{U}_1/ {{\mathit{\boldsymbol{\Phi}}}}_0^{(1)}) $:	Exp. 2 $ ( \mathbf{U}_2/ {{\mathit{\boldsymbol{\Phi}}}}_0^{(2)}) $:
	image domain	image domain	sinogram domain
$ \# $training samples	1024	35584	35584
$ \# $validation samples	256	3522	3522
$ \# $test samples	1024	3553	3553
depth	4	4	4
width	2	2	2
$ \# $channels in top layer	8	16	16
convolution size	$ 3\times3 $	$ 3\times3 $	$ 3\times3 $
nonlinearity	ReLU	ReLU	ReLU
start learning rate	$ 10^{-3} $	$ 10^{-3} $	$ 10^{-3} $
final learning rate	$ 10^{-4} $	$ 2\cdot10^{-4} $	$ 2\cdot10^{-4} $
batch size	64	32	32
$ \# $epochs	1000	25	25

| Show Table

DownLoad: CSV

Table 2. Comparison of PSNR and SSIM for all reconstruction methods

	PSNR			SSIM
	Pseudo-inverse	U-Net	Data-consistent	Pseudo-inverse	U-Net	Data-consistent
Regular set	$ 24.2 \pm2.2 $	$ 60.6 \pm2.1 $	$ 66.7 \pm1.6 $	$ 0.56 \pm 0.08 $	$ 1.00 \pm0.00 $	$ 1.00 \pm0.00 $
Modified set	$ 48.0 \pm7.8 $	$ 36.9 \pm2.9 $	$ 48.0 \pm4.4 $	$ 0.99 \pm 0.01 $	$ 0.92 \pm0.03 $	$ 0.97 \pm0.01 $

| Show Table

DownLoad: CSV

Table 3. Comparison of PSNR, SSIM and data-fidelity $ \Vert{ \mathbf{F}(\tilde{ x})- \mathbf{F}( x)}\Vert_{\ell^2} $ for all reconstruction methods, where $ \tilde{ x} $ is the reconstruction and $ x $ is the ground truth. Note that the data-fidelity of the pseudo-inverse in theory should be zero since it is a right inverse of the forward operator. The non-zero values result from numerical instabilities in the computation of this operator

	PSNR
	Pseudo-inverse	One U-Net	Two U-Nets	Data-consistent
Regular set	$ 23.1 \pm2.3 $	$ 30.5 \pm1.5 $	$ 31.0 \pm1.5 $	$ 30.1 \pm1.9 $
Modified set	$ 29.1 \pm1.6 $	$ 27.5 \pm1.7 $	$ 28.3 \pm1.4 $	$ 29.9 \pm1.2 $
	SSIM
	Pseudo-inverse	One U-Net	Two U-Nets	Data-consistent
Regular set	$ 0.50 \pm0.07 $	$ 0.82 \pm0.04 $	$ 0.83 \pm0.04 $	$ 0.74 \pm0.07 $
Modified set	$ 0.71 \pm0.07 $	$ 0.74 \pm0.05 $	$ 0.73 \pm0.05 $	$ 0.75 \pm0.05 $
	Data-fidelity
	Pseudo-inverse	One U-Net	Two U-Nets	Data-consistent
Regular set	$ 6.1 \pm3.3 $	$ \quad 4.8 \pm1.5 $	$ 3.9 \pm1.1 $	$ 0.9 \pm0.4 $
Modified set	$ 0.4 \pm0.2 $	$ 11.9 \pm5.4 $	$ 8.5 \pm2.8 $	$ 0.6 \pm0.2 $

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	A. Adler, V. Emiya, M. G. Jafari, M. Elad, R. Gribonval and M. D. Plumbley, A constrained matching pursuit approach to audio declipping, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2011), 329–332. doi: 10.1109/ICASSP.2011.5946407.
[2]	J. Adler and O. Öktem, Solving ill-posed inverse problems using iterative deep neural networks, Inverse Probl., 33 (2017), 124007. doi: 10.1088/1361-6420/aa9581.
[3]	S. Antholzer, M. Haltmeier and J. Schwab, Deep learning for photoacoustic tomography from sparse data, Inverse Problems in Science and Engineering, 27 (2019), 987-1005. doi: 10.1080/17415977.2018.1518444.
[4]	S. Armato III, G. McLennan, L. Bidaut, M. McNitt-Gray, C. Meyer, A. Reeves, B. Zhao, D. Aberle, C. Henschke and E. Hoffman, et al., The lung image database consortium (LIDC) and image database resource initiative (IDRI): A completed reference database of lung nodules on CT scans, Medical Physics, 38 (2011), 915-931.
[5]	S. Arridge, P. Maass, O. Öktem and C. B. Schönlieb, Solving inverse problems using data-driven models, Acta Numerica, 28 (2019), 1-174. doi: 10.1017/S0962492919000059.
[6]	A. B. Bakushinsky and M. Yu. Kokurin, Iterative Methods for Approximate Solution of Inverse Problems, Springer, 2004
[7]	H. H. Bauschke and J. M. Borwein, On projection algorithms for solving convex feasibility problems, SIAM Review, 38 (1996), 367-426. doi: 10.1137/S0036144593251710.
[8]	Y. Boink and C. Brune, Learned SVD: Solving inverse problems via hybrid autoencoding, arXiv: 1912.10840, (2020).
[9]	Y. Boink, S. Manohar and C. Brune, A partially learned algorithm for joint photoacoustic reconstruction and segmentation, IEEE Transactions on Medical Imaging, 39 (2020), 129-139. doi: 10.1109/TMI.2019.2922026.
[10]	T. A. Bubba, G. Kutyniok, M. Lassas, M. März, W. Samek, S. Siltanen and V. Srinivasan, Learning the invisible: A hybrid deep learning-shearlet framework for limited angle computed tomography, Inverse Problems, 35 (2019), 064002. doi: 10.1088/1361-6420/ab10ca.
[11]	H. W. Engl, M. Hanke and A. Neubauer, Regularization of Inverse Problems, Kluwer Academic Publishers Group, 1996.
[12]	D. Gilton, G. Ongie and R. Willett, Neumann networks for inverse problems in imaging, IEEE Trans. Comput. Imaging, 6 (2020), 328–343. arXiv: 1901.03707, (2019). doi: 10.1109/TCI.2019.2948732.
[13]	H. Gouk, E. Frank, B. Pfahringer and M. J. Cree, Regularisation of neural networks by enforcing lipschitz continuity, Machine Learning, 110 (2021), 393-416. doi: 10.1007/s10994-020-05929-w.
[14]	I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin and A. Courville, Improved training of wasserstein gans, arXiv Preprint, arXiv: 1704.00028, (2017).
[15]	S. J. Hamilton and A. Hauptmann, Deep D-bar: Real time electrical impedance tomography imaging with deep neural networks, IEEE Trans. Med. Imag., 37 (2018), 2367-2377. doi: 10.1109/TMI.2018.2828303.
[16]	P. C. Hansen, Discrete Inverse Problems: Insight and Algorithms, SIAM, 2010. doi: 10.1137/1.9780898718836.
[17]	K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition, (2016), 770–778. doi: 10.1109/CVPR.2016.90.
[18]	S. Honig and M. Werman, Image declipping with deep networks, 2018 25th IEEE International Conference on Image Processing (ICIP), (2018), 3923–3927. doi: 10.1109/ICIP.2018.8451780.
[19]	M. V. de Hoop, M. Lassas and C. A. Wong, Deep learning architectures for nonlinear operator functions and nonlinear inverse problems, Math. Stat. Learn., 4 (2021), 1–86. arXiv: 1912.11090, (2019). doi: 10.4171/msl/28.
[20]	K. H. Jin, M. T. McCann, E. Froustey and M. Unser, Deep convolutional neural network for inverse problems in imaging, IEEE Trans. Image Process., 26 (2017), 4509-4522. doi: 10.1109/TIP.2017.2713099.
[21]	B. Kaltenbacher, A. Neubauer and O. Scherzer, Iterative Regularization Methods for Nonlinear Ill-Posed Problems, , Walter de Gruyter GmbH & Co. KG, Berlin, 2008. doi: 10.1515/9783110208276.
[22]	E. Kobler, T. Klatzer, K. Hammernik and T. Pock, Variational networks: Connecting variational methods and deep learning, German Conference on Pattern Recognition, (2017), 281–293. doi: 10.1007/978-3-319-66709-6.
[23]	A. Kofler, M. Haltmeier, C. Kolbitsch, M. Kachelrieß and M. Dewey, A U-Nets cascade for sparse view computed tomography, International Workshop on Machine Learning for Medical Image Reconstruction, (2018), 91–99. doi: 10.1007/978-3-030-00129-2_11.
[24]	D. Lee, J. Yoo and J. C. Ye, Deep residual learning for compressed sensing MRI, IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), 15–18. doi: 10.1109/ISBI.2017.7950457.
[25]	R. M. Lewitt, Multidimensional digital image representations using generalized Kaiser-Bessel window functions, JOSA A, 7 (1990), 1834-1846. doi: 10.1364/JOSAA.7.001834.
[26]	J. Leuschner, M. Schmidt, D. Baguer and P. Maaß, The LoDoPaB-CT dataset: A benchmark dataset for low-dose CT reconstruction methods, arXiv: 1910.01113, (2019).
[27]	H. Li, J. Schwab, S. Antholzer and M. Haltmeier, NETT: Solving inverse problems with deep neural networks, Inverse Problems, Online First, 36 (2020), 065005. doi: 10.1088/1361-6420/ab6d57.
[28]	S. Lunz, O. Öktem and C. Schönlieb, Adversarial regularizers in inverse problems, Advances in Neural Information Processing Systems, (2018), 8507–8516.
[29]	M. Mardani, E. Gong, J. Cheng, S. Vasanawala, G. Zaharchuk, M. Alley, N. Thakur, W. Han, J. Pauly, et al., Deep generative adversarial networks for compressed sensing automates MRI, arXiv: 1706.00051, (2017)
[30]	V. A. Morozov, Methods for Solving Incorrectly Posed Problems, Springer Verlag, 1984. doi: 10.1007/978-1-4612-5280-1.
[31]	S. Mukherjee, S. Dittmer, Z. Shumaylov, S. Lunz, O. Öktem and C. Schönlieb, Learned convex regularizers for inverse problems, arXiv Preprint, arXiv: 2008.02839, (2020).
[32]	J. Rick Chang, C. Li, B. Poczos, B. Vijaya and A. Sankaranarayanan, One network to solve them all-solving linear inverse problems using deep projection models, Proceedings of The IEEE International Conference on Computer Vision, (2017), 5888–5897. doi: 10.1109/ICCV.2017.627.
[33]	Y. Rivenson, Z. Göröcs, H. Günaydin, Y. Zhang, H. Wang and A. Ozcan, Deep learning microscopy, Optica, 4 (2017), 1437-1443. doi: 10.1364/OPTICA.4.001437.
[34]	O. Ronneberger, P. Fischer and T. Brox, U-net: Convolutional networks for biomedical image segmentation, International Conference on Medical Image Computing and Computer-Assisted Intervention, (2015), 234–241. doi: 10.1007/978-3-319-24574-4_28.
[35]	O. Scherzer, M. Grasmair, H. Grossauer, M. Haltmeier and F. Lenzen, Variational Methods in Imaging, Springer, 2009.
[36]	J. Schwab, S. Antholzer and M. Haltmeier, Deep null space learning for inverse problems: Convergence analysis and rates, Inverse Problems, 35 (2019), 025008. doi: 10.1088/1361-6420/aaf14a.
[37]	J. Schwab, S. Antholzer and M. Haltmeier, Big in Japan: Regularizing networks for solving inverse problems, Journal of Mathematical Imaging and Vision, 62 (2020), 445-455. doi: 10.1007/s10851-019-00911-1.
[38]	J. Sun, H. Li, Z. Xu, et al., Deep ADMM-Net for compressive sensing MRI, Advances in Neural Information Processing Systems, (2016), 10–18.
[39]	A. N. Tikhonov and V. Y. Arsenin, Solutions of Ill-Posed Problems, John Wiley & Sons, 1977.
[40]	X. Zhang, J. Wang and L. Xing, Metal artifact reduction in x-ray computed tomography (CT) by constrained optimization, Medical Physics, 38 (2011), 701-711. doi: 10.1118/1.3533711.
[41]	B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen and M. S. Rosen, Image reconstruction by domain-transform manifold learning, Nature, 555 (2018), 487-492. doi: 10.1038/nature25988.