Model-based deep learning approaches to the Helsinki Tomography Challenge 2022

Clemens Arndt; Alexander Denker; Sören Dittmer; Johannes Leuschner; Judith Nickel; Maximilian Schmidt

doi:10.3934/ammc.2023007

Article Contents

2023, Volume 1, Issue 2: 87-104. Doi: 10.3934/ammc.2023007

This issue Previous Article Next Article Deep image prior with sparsity constraint for limited-angle computed tomography reconstruction

Model-based deep learning approaches to the Helsinki Tomography Challenge 2022

1.
Center for Industrial Mathematics, University of Bremen, Germany
2.
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, United Kingdom

^*Corresponding author: junickel(at)uni-bremen.de. Alphabetical author order
^*Corresponding author: junickel(at)uni-bremen.de. Alphabetical author order

Received: June 2023

Revised: October 2023

Early access: October 2023

Published: December 2023

Abstract / Introduction Full Text(HTML) Figure(15) / Table(2) Related Papers Cited by

Abstract

The Finnish Inverse Problems Society organized the Helsinki Tomography Challenge (HTC) in 2022 to reconstruct an image with limited-angle measurements. We participated in this challenge and developed two methods: an Edge Inpainting method and a Learned Primal-Dual (LPD) network. The Edge Inpainting method involves multiple stages, including classical reconstruction using Perona-Malik, detection of visible edges, inpainting invisible edges using a U-Net, and final segmentation using a U-Net. The LPD approach adapts the classical LPD by using large U-Nets in the primal update and replacing the adjoint with the filtered back projection (FBP). Since the challenge only provided five samples, we generated synthetic data to train the networks. The Edge Inpainting Method performed well for viewing ranges above 70 degrees, while the LPD approach performed well across all viewing ranges and ranked second overall in the challenge.

Keywords:

Mathematics Subject Classification: Primary: 94A08.

Citation:

FullText(HTML)

Figure 1. 2D Fan-beam geometry used for data collection. Adapted from https://fips.fi/HTC2022.php

Download: Full-size image PowerPoint slide

Figure 2. Examples of generated synthetic data samples. The different methods for data generation from left to right: circular holes, polygons, grid of lines, and Gaussian mixture

Download: Full-size image PowerPoint slide

Figure 3. Visualization of the steps of the Edge Inpainting method: (1) variational reconstruction and extraction of visible edges, (2) inpainting of invisible edges, (3) segmentation

Download: Full-size image PowerPoint slide

Figure 4. Here we compare the extracted visible edges via the variational method (middle) and the FBP (right). Both methods use the noisy sinogram on the left with 3% relative additive Gaussian noise

Download: Full-size image PowerPoint slide

Figure 5. The official overall scores of our main methods in the challenge on the different levels in comparison with the method of the winning team. Accessed at https://www.fips.fi/HTCresults.php

Download: Full-size image PowerPoint slide

Figure 6. The official scores of the different variants of the LPD in the challenge

Download: Full-size image PowerPoint slide

Figure 7. Reconstructions of the Edge Inpainting method and of LPD (fine-tuned variant) in level 1 (90°)

Download: Full-size image PowerPoint slide

Figure 8. Reconstructions of the Edge Inpainting method and of LPD (fine-tuned variant) in level 3 (70°)

Download: Full-size image PowerPoint slide

Figure 9. Reconstructions of the Edge Inpainting method and of LPD (fine-tuned variant) in level 5 (50°)

Download: Full-size image PowerPoint slide

Figure 10. Reconstructions of the Edge Inpainting method and of LPD (fine-tuned variant) in level 7 (30°)

Download: Full-size image PowerPoint slide

Figure 11. Visualization of the reconstruction errors for the Edge Inpainting method and LPD (fine-tuned variant) in level 1 (90°), 3 (70°), 5 (50°), 7 (30°): white = true positive (material), black = true negative (air), red = false positive, blue = false negative

Download: Full-size image PowerPoint slide

Figure 12. Visualization of the reconstruction errors for the LPD variants (A) fine-tuned, (B) pre-trained, (C) fine-tuned and equivariance) in level 6 (40°): white = true positive (material), black = true negative (air), red = false positive, blue = false negative

Download: Full-size image PowerPoint slide

Figure 13. Example phantoms, which look significantly different from the challenge phantoms for testing the models on out-of-distribution data

Download: Full-size image PowerPoint slide

Figure 14. Reconstructions and intermediate steps of the Edge Inpainting method on out-of-distribution data with angular range of 90°

Download: Full-size image PowerPoint slide

Figure 15. Reconstructions of the LPD variants on out-of-distribution data with angular range of 80°

Download: Full-size image PowerPoint slide

Table 1. The implementation details of the primal, dual, and segmentation network in the LPD model

	Primal U-Net $ G_{\theta_k} $	Segmentation U-Net $ T_\theta $
scales	4	4
channels	16, 32, 64, 64	16, 32, 64,128
skip channels	16, 32, 32	8, 8, 8
activation function	Leaky ReLU	Leaky ReLU
downsampling	max pooling	max pooling
upsampling	nearest neighbor	nearest neighbor
kernel size	3	3

	Dual CNN $ F_{\theta_k} $
number of layers	4
channels	64
activation function	LeakyReLU
kernel size	3

| Show Table

DownLoad: CSV

Table 2. Details of the training setup and U-Net 's architecture for the inpainting and segmentation task

Kernel size:	9×9	Channels:	16, 32, 64,128,256,256
Scales:	6	Skip channels:	16, 32, 64,128,256
Parameters:	$ \approx $ 34M	Downsampling:	max pooling
Optimizer:	Adam	Upsampling:	nearest neighbor
Batch size:	4	Activation:	LeakyReLU
Loss function:	weighted BCE	Overall gradient steps:	16000

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	J. Adler, H. Kohr and O. Öktem, Operator discretization library (ODL), Zenodo, (2017). doi: 10.5281/zenodo.249479.
[2]	J. Adler, S. Lunz, O. Verdier, C.-B. Schönlieb and O. Öktem, Task adapted reconstruction for inverse problems, Inverse Problems, 38, (2022), Paper No. 075006, 21 pp. doi: 10.1088/1361-6420/ac28ec.
[3]	J. Adler and O. Öktem, Learned primal-dual reconstruction, IEEE Transactions on Medical Imaging, 37, (2018).
[4]	H. Andrade-Loarca, G. Kutyniok, O. Öktem and P. Petersen, Deep microlocal reconstruction for limited-angle tomography, Applied and Computational Harmonic Analysis, 59 (2022), 155-197. doi: 10.1016/j.acha.2021.12.007.
[5]	S. R. Arridge, M. M. Betcke and L. Harhanen, Iterated preconditioned LSQR method for inverse problems on unstructured grids, Inverse Problems, 30 (2014), 075009, 27 pp. doi: 10.1088/0266-5611/30/7/075009.
[6]	S. Arridge, P. Maass, O. Öktem and C.-B. Schönlieb, Solving inverse problems using data-driven models, Acta Numerica, 28 (2019), 1-174. doi: 10.1017/S0962492919000059.
[7]	T. A. Bubba, G Kutyniok, M. Lassa, M. März, W. Samek, S. Siltanen and V. Srinivasan, Learning the invisible: A hybrid deep learning-shearlet framework for limited angle computed tomography, Inverse Problems, 35 (2019), 064002, 38 pp. doi: 10.1088/1361-6420/ab10ca.
[8]	A. Chambolle and T. Pock, An introduction to continuous optimization for imaging, Actra Numerica, Cambridge University Press, 25 (2016), 161-319. doi: 10.1017/S096249291600009X.
[9]	A. Hauptmann, J. Adler, S. Arrdige and O. Öktem, Multi-scale learned iterative reconstruction, IEEE Transactions on Computational Imaging, 6, (2020).
[10]	L. Hörmander, The Analysis of Linear Partial Differential Operators. I. Distribution Theory and Fourier Analysis (reprint of the 2nd edn 1990), Springer, Berlin, 2003. doi: 10.1007/978-3-642-61497-2.
[11]	D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv preprint, arXiv: 1412.6980, (2014).
[12]	A. Meaney, F. Silva de Moura and S. Siltanen, Helsinki Tomography Challenge 2022 open tomographic dataset (HTC 2022), Zenodo, 2022. doi: 10.5281/zenodo.6984868.
[13]	S. Mukherjee, A. Hauptmann, O. Öktem, M. Pereyra and C.-B. Schönlieb, Learned reconstruction methods with convergence guarantees: A survey of concepts and applications, IEEE Signal Processing Magazine, 40 (2023), 164-182. doi: 10.1109/MSP.2022.3207451.
[14]	F. Natterer, The Mathematics of Computerized Tomography, SIAM, 2001. doi: 10.1137/1.9780898719284.
[15]	E. T. Quinto, Singularities of the X-ray transform and limited data tomography in $ \mathbb{R}^2$ and $ \mathbb{R}^3$, SIAM J. Math. Anal., 24, (1993), 1215-1225. doi: 10.1137/0524069.
[16]	O. Ronneberger, P. Fischer and T. Brox, U-net: Convolutional networks for biomedical image segmentation, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Springer, 9351 (2015), 234-241. doi: 10.1007/978-3-319-24574-4_28.
[17]	W. van Aarle, W. J. Palenstijn, J. De Beenhouwer, T. Altantzis, S. Bals, K. Joost Batenburg and J. Sijbers, The ASTRA toolbox: A platform for advanced algorithm development in electron tomography, Ultramicroscopy, 157 (2015), 35-47.. doi: 10.1016/j.ultramic.2015.05.002.
[18]	Y. Wu and K. He, Group normalization, Proceedings of the European Conference on Computer Vision (ECCV), (2018), 3-19. doi: 10.1007/978-3-030-01261-8_1.
[19]	S. Xie and Z. Tu, Holistically-nested edge detection, IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, (2015), 1395-1403.