Limited-angle tomography reconstruction via deep end-to-end learning on synthetic data

Thomas Germer; Jan Robine; Sebastian Konietzny; Stefan Harmeling; Tobias Uelwer

doi:10.3934/ammc.2023006

Article Contents

2023, Volume 1, Issue 2: 126-142. Doi: 10.3934/ammc.2023006

This issue Previous Article Deep image prior with sparsity constraint for limited-angle computed tomography reconstruction Next Article A directional regularization method for the limited-angle Helsinki Tomography Challenge using the Core Imaging Library (CIL)

Limited-angle tomography reconstruction via deep end-to-end learning on synthetic data

1.
Heinrich Heine University Düsseldorf, Germany
2.
Technical University of Dortmund, Germany

^*Corresponding author: Thomas Germer
^*Corresponding author: Thomas Germer

Received: June 2023

Revised: September 2023

Early access: October 2023

Published: December 2023

Abstract / Introduction Full Text(HTML) Figure(14) / Table(2) Related Papers Cited by

Abstract

Computed tomography (CT) has become an essential part of modern science and medicine. A CT scanner consists of an X-ray source that is spun around an object of interest. On the opposite end of the X-ray source, a detector captures X-rays that are not absorbed by the object. The reconstruction of an image is a linear inverse problem, which is usually solved by filtered back projection. However, when the number of measurements is small, the reconstruction problem is ill-posed. This is for example the case when the X-ray source is not spun completely around the object, but rather irradiates the object only from a limited angle. To tackle this problem, we present a deep neural network that is trained on a large amount of carefully-crafted synthetic data and can perform limited-angle tomography reconstruction even for only 30° or 40° sinograms. With our approach we won the first place in the Helsinki Tomography Challenge 2022.

Keywords:

Mathematics Subject Classification: Primary: 92C55, 68T07; Secondary: 68U10, 94A08.

Citation:

FullText(HTML)

Figure 1. X-ray tomography experimental setup for the Helsinki Tomography Challenge 2022 [24]

Download: Full-size image PowerPoint slide

Figure 2. Example object and its corresponding sinogram

Download: Full-size image PowerPoint slide

Figure 3. Given a full sinogram (A), the filtered back projection (FBP) algorithm can reconstruct the original image (C). However, when the angular range of the sinogram is limited (B), the FBP reconstruction (D) shows severe artifacts. Our method (E) can reconstruct the image from the limited angle sinogram

Download: Full-size image PowerPoint slide

Figure 4. An exemplary overview of the four steps that make up the generation process

Download: Full-size image PowerPoint slide

Figure 5. Illustration of our neural network architecture

Download: Full-size image PowerPoint slide

Figure 6. Comparison of Matthews Correlation Coefficient (MCC) for different teams participating in the Helsinki Tomography Challenge 2022. The score is the sum over three different reconstructions per difficulty level. The scores of our method (Team 15) are denoted by the blue line with star-shaped markers

Download: Full-size image PowerPoint slide

Figure 7. Qualitative comparison of binarized reconstructions from limited angle sinograms

Download: Full-size image PowerPoint slide

Figure 8. MCC score during training of a model over 100 epochs on sinograms with an angular range of 40 degrees (left, orange) and 30 degrees (right, orange) compared to our baseline trained on multiple angular ranges (blue)

Download: Full-size image PowerPoint slide

Figure 9. Test scores for two models trained on sinograms with different numbers of angular ranges

Download: Full-size image PowerPoint slide

Figure 10. Averaged MCC scores for 100 synthetically generated images translated horizontally by up to 30 pixels

Download: Full-size image PowerPoint slide

Figure 11. Additional cross shapes

Download: Full-size image PowerPoint slide

Figure 12. MCC scores for model trained on dataset with additional cross shapes

Download: Full-size image PowerPoint slide

Figure 13. Comparison of MCC scores for models trained on datasets with different size on difficulty level 7 of the HTC 2022

Download: Full-size image PowerPoint slide

Figure 14. Example of a difficult shape configuration

Download: Full-size image PowerPoint slide

Table 1. The levels of difficulty of the reconstruction task of the HTC 2022. The fewer measurements $ m $ are given, the harder is the reconstruction task (left to right)

Level	1	2	3	4	5	6	7
$ m $	181	161	141	121	101	81	61
Angular range	90°	80°	70°	60°	50°	40°	30°

| Show Table

DownLoad: CSV

Table 2. Quantitative evaluation on the HTC 2022 test data with 3 images per level. PSNR and SSIM are calculated on the non-binary reconstructions. The reported MCC value is calculated as the sum of the MCC values of the binary reconstructions

Level	MCC			PSNR			SSIM
	FBP	FBP+NN	Our method	FBP	FBP+NN	Our method	FBP	FBP+NN	Our method
1	1.92	2.92	2.96	10.74	24.34	26.01	0.46	0.66	0.67
2	2.00	2.96	2.97	10.54	23.43	25.61	0.45	0.62	0.68
3	1.89	2.94	2.93	10.48	22.70	23.34	0.42	0.69	0.69
4	1.83	2.92	2.92	10.33	21.65	23.36	0.39	0.70	0.70
5	1.54	2.89	2.93	9.86	20.88	23.94	0.38	0.66	0.68
6	1.07	2.45	2.81	8.93	15.36	21.17	0.34	0.61	0.68
7	0.84	2.31	2.41	7.89	14.44	16.40	0.31	0.59	0.63

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	J. Adler and O. Öktem, Learned primal-dual reconstruction, IEEE Transactions on Medical Imaging, 37 (2018), 1322-1332. doi: 10.1109/TMI.2018.2799231.
[2]	S. Barutcu, S. Aslan, A. K. Katsaggelos and D. Gürsoy, Limited-angle computed tomography with deep image and physics priors, Scientific Reports, 11 (2021), Article number: 17740. doi: 10.1038/s41598-021-97226-2.
[3]	R. A. Brooks and G. Di Chiro, Beam hardening in X-ray reconstructive tomography, Physics in Medicine and Biology, 21 (1976), 390. doi: 10.1088/0031-9155/21/3/004.
[4]	T. A. Bubba, G. Kutyniok, M. Lassas, M. März, W. Samek, S. Siltanen and V. Srinivasan, Learning the invisible: A hybrid deep learning-shearlet framework for limited angle computed tomography, Inverse Problems, 35 (2019), 064002, 38 pp. doi: 10.1088/1361-6420/ab10ca.
[5]	J. Dong, J. Fu and Z. He, A deep learning reconstruction framework for X-ray computed tomography with incomplete data, PLoS ONE, 14 (2019), e0224426. doi: 10.1371/journal.pone.0224426.
[6]	C. Douarre, R. Schielein, C. Frindel, S. Gerth and D. Rousseau, Transfer learning from synthetic data applied to soil-root segmentation in x-ray tomography images, Journal of Imaging, 4 (2018), 65. doi: 10.3390/jimaging4050065.
[7]	D. S. Ebert, F. Kenton Musgrave, D. Peachey, K. Perlin and S. Worley, Texturing and Modeling: A Procedural Approach, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 3rd edition, 2002.
[8]	L. A. Feldkamp, L. C. Davis and J. W. Kress, Practical cone-beam algorithm, Journal of the Optical Society of America, 1 (1984), 612-619. doi: 10.1364/JOSAA.1.000612.
[9]	J. Frikel and E. T. Quinto, Characterization and reduction of artifacts in limited angle tomography, Inverse Problems, 29 (2013), 125007, 21 pp. doi: 10.1088/0266-5611/29/12/125007.
[10]	M. Genzel, I. Gühring, J. Macdonald and M. März, Near-exact recovery for tomographic inverse problems via deep learning, in Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, PMLR, 2022, 7368-7381.
[11]	I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, Generative adversarial networks, Commun. ACM, 63 (2020), 139-144. doi: 10.1145/3422622.
[12]	A. Goy, G. Rughoobur, S. Li, K. Arthur, A. I. Akinwande and G. Barbastathis, High-resolution limited-angle phase tomography of dense layered objects using deep neural networks, Proceedings of the National Academy of Sciences, 116 (2019), 19848-19856. doi: 10.1073/pnas.1821378116.
[13]	K. He, X. Zhang, S. Ren and J. Sun, Deep residual learning for image recognition, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 770-778.
[14]	D. Hendrycks and K. Gimpel, Gaussian error linear units (GELUs), arXiv preprint, arXiv: 1606.08415, 2016.
[15]	G. N. Hounsfield, Computerized transverse axial scanning (tomography): Part 1. Description of system, The British Journal of Radiology, 46 (1973), 1016-1022. doi: 10.1259/0007-1285-46-552-1016.
[16]	T. Inouye, Image reconstruction with limited angle projection data, IEEE Transactions on Nuclear Science, 26 (1979), 2665-2669. doi: 10.1109/TNS.1979.4330507.
[17]	S. Ioffe and C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in International Conference on Machine Learning, PMLR, (2015), 448-456.
[18]	K. H. Jin, M. T. McCann, E. Froustey and M. Unser, Deep convolutional neural network for inverse problems in imaging, IEEE Transactions on Image Processing, 26 (2017), 4509-4522. doi: 10.1109/TIP.2017.2713099.
[19]	D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv preprint, arXiv: 1412.6980, 2014.
[20]	H. Lee, J. Lee, H. Kim, B. Cho and S. Cho, Deep-neural-network-based sinogram synthesis for sparse-view CT image reconstruction, IEEE Transactions on Radiation and Plasma Medical Sciences, 3 (2019), 109-119. doi: 10.1109/TRPMS.2018.2867611.
[21]	B. M. Licea-Kane, R. J. Rost, D. Ginsburg, J. M. Kessenich, B. Lichtenbelt, H. Malan and M. Weiblen, OpenGL Shading Language, Addison-Wesley Educational, Boston, MA, 3 edition, 2009.
[22]	Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell and S. Xie, A convNet for the 2020s, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, 11966-11976. doi: 10.1109/CVPR52688.2022.01167.
[23]	C. M. McLeavy, M. H. Chunara, R. J. Gravell, A. Rauf, A. Cushnie, C. Staley Talbot and R. M. Hawkins, The future of CT: Deep learning reconstruction, Clinical Radiology, 76 (2021), 407-415. doi: 10.1016/j.crad.2021.01.010.
[24]	A. Meaney, F. Silva de Moura and S. Siltanen, Helsinki tomography challenge 2022 open tomographic dataset (HTC 2022), 2022.
[25]	J. L. Mueller and S. Siltanen, Linear and Nonlinear Inverse Problems with Practical Applications, Society for Industrial and Applied Mathematics, Philadelphia, PA, 2012. doi: 10.1137/1.9781611972344.
[26]	L. T. Niklason, B. T. Christian, L. E. Niklason, D. B. Kopans, D. E. Castleberry, B. H. Opsahl-Ong, C. E. Landberg, P. J. Slanetz, A. A. Giardino, R. Moore, D. Albagli, M. C. DeJule, P. F. Fitzgerald, D. F. Fobare, B. W. Giambattista, R. F. Kwasnick, J. Liu, S. J. Lubowski, G. E. Possin, J. F. Richotte, C. Y. Wei and R. F. Wirth, Digital tomosynthesis in breast imaging, Radiology, 205 (1997), 399-406. doi: 10.1148/radiology.205.2.9356620.
[27]	P. Paschalis, N. D. Giokaris, A. Karabarbounis, G. K. Loudos, D. Maintas, C. N. Papanicolas, V. Spanoudaki, C. h. Tsoumpas and E. Stiliaris, Tomographic image reconstruction using artificial neural networks, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Proceedings of the 2nd International Conference on Imaging Technologies in Biomedical Sciences, 527 (2004), 211-215. doi: 10.1016/j.nima.2004.03.122.
[28]	E. T. Quinto, Singularities of the X-ray transform and limited data tomography in $\mathbb{R}^2$ and $\mathbb{R}^3$, SIAM Journal on Mathematical Analysis, 24 (1993), 1215-1225. doi: 10.1137/0524069.
[29]	E. Riba, D. Mishkin, D. Ponsa, E. Rublee and G. Bradski, Kornia: An open source differentiable computer vision library for PyTorch, in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020, 3663-3672. doi: 10.1109/WACV45572.2020.9093363.
[30]	O. Ronneberger, P. Fischer and T. Brox, U-net: Convolutional networks for biomedical image segmentation, in Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Cham, Springer International Publishing, 2015, 234-241. doi: 10.1007/978-3-319-24574-4_28.
[31]	J. Schwab, S. Antholzer and M. Haltmeier, Deep null space learning for inverse problems: Convergence analysis and rates, Inverse Problems, 35 (2019), 025008, 13 pp. doi: 10.1088/1361-6420/aaf14a.
[32]	L. A. Shepp and B. F. Logan, The Fourier reconstruction of a head section, IEEE Transactions on Nuclear Science, 21 (1974), 21-43. doi: 10.1109/TNS.1974.6499235.
[33]	E. Y. Sidky and X. Pan, Report on the AAPM deep-learning sparse-view CT grand challenge, Medical Physics, 49 (2022), 4935-4943. doi: 10.1002/mp.15489.
[34]	W. van Aarle, W. Jan Palenstijn, J. Cant, E. Janssens, F. Bleichrodt, A. Dabravolski, J. De Beenhouwer, K. Joost Batenburg and J. Sijbers, Fast and flexible X-ray tomography using the ASTRA toolbox, Optics Express, 24 (2016), 25129-25147.
[35]	W. van Aarle, W. Jan Palenstijn, J. De Beenhouwer, T. Altantzis, S. Bals, K. Joost Batenburg and J. Sijbers, The ASTRA toolbox: A platform for advanced algorithm development in electron tomography, Ultramicroscopy, 157 (2015), 35-47.
[36]	T. Würfl, F. C. Ghesu, V. Christlein and A. Maier, Deep learning computed tomography, in Sebastien Ourselin, Leo Joskowicz, Mert R. Sabuncu, Gozde Unal, and William Wells, editors, Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016, Cham, Springer International Publishing, 2016, 432-440.
[37]	T. Würfl, M. Hoffmann, V. Christlein, K. Breininger, Y. Huang, M. Unberath and A. K. Maier, Deep learning computed tomography: Learning projection-domain weights from image domain in limited angle problems, IEEE Transactions on Medical Imaging, 37 (2018), 1454-1463.
[38]	X. Yang, M. Kahnt, D. Brückner, A. Schropp, Y. Fam, J. Becher, J.-D. Grunwaldt, T. L. Sheppard and C. G. Schroer, Tomographic reconstruction with a generative adversarial network, Journal of Synchrotron Radiation, 27 (2020), 486-493. doi: 10.1107/S1600577520000831.
[39]	D. Yim, B. Kim and S. Lee, Limited-angle CT reconstruction via data-driven deep neural network, in Hilde Bosmans, Wei Zhao and Lifeng Yu, editors, Medical Imaging 2021: Physics of Medical Imaging. SPIE, 2021. doi: 10.1117/12.2580692.
[40]	Q. Zhang, Z. Hu, C. Jiang, H. Zheng, Y. Ge and D. Liang, Artifact removal using a hybrid-domain convolutional neural network for limited-angle computed tomography imaging, Physics in Medicine & Biology, 65 (2020), 155010. doi: 10.1088/1361-6560/ab9066.
[41]	B. Zhu, J. Z. Liu, S. F. Cauley, B. R. Rosen and M. S. Rosen, Image reconstruction by domain-transform manifold learning, Nature, 555 (2018), 487-492. doi: 10.1038/nature25988.