Learning to scan: A deep reinforcement learning approach for personalized scanning in CT imaging

Ziju Shen; Yufei Wang; Dufan Wu; Xu Yang; Bin Dong

doi:10.3934/ipi.2021045

Article Contents

2022, Volume 16, Issue 1: 179-195. Doi: 10.3934/ipi.2021045

This issue Previous Article Smoothing Newton method for

$ \ell^0 $

$ \ell^2 $

regularized linear inverse problem Next Article Identification and stability of small-sized dislocations using a direct algorithm

Learning to scan: A deep reinforcement learning approach for personalized scanning in CT imaging

1.
Academy for Advanced Interdisciplinary Studies, Peking University, China
2.
Computer Science Department, Carnegie Mellon University, USA
3.
Center for Advanced Medical Computing and Analysis, Massachusetts General Hospital and Harvard Medical School, USA
4.
Department of Mathematics, University of California, Santa Barbara, USA
5.
Beijing International Center for Mathematical Research, Center for Data Science, Institute for Artificial Intelligence, Peking University, China

^† Corresponding Author
^† Corresponding Author

^* Equal Contribution

Received: September 30, 2020

Revised: March 31, 2021

Early access: July 2021

Published: February 2022

Abstract Full Text(HTML) Figure(6) / Table(1) Related Papers Cited by

Abstract

. Computed Tomography (CT) takes X-ray measurements on the subjects to reconstruct tomographic images. As X-ray is radioactive, it is desirable to control the total amount of dose of X-ray for safety concerns. Therefore, we can only select a limited number of measurement angles and assign each of them limited amount of dose. Traditional methods such as compressed sensing usually randomly select the angles and equally distribute the allowed dose on them. In most CT reconstruction models, the emphasize is on designing effective image representations, while much less emphasize is on improving the scanning strategy. The simple scanning strategy of random angle selection and equal dose distribution performs well in general, but they may not be ideal for each individual subject. It is more desirable to design a personalized scanning strategy for each subject to obtain better reconstruction result. In this paper, we propose to use Reinforcement Learning (RL) to learn a personalized scanning policy to select the angles and the dose at each chosen angle for each individual subject. We first formulate the CT scanning process as an Markov Decision Process (MDP), and then use modern deep RL methods to solve it. The learned personalized scanning strategy not only leads to better reconstruction results, but also shows strong generalization to be combined with different reconstruction algorithms.

Keywords:

Mathematics Subject Classification: Primary: 94A08, 92C55.

Citation:

Full Text(HTML)

Figure 1. Policy network architecture. Each MLP contains two hidden layers with 512 neurons. We use a multi-layer GRU which contains 3 recurrent layers and each layer has 256 neurons. The Angle MLP has one hidden layer of 512 neurons, and the Dose MLP has 2 hidden layers with 512 neurons

Download: Full-size image PowerPoint slide

Figure 2. Histograms of PSNR and SSIM from all the 350 test images as shown in Table 1. Figures in (a), (b) and (c) correspond to the three different noise levels. For each (a), (b) and (c), the first row is PSNR and the second is SSIM. Figures from left to right are results from reconstruction methods SART, TV, WF and PD-net respectively. Every sub-figure contains histograms of three scanning strategies, i.e., RL-AD, DS-ED and UF-AEC

Download: Full-size image PowerPoint slide

Figure 3. Two examples of the reconstructed images. The top row contains the ground truth images and their zoom-in views. The second through the fourth row contain results from UF-AEC, DS-ED and RL-AD respectively, and combined with PD-net's reconstruction. Note that RL-AD selects 65 measurement angles for the subject in (a) and 54 measurement angles for the subject in (b)

Download: Full-size image PowerPoint slide

Figure 4. (a) Distribution of number of measurements of the learned policy (RL-AD) on all the 350 testing CT images. (b) Dose usage distribution of 8 images that use around 54 measurements. (c) Dose usage distribution of 8 images that use around 65 measurements

Download: Full-size image PowerPoint slide

Figure 5. (a): an example image that takes 54 measurements. (b): an example image that takes 64 measurements. We can see that the images for which RL selects more measurement angles contains more structures

Download: Full-size image PowerPoint slide

Figure 6. The angle selection of the CT image in Figure 5. Top row: RL-AD, bottom row: DS-ED. The lines show the selected angles

Download: Full-size image PowerPoint slide

Table 1. This table presents comparisons of different scanning strategies (1-3rd column for RL-AD, DS-ED and UF-AEC respectively) combined with different image reconstruction methods (1-4th row for SART, TV, WF and PD-net respectively). Last row presents the inference times of angle selection (in seconds) of the three compared scanning strategies. The mean (std) of the PSNR and SSIM of the reconstructed images and the inference times are computed among all 350 testing CT images. The best results among the compared algorithms are shown in bold numbers

Reconstruction Method		RL-AD	DS-ED	UF-AEC
Noise 1
SART	PSNR	23.48(0.47)	23.30(0.64)	23.01(0.64)
SART	SSIM	0.424(0.020)	0.403(0.023)	0.391(0.022)
TV	PSNR	23.85(0.42)	23.75(0.41)	23.63(0.38)
TV	SSIM	0.582(0.030)	0.579(0.030)	0.578(0.028)
WF	PSNR	25.14(0.40)	25.05(0.42)	24.91(0.39)
WF	SSIM	0.659(0.027)	0.652(0.027)	0.649(0.026)
PD-net	PSNR	30.87(0.64)	30.44(0.51)	30.23(0.46)
PD-net	SSIM	0.776(0.036)	0.771(0.029)	0.773(0.028)
Noise 2
SART	PSNR	23.15(0.48)	22.91(0.53)	22.60(0.64)
SART	SSIM	0.413(0.020)	0.390(0.024)	0.378(0.024)
TV	PSNR	23.74(0.40)	23.50(0.36)	23.27(0.40)
TV	SSIM	0.580(0.030)	0.576(0.030)	0.573(0.028)
WF	PSNR	24.98(0.29)	24.84(0.41)	24.68(0.39)
WF	SSIM	0.657(0.027)	0.649(0.026)	0.646(0.026)
PD-net	PSNR	30.78(0.64)	30.35(0.51)	30.15(0.77)
PD-net	SSIM	0.774(0.037)	0.769(0.030)	0.771(0.029)
Noise 3
SART	PSNR	20.71(0.55)	20.26(0.72)	19.83(0.66)
SART	SSIM	0.334(0.026)	0.304(0.030)	0.291(0.029)
TV	PSNR	21.73(0.57)	21.43(0.48)	21.08(0.47)
TV	SSIM	0.568(0.027)	0.555(0.026)	0.545(0.026)
WF	PSNR	23.35(0.48)	23.05(0.51)	22.72(0.55)
WF	SSIM	0.636(0.0326)	0.616(0.027)	0.605(0.028)
PD-net	PSNR	29.97(0.66)	29.56(0.51)	29.36(0.47)
PD-net	SSIM	0.753(0.038)	0.746(0.032)	0.747(0.031)
Inference Time (s)		0.46(0.02)	0.21(0.008)	0.20(0.001)

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	W. van Aarle, W. J. Palenstijn, J. Cant, E. Janssens, F. Bleichrodt, A. Dabravolski, J. De Beenhouwer, K. J. Batenburg and J. Sijbers, Fast and flexible x-ray tomography using the astra toolbox, Optics Express, 22 (2016), 25129-25147.
[2]	W. van Aarle, W. J. Palenstijn, J. D. Beenhouwer, T. Altantzis, S. Bals, K. J. Batenburg and J. Sijbers, The astra toolbox: A platform for advanced algorithm development in electron tomography, Ultramicroscopy, 24 (2015), 35-47.
[3]	J. Adler and O. Oktem, Learned primal-dual reconstruction, IEEE Transactions on Medical Imaging, 37 (2018), 1322-1332.
[4]	K. J. Batenburg, W. J. Palenstijn, P. Balázs and J. Sijbers, Dynamic angle selection in binary tomography, Computer Vision and Image Understanding, 117 (2013), 306-318.
[5]	I. Bello, H. Pham, Q. V Le, M. Norouzi and S. Bengio, Neural combinatorial optimization with reinforcement learning, preprint, arXiv: 1611.09940, 2016.
[6]	S. Boyd and N. Parikh, et al., Distributed optimization and statistical learning via the alternating direction method of multipliers, Foundations and Trends in Machine Learning, 3 (2011), 1-122.
[7]	A. Buades, B. Coll and J. M. Morel, A review of image denoising algorithms, with a new one, Multiscale Model. Simul., 2 (2005), 490-530. doi: 10.1137/040616024.
[8]	J.-F. Cai, S. Osher and Z. Shen, Split bregman methods and frame based image restoration, Multiscale Modeling and Simulation, 8 (2009), 337-369. doi: 10.1137/090753504.
[9]	J.-F. Cai, H. Ji, Z. Shen and G. B. Ye, Data-driven tight frame construction and image denoising, Applied and Computational Harmonic Analysis, 37 (2014), 89-105. doi: 10.1016/j.acha.2013.10.001.
[10]	E. J. Candes, Y. C. Eldar, et al., Compressed Sensing With Coherent and Redundant Dictionaries, 2010.
[11]	E. J. Candes, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Transactions on Information Theory, 52 (2006), 489-509. doi: 10.1109/TIT.2005.862083.
[12]	A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with applications to imaging, Journal of Mathematical Imaging and Vision, 40 (2011), 120-145. doi: 10.1007/s10851-010-0251-1.
[13]	H. Chen, Y. Zhang, M. K. Kalra, F. Lin, Y. Chen, P. Liao, J. Zhou and G. Wang, Low-dose CT with a residual encoder-decoder convolutional neural network, IEEE Transactions on Medical Imaging, 36 (2017), 2524-2535.
[14]	K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Image denoising by sparse 3-d transform-domain collaborative filtering, IEEE Transactions on Image Processing, 16 (2007), 2080-2095. doi: 10.1109/TIP.2007.901238.
[15]	A. Dabravolski, K. J. Batenburg and J. Sijbers, Dynamic angle selection in x-ray computed tomography, Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms, 324 (2014), 17-24.
[16]	I. Daubechies, Ten Lectures on Wavelets, CBMS-NSF Regional Conference Series in Applied Mathematics, 61. Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1992. doi: 10.1137/1.9781611970104.
[17]	B. Dong, J. Li and Z. Shen, X-ray CT image reconstruction via wavelet frame based regularization and radon domain inpainting, Journal of Scientific Computing, 54 (2013), 333-349. doi: 10.1007/s10915-012-9579-6.
[18]	B. Dong and Z. Shen, et al., Mra based wavelet frames and applications, IAS Lecture Notes Series, Summer Program on "The Mathematics of Image Processing", Park City Mathematics Institute, 19 (2010), 9-158. doi: 10.1090/pcms/019/02.
[19]	D. Donoho, Compressed sensing, IEEE Transactions on Information Theory, 52 (2006), 1289-1306.
[20]	J. M. Ede, Adaptive partial scanning transmission electron microscopy with reinforcement learning, preprint, arXiv: 2004.02786.
[21]	M. Elad and M. Aharon, Image denoising via sparse and redundant representations over learned dictionaries, IEEE Transactions on Image processing, 15 (2006), 3736-3745. doi: 10.1109/TIP.2006.881969.
[22]	E. Esser and X. Zhang, et al., A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science, SIAM Journal on Imaging Sciences, 3 (2010), 1015-1046. doi: 10.1137/09076934X.
[23]	M. Gies, W. A. Kalender, H. Wolf and C. Suess, Dose reduction in CT by anatomically adapted tube current modulation. i. Simulation studies, Medical Physics, 26 (1999), 2235-2247.
[24]	G. D. Godaliyadda, M. A. Uchic, D. H. Ye, M. A. Groeber, G. T. Buzzard and C. A. Bouman, A supervised learning approach for dynamic sampling, S & T Imaging. International Society for Optics and Photonics, 2016.
[25]	G. M. D. P. Godaliyadda, D. H. Ye, M. D. Uchic, M. A. Groeber, G. T. Buzzard and C. A. Bouman, A framework for dynamic image sampling based on supervised learning (slads), IEEE Trans. Comput. Imaging, 4 (2018), 1-16. doi: 10.1109/TCI.2017.2777482.
[26]	T. Goldstein and S. Osher, The split bregman method for l1-regularized problems, SIAM Journal Imaging Sciences, 2 (2009), 323-343. doi: 10.1137/080725891.
[27]	T. Goldstein and S. Osher, The split bregman method for $l_1$-regularized problems, SIAM Journal on Imaging Sciences, 2 (2009), 323-343. doi: 10.1137/080725891.
[28]	R. Gordon, R. Benderab and G. T. Herman, Algebraic Reconstruction Techniques (ART) for Three-Dimensional Electron Microscopy and X-ray Photography, Journal of Theoretical Biology, 1970.
[29]	S. Gu, L. Zhang, W. Zuo and X. Feng, Weighted nuclear norm minimization with application to image denoising, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, (2014), 2862–2869.
[30]	A. Halimi, P. Ciuciu, A. Mccarthy, S. Mclaughlin and G. Buller, Fast adaptive scene sampling for single-photon 3d lidar images, IEEE CAMSAP 2019 - International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, 2019.
[31]	S. Ji, Y. Xue and L. Carin, Bayesian compressive sensing, IEEE Transactions on Signal Processing, 56 (2008), 2346-2356. doi: 10.1109/TSP.2007.914345.
[32]	K. H. Jin, M. T. McCann, E. Froustey and M. Unser, Deep convolutional neural network for inverse problems in imaging, IEEE Transactions on Image Processing, 26 (2017), 4509-4522. doi: 10.1109/TIP.2017.2713099.
[33]	W. A. Kalender, H. Wolf and C. Suess, Dose reduction in CT by anatomically adapted tube current modulation. ii. Phantom measurements, Medical Physics, 26 (1999), 2248-2253.
[34]	E. Kang, J. Min and J. C. Ye, A deep conversational neural network using directional wavelets for low-dose x-ray ct reconstruction, Medical Physics, 44 (2017), e360–e375. doi: 10.1002/mp.12344.
[35]	A. Katsevich, Theoretically exact filtered backprojection-type inversion algorithm for spiral CT, SIAM Journal on Applied Mathematics, 62 (2002), 2012-2026. doi: 10.1137/S0036139901387186.
[36]	D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv: 1412.6980, 2014.
[37]	W. Kool, H. V. Hoof and M. Welling, Attention, learn to solve routing problems!, preprint, arXiv: 1803.08475, 2018.
[38]	Y. Li, Deep Reinforcement Learning: An overview, arXiv: 1701.07274, 2017.
[39]	L. Ly and Y.-H. R. Tsai, Autonomous exploration, reconstruction, and surveillance of 3d environments aided by deep learning, arXiv: 1809.06025, 2018.
[40]	S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, Inc., San Diego, CA, 1998.
[41]	M. T. McCann, K. H. Jin and M. Unser, Convolutional neural networks for inverse problems in imaging: A review, IEEE Signal Processing Magazine, 34 (2017), 85-95.
[42]	C. McCollough, Tu-fg-207a-04: Overview of the low dose ct grand challenge, Medical Physics, 43 (2016), 3760-3760. doi: 10.1118/1.4957556.
[43]	A. Mittal, A. Dhawan, S. Manchanda, S. Medya, S. Ranu and A. Singh, Learning heuristics over large graphs via deep reinforcement learning, preprint, arXiv: 1903.03332, 2019.
[44]	V. Mnih, K. Kavukcuoglu and D. Silver, Human-level control through deep reinforcement learning, Nature, 518 (2015).
[45]	V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland and G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature, 518 (2015), 529-533.
[46]	K. A. Mohan, S. V. Venkatakrishnan, E. B. Gulsoy, J. W. Gibbs, X. Xiao, M. D. Graef, P. W. Voorhees and C. A. Bouman, Timbir: A method for time-space reconstruction from interlaced views, IEEE Transactions on Computational Imaging, 1 (2015), 96-111. doi: 10.1109/TCI.2015.2431913.
[47]	E. Monier, N. Brun, T. Oberlin, X. Li, M. Tenc and N. Dobigeon, Fast reconstruction of atomic-scale stem-eels images from sparse sampling, Ultramicroscopy, 2020.
[48]	K. Mueller, Selection of optimal views for computed tomography reconstruction, Patent WO, Jan, 28 (2011).
[49]	K. Mueller, R. Yagel and J. J. Wheller, Anti-aliased three-dimensional cone-beam reconstruction of low-contrast objects with algebraic methods, IEEE Transactions On Medical Imaging, 6 (1999), 519-537.
[50]	R. Ohbuchi and M. Aono, Quasi-Monte Carlo Rendering With Adaptive Sampling, 1996.
[51]	S. Osher, Z. Shi and W. Zhu, Low dimensional manifold model for image processing, SIAM Journal on Imaging Sciences, 10 (2017), 1669-1690. doi: 10.1137/16M1058686.
[52]	J. Park, J. Jung, A. P. Gupta, J. Soh, C. Jeong, J. Ahn, S. Cho, K. -H. Yoon, D. Kim, M. Mativenga, et al., Multi-beam x-ray source based on carbon nanotube emitters for tomosynthesis system, in Medical Imaging 2020: Physics of Medical Imaging, International Society for Optics and Photonics, 11312 (2020), 113122.
[53]	G. Placidi, M. Alecci and A. Sotgiu, Theory of adaptive acquisition method for image reconstruction from projections and application to epr image, Journal of Magnetic Resonance, (1995), 50–57.
[54]	S. RL, Fast calculation of the exact radiological path for a three-dimensional CT array, Medical Physics, 2 (1985), 252-5.
[55]	A. Ron and Z. Shen, Affine systems in $ L_{2}(\mathbb{R}^{d})$: The analysis of the analysis operator, Journal of Functional Analysis, 148 (1997), 408-447. doi: 10.1006/jfan.1996.3079.
[56]	L. I. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physical D: Nonlinear Phenomena, 60 (1992), 259-268. doi: 10.1016/0167-2789(92)90242-F.
[57]	J. Schulman and F. Wolski, Proximal policy optimization algorithms, arXiv: 1707.06347v2, 2017.
[58]	M. W. Seeger and H. Nickisch, Compressed sensing and bayesian experimental design, in Proceedings of the 25th International Conference on Machine Learning, (2008), 912–919.
[59]	B. Settles, Active Iearning Literature Survey, Technical report, University of Wisconsin-Madison Department of Computer Sciences, 2009.
[60]	C. Shen, Y. Gonzalez, L. Chen, S. B. Jiang and X. Jia, Intelligent parameter tuning in optimization-based iterative CT reconstruction via deep reinforcement learning, IEEE Transactions on Medical Imaging, 37 (2018), 1430-1439.
[61]	E. Y. Sidky and X. Pan, Image reconstruction in circular cone-beam computed tomography by constrained, total-variation minimization, Physics in medicine and biology, 4777, 2008.
[62]	D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra and M. Riedmiller, Deterministic policy gradient algorithms, In International Conference on Machine Learning, (2014), 387–395.
[63]	R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, 2018.
[64]	R. S. Sutton, D. A. McAllester, S. P. Singh and Y. Mansour, Policy gradient methods for reinforcement learning with function approximation, In Advances in Neural Information Processing Systems, (2000), 1057–1063.
[65]	C. Tai and E. Weinan, Multiscale adaptive representation of signals: I. The basic framework, The Journal of Machine Learning Research, 17 (2016), 4875-4912.
[66]	G. Wang, A perspective on deep imaging, IEEE Access, 4 (2016), 8914-8924.
[67]	Z. Wang and G. R. Arce, Variable density compressed image sampling, Image Processing, IEEE Transactions, 19 (2010), 264-270. doi: 10.1109/TIP.2009.2032889.
[68]	G. Wang, M. Kalra and C. G.Orton, Machine learning will transform radiology significantly within the next 5 years, Medical Physics, 44 (2017), 2041-2044.
[69]	G. Wang, J. Chu Ye, K. Mueller and J. A Fessler, Image reconstruction is a new frontier of machine learning, IEEE Transactions on Medical Imaging, 37 (2018), 1289-1296.
[70]	G. Wang and H. Yu, A scheme for multisource interior tomography, Medical physics, 36 (2009), 3575-3581.
[71]	C. J. Watkins and P. Dayan, Q-learning, Machine learning, 8 (1992), 279-292.
[72]	Q. Yang, P. Yan, Y. Zhang, H. Yu, Y. Shi, X. Mou, M. K. Kalra, Y. Zhang, L. Sun and G. Wang, Low-dose CT image denoising using a generative adversarial network with wasserstein distance and perceptual loss, IEEE Transactions on Medical Imaging, 37 (2018) 1348–1357.
[73]	L. Yu, M. Shiung, D. Jondal and C. H. McCollough, Development and validation of a practical lower-dose-simulation tool for optimizing computed tomography scan protocols, Journal of Computer Assisted Tomography, 36 (20124), 477-487.
[74]	J. Zhang, G. Yang, Y. Lee, Y. Cheng, B. Gao, Q. Qiu, J. Lu and O. Zhou, A multi-beam x-ray imaging system based on carbon nanotube field emitters, in Medical Imaging 2006: Physics of Medical Imaging, International Society for Optics and Photonics, 6142 (2006), 614204.
[75]	S. Zhang, Z. Song, G. D. P. Godaliyadda, D. H. Ye, A. U. Chowdhury, A. Sengupta, G. T. Buzzard, C. A. Bouman and G. J. Simpson, Dynamic sparse sampling for confocal raman microscopy, Analytical Chemistry, 90 (2018), 4461-4469.
[76]	Y. Zhang, G. M. D. Godaliyadda, N. Ferrier, E. B. Gulsoy, C. A. Bouman and C. Phatak, Slads-Net: Supervised Learning Approach for Dynamic Sampling Using Deep Neural Networks, Electronic Imaging, Computational Imaging XVI, 2018.
[77]	H. -M. Zhang and B. Dong, A review on deep learning in medical image reconstruction, Journal of the Operations Research Society of China, 8 (2020) 311–340. doi: 10.1007/s40305-019-00287-4.
[78]	Z. Zhang, X. Liang, X. Dong, Y. Xie and G. Cao, A sparse-view CT reconstruction method based on combination of DenseNet and deconvolution, IEEE Transactions on Medical Imaging, 37 (2018), 1407-1417.
[79]	M. Zhu and T. Chan, An efficient primal-dual hybrid gradient algorithm for total variation image restoration, UCLA CAM Report, 34 (2008), 8-34.