Image fusion network for dual-modal restoration

Ying Zhang; Xuhua Ren; Bryan Alexander Clifford; Qian Wang; Xiaoqun Zhang

doi:10.3934/ipi.2021067

Article Contents

2021, Volume 15, Issue 6: 1409-1419. Doi: 10.3934/ipi.2021067 open access

This issue Previous Article Image retinex based on the nonconvex TV-type regularization Next Article An adaptive total variational despeckling model based on gray level indicator frame

Image fusion network for dual-modal restoration

1.
School of Statistics and Mathematics, Shanghai Lixin University of Accounting and Finance, Shanghai, 201209, China
2.
School of Biomedical Engineering, Shanghai Jiao Tong University, China
3.
Department of Electrical and Computer Engineering, Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, USA
4.
School of Mathematical Sciences, MOE-LSC, Shanghai Jiao Tong University, China, Institute of Natural Sciences, Shanghai Jiao Tong University, China

^*Corresponding author: Xiaoqun Zhang
^*Corresponding author: Xiaoqun Zhang

Received: November 2021

Revised: August 2021

Early access: October 2021

Published: December 2021

Abstract / Introduction Full Text(HTML) Figure(6) / Table(2) Related Papers Cited by

Abstract

In recent years multi-modal data processing methods have gained considerable research interest as technological advancements in imaging, computing, and data storage have made the collection of redundant, multi-modal data more commonplace. In this work we present an image restoration method tailored for scenarios where pre-existing, high-quality images from different modalities or contrasts are available in addition to the target image. Our method is based on a novel network architecture which combines the benefits of traditional multi-scale signal representation, such as wavelets, with more recent concepts from data fusion methods. Results from numerical simulations in which T1-weighted MRI images are used to restore noisy and undersampled T2-weighted images demonstrate that the proposed network successfully utilizes information from high-quality reference images to improve the restoration quality of the target image beyond that of existing popular methods.

Keywords:

Mathematics Subject Classification: Primary: 68U10.

Citation:

FullText(HTML)

Figure 1. Proposed network architecture: An observed image $ \mathbf{f} $ and a reference image $ \mathbf{f}_\mathrm{ref} $ are passed as inputs into an encoding network consisting of a cascade of decomposition stages. The outputs of these stages are then passed as the input for a symmetric decoding stage which combines the information from $ \mathbf{f} $ and $ \mathbf{f}_\mathrm{ref} $. The structures labeled by $ \Psi_i $, $ \Psi_i^{'} $, and $ \Psi_i^{†} $ are convolutional layers representing decomposition and composition using learned kernels, while the wavelet transform layers represent decomposition and composition by a set of fixed kernels. The fusion of coefficients indicated by $ \bigoplus $ is multiplied by separated $ w $ and $ w^{'} $, and then performed with a multi-layer convolutional neural network (CNN) H, both of which need to be learned

Download: Full-size image PowerPoint slide

Figure 2. Network inputs for three different denoising scenarios. In each column, the left panel is the target image to be denoised ($ \mathbf{f} $) and the right column is the corresponding reference image ($ \mathbf{f}_\mathrm{ref} $). (a) Denoising of a noisy T2W image with a high-SNR T1W reference image. (b) Denoising a noisy T2W image with an all-zero reference image. (c) "Denoising'' an all-zero image with a high-SNR T1W reference image. The scenarios in (b) and (c) were evaluated to observe how the network incorporates information from the reference image into the denoising process

Download: Full-size image PowerPoint slide

Figure 3. T2W image denoising results for the scenarios depicted in Fig. 2. Top row: ground truth and denoising results of BM3D, DnCNN, and the proposed network, respectively. Bottom row: magnified results from the region indicated by the red box in the ground truth image in the top row. The PSNR and MSE over the entire image are given below each result

Download: Full-size image PowerPoint slide

Figure 4. T2W image with high noise level denoising results. Top row: noisy T2W image and ground truth. Bottom row: denoising results of BM3D, DnCNN, and the proposed network, respectively

Download: Full-size image PowerPoint slide

Figure 5. Reconstruction of sparsely sampled Fourier data. Top row: ground truth versions of several T2W images. Rows 2-4 show the corresponding reconstruction results for a sampling pattern with uniform undersampling in outer $ k $-space. Rows 5-7 show the corresponding reconstruction results for a sampling pattern with random under sampling. The MSE is given on each result

Download: Full-size image PowerPoint slide

Figure 6. Reconstruction of random sampled Fourier data under different noise levels

Download: Full-size image PowerPoint slide

Table 1. Specific parameters setting for e×periments shown in Fig. 5

	Encoding										Decoding
	1^th layer	comb layer	2^th layer	comb layer	3^th layer	comb layer	4^th layer	comb layer	5^th layer	comb layer	1^th layer	2^th layer	3^th layer	4^th layer	5^th layer
obj	$\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array} $	$\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered} $	$\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered} $	$\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered} $	$ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$	$\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered} $	\begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered}$ $	$ \begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$	$\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered} $	$ \begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$	$\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered} $	$ \begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$	$\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered} $	$ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$	$ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$
ref	$ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$		$ \begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$		$ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$		$ \begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$		$ \begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$

| Show Table

DownLoad: CSV

Table 2. The average MSE of several methods under different noise levels

Noise level	E-WTV	DnCNN	Proposed
0.01	0.000956	0.000703	0.000657
0.05	0.001600	0.000989	0.000926
0.1	0.002700	0.004997	0.001533

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	E. J. Candes, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509. doi: 10.1109/TIT.2005.862083.
[2]	K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095. doi: 10.1109/TIP.2007.901238.
[3]	L. Deng, M. Feng and X. Tai, The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89.
[4]	S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018.
[5]	H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706. doi: 10.1109/ICCV.2017.187.
[6]	C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018).
[7]	T. B. Kai, M. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098.
[8]	D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015.
[9]	H. Li, B. S. Manjunath and S. K. Mitra, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245. doi: 10.1006/gmip.1995.1022.
[10]	L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016.
[11]	P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018. doi: 10.1109/CVPRW.2018.00121.
[12]	G. Pajares and J. M. D. L. Cruz, A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872.
[13]	Y. Quan, H. Ji and Z. Shen, Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329. doi: 10.1007/s10915-014-9893-2.
[14]	R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235. doi: 10.1109/ICAPR.2009.97.
[15]	D. W. Townsend, Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39. doi: 10.1088/0031-9155/53/4/R01.
[16]	L. Xiang, Y. Chen, W. Chang, Y. Zhan, W. Lin, Q. Wang and D. Shen, Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223. doi: 10.1007/978-3-030-00928-1_25.
[17]	J. C. Ye, Y. Han and E. Cha, Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048. doi: 10.1137/17M1141771.
[18]	R. Yin, T. Gao, Y. M. Lu and I. Daubechies, A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750. doi: 10.1137/16M1091447.
[19]	K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155. doi: 10.1109/TIP.2017.2662206.
[20]	Y. Zhang and X. Zhang, PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89. doi: 10.1088/1361-6420/aabce9.