December  2021, 15(6): 1409-1419. doi: 10.3934/ipi.2021067

Image fusion network for dual-modal restoration

1. 

School of Statistics and Mathematics, Shanghai Lixin University of Accounting and Finance, Shanghai, 201209, China

2. 

School of Biomedical Engineering, Shanghai Jiao Tong University, China

3. 

Department of Electrical and Computer Engineering, Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, USA

4. 

School of Mathematical Sciences, MOE-LSC, Shanghai Jiao Tong University, China, Institute of Natural Sciences, Shanghai Jiao Tong University, China

*Corresponding author: Xiaoqun Zhang

Received  November 2021 Revised  August 2021 Published  December 2021 Early access  October 2021

In recent years multi-modal data processing methods have gained considerable research interest as technological advancements in imaging, computing, and data storage have made the collection of redundant, multi-modal data more commonplace. In this work we present an image restoration method tailored for scenarios where pre-existing, high-quality images from different modalities or contrasts are available in addition to the target image. Our method is based on a novel network architecture which combines the benefits of traditional multi-scale signal representation, such as wavelets, with more recent concepts from data fusion methods. Results from numerical simulations in which T1-weighted MRI images are used to restore noisy and undersampled T2-weighted images demonstrate that the proposed network successfully utilizes information from high-quality reference images to improve the restoration quality of the target image beyond that of existing popular methods.

Citation: Ying Zhang, Xuhua Ren, Bryan Alexander Clifford, Qian Wang, Xiaoqun Zhang. Image fusion network for dual-modal restoration. Inverse Problems & Imaging, 2021, 15 (6) : 1409-1419. doi: 10.3934/ipi.2021067
References:
[1]

E. J. CandesJ. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509.  doi: 10.1109/TIT.2005.862083.  Google Scholar

[2]

K. DabovA. FoiV. Katkovnik and K. Egiazarian, Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095.  doi: 10.1109/TIP.2007.901238.  Google Scholar

[3]

L. DengM. Feng and X. Tai, The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89.   Google Scholar

[4]

S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018. Google Scholar

[5]

H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706. doi: 10.1109/ICCV.2017.187.  Google Scholar

[6]

C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018). Google Scholar

[7]

T. B. KaiM. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098.   Google Scholar

[8]

D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015. Google Scholar

[9]

H. LiB. S. Manjunath and S. K. Mitra, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245.  doi: 10.1006/gmip.1995.1022.  Google Scholar

[10]

L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016. Google Scholar

[11]

P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018. doi: 10.1109/CVPRW.2018.00121.  Google Scholar

[12]

G. Pajares and J. M. D. L. Cruz, A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872.   Google Scholar

[13]

Y. QuanH. Ji and Z. Shen, Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329.  doi: 10.1007/s10915-014-9893-2.  Google Scholar

[14]

R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235. doi: 10.1109/ICAPR.2009.97.  Google Scholar

[15]

D. W. Townsend, Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39.  doi: 10.1088/0031-9155/53/4/R01.  Google Scholar

[16]

L. XiangY. ChenW. ChangY. ZhanW. LinQ. Wang and D. Shen, Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223.  doi: 10.1007/978-3-030-00928-1_25.  Google Scholar

[17]

J. C. YeY. Han and E. Cha, Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048.  doi: 10.1137/17M1141771.  Google Scholar

[18]

R. YinT. GaoY. M. Lu and I. Daubechies, A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750.  doi: 10.1137/16M1091447.  Google Scholar

[19]

K. ZhangW. ZuoY. ChenD. Meng and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155.  doi: 10.1109/TIP.2017.2662206.  Google Scholar

[20]

Y. Zhang and X. Zhang, PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89.  doi: 10.1088/1361-6420/aabce9.  Google Scholar

show all references

References:
[1]

E. J. CandesJ. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509.  doi: 10.1109/TIT.2005.862083.  Google Scholar

[2]

K. DabovA. FoiV. Katkovnik and K. Egiazarian, Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095.  doi: 10.1109/TIP.2007.901238.  Google Scholar

[3]

L. DengM. Feng and X. Tai, The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89.   Google Scholar

[4]

S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018. Google Scholar

[5]

H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706. doi: 10.1109/ICCV.2017.187.  Google Scholar

[6]

C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018). Google Scholar

[7]

T. B. KaiM. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098.   Google Scholar

[8]

D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015. Google Scholar

[9]

H. LiB. S. Manjunath and S. K. Mitra, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245.  doi: 10.1006/gmip.1995.1022.  Google Scholar

[10]

L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016. Google Scholar

[11]

P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018. doi: 10.1109/CVPRW.2018.00121.  Google Scholar

[12]

G. Pajares and J. M. D. L. Cruz, A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872.   Google Scholar

[13]

Y. QuanH. Ji and Z. Shen, Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329.  doi: 10.1007/s10915-014-9893-2.  Google Scholar

[14]

R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235. doi: 10.1109/ICAPR.2009.97.  Google Scholar

[15]

D. W. Townsend, Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39.  doi: 10.1088/0031-9155/53/4/R01.  Google Scholar

[16]

L. XiangY. ChenW. ChangY. ZhanW. LinQ. Wang and D. Shen, Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223.  doi: 10.1007/978-3-030-00928-1_25.  Google Scholar

[17]

J. C. YeY. Han and E. Cha, Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048.  doi: 10.1137/17M1141771.  Google Scholar

[18]

R. YinT. GaoY. M. Lu and I. Daubechies, A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750.  doi: 10.1137/16M1091447.  Google Scholar

[19]

K. ZhangW. ZuoY. ChenD. Meng and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155.  doi: 10.1109/TIP.2017.2662206.  Google Scholar

[20]

Y. Zhang and X. Zhang, PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89.  doi: 10.1088/1361-6420/aabce9.  Google Scholar

Figure 1.  Proposed network architecture: An observed image $ \mathbf{f} $ and a reference image $ \mathbf{f}_\mathrm{ref} $ are passed as inputs into an encoding network consisting of a cascade of decomposition stages. The outputs of these stages are then passed as the input for a symmetric decoding stage which combines the information from $ \mathbf{f} $ and $ \mathbf{f}_\mathrm{ref} $. The structures labeled by $ \Psi_i $, $ \Psi_i^{'} $, and $ \Psi_i^{†} $ are convolutional layers representing decomposition and composition using learned kernels, while the wavelet transform layers represent decomposition and composition by a set of fixed kernels. The fusion of coefficients indicated by $ \bigoplus $ is multiplied by separated $ w $ and $ w^{'} $, and then performed with a multi-layer convolutional neural network (CNN) H, both of which need to be learned
Figure 2.  Network inputs for three different denoising scenarios. In each column, the left panel is the target image to be denoised ($ \mathbf{f} $) and the right column is the corresponding reference image ($ \mathbf{f}_\mathrm{ref} $). (a) Denoising of a noisy T2W image with a high-SNR T1W reference image. (b) Denoising a noisy T2W image with an all-zero reference image. (c) "Denoising'' an all-zero image with a high-SNR T1W reference image. The scenarios in (b) and (c) were evaluated to observe how the network incorporates information from the reference image into the denoising process
Fig. 2. Top row: ground truth and denoising results of BM3D, DnCNN, and the proposed network, respectively. Bottom row: magnified results from the region indicated by the red box in the ground truth image in the top row. The PSNR and MSE over the entire image are given below each result">Figure 3.  T2W image denoising results for the scenarios depicted in Fig. 2. Top row: ground truth and denoising results of BM3D, DnCNN, and the proposed network, respectively. Bottom row: magnified results from the region indicated by the red box in the ground truth image in the top row. The PSNR and MSE over the entire image are given below each result
Figure 4.  T2W image with high noise level denoising results. Top row: noisy T2W image and ground truth. Bottom row: denoising results of BM3D, DnCNN, and the proposed network, respectively
Figure 5.  Reconstruction of sparsely sampled Fourier data. Top row: ground truth versions of several T2W images. Rows 2-4 show the corresponding reconstruction results for a sampling pattern with uniform undersampling in outer $ k $-space. Rows 5-7 show the corresponding reconstruction results for a sampling pattern with random under sampling. The MSE is given on each result
Figure 6.  Reconstruction of random sampled Fourier data under different noise levels
Table 1.  Specific parameters setting for e×periments shown in Fig. 5
Encoding Decoding
1th
layer
comb
layer
2th
layer
comb
layer
3th
layer
comb
layer
4th
layer
comb
layer
5th
layer
comb
layer
1th
layer
2th
layer
3th
layer
4th
layer
5th
layer
obj $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array} $ $\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered} $ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered} $ $\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered} $ \begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered}$ $ $ \begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered} $ $ \begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered} $ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$
ref $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $ \begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$
Encoding Decoding
1th
layer
comb
layer
2th
layer
comb
layer
3th
layer
comb
layer
4th
layer
comb
layer
5th
layer
comb
layer
1th
layer
2th
layer
3th
layer
4th
layer
5th
layer
obj $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array} $ $\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered} $ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered} $ $\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered} $ \begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered}$ $ $ \begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered} $ $ \begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered} $ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$
ref $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $ \begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$
Table 2.  The average MSE of several methods under different noise levels
Noise level E-WTV DnCNN Proposed
0.01 0.000956 0.000703 0.000657
0.05 0.001600 0.000989 0.000926
0.1 0.002700 0.004997 0.001533
Noise level E-WTV DnCNN Proposed
0.01 0.000956 0.000703 0.000657
0.05 0.001600 0.000989 0.000926
0.1 0.002700 0.004997 0.001533
[1]

Nicolas Lermé, François Malgouyres, Dominique Hamoir, Emmanuelle Thouin. Bayesian image restoration for mosaic active imaging. Inverse Problems & Imaging, 2014, 8 (3) : 733-760. doi: 10.3934/ipi.2014.8.733

[2]

Jiangchuan Fan, Xinyu Guo, Jianjun Du, Weiliang Wen, Xianju Lu, Brahmani Louiza. Analysis of the clustering fusion algorithm for multi-band color image. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1233-1249. doi: 10.3934/dcdss.2019085

[3]

Yunmei Chen, Jiangli Shi, Murali Rao, Jin-Seop Lee. Deformable multi-modal image registration by maximizing Rényi's statistical dependence measure. Inverse Problems & Imaging, 2015, 9 (1) : 79-103. doi: 10.3934/ipi.2015.9.79

[4]

Mehdi Bastani, Davod Khojasteh Salkuyeh. On the GSOR iteration method for image restoration. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 27-43. doi: 10.3934/naco.2020013

[5]

Eugene Kashdan, Svetlana Bunimovich-Mendrazitsky. Multi-scale model of bladder cancer development. Conference Publications, 2011, 2011 (Special) : 803-812. doi: 10.3934/proc.2011.2011.803

[6]

Amir Averbuch, Pekka Neittaanmäki, Valery Zheludev. Periodic spline-based frames for image restoration. Inverse Problems & Imaging, 2015, 9 (3) : 661-707. doi: 10.3934/ipi.2015.9.661

[7]

Ruiqiang He, Xiangchu Feng, Xiaolong Zhu, Hua Huang, Bingzhe Wei. RWRM: Residual Wasserstein regularization model for image restoration. Inverse Problems & Imaging, 2021, 15 (6) : 1307-1332. doi: 10.3934/ipi.2020069

[8]

Ke Chen, Yiqiu Dong, Michael Hintermüller. A nonlinear multigrid solver with line Gauss-Seidel-semismooth-Newton smoother for the Fenchel pre-dual in total variation based image restoration. Inverse Problems & Imaging, 2011, 5 (2) : 323-339. doi: 10.3934/ipi.2011.5.323

[9]

Yong Zheng Ong, Haizhao Yang. Generative imaging and image processing via generative encoder. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021060

[10]

Thomas Blanc, Mihaï Bostan. Multi-scale analysis for highly anisotropic parabolic problems. Discrete & Continuous Dynamical Systems - B, 2020, 25 (1) : 335-399. doi: 10.3934/dcdsb.2019186

[11]

Michel Potier-Ferry, Foudil Mohri, Fan Xu, Noureddine Damil, Bouazza Braikat, Khadija Mhada, Heng Hu, Qun Huang, Saeid Nezamabadi. Cellular instabilities analyzed by multi-scale Fourier series: A review. Discrete & Continuous Dynamical Systems - S, 2016, 9 (2) : 585-597. doi: 10.3934/dcdss.2016013

[12]

Xiaoman Liu, Jijun Liu. Image restoration from noisy incomplete frequency data by alternative iteration scheme. Inverse Problems & Imaging, 2020, 14 (4) : 583-606. doi: 10.3934/ipi.2020027

[13]

Alina Toma, Bruno Sixou, Françoise Peyrin. Iterative choice of the optimal regularization parameter in TV image restoration. Inverse Problems & Imaging, 2015, 9 (4) : 1171-1191. doi: 10.3934/ipi.2015.9.1171

[14]

Jing Xu, Xue-Cheng Tai, Li-Lian Wang. A two-level domain decomposition method for image restoration. Inverse Problems & Imaging, 2010, 4 (3) : 523-545. doi: 10.3934/ipi.2010.4.523

[15]

Bartomeu Coll, Joan Duran, Catalina Sbert. Half-linear regularization for nonconvex image restoration models. Inverse Problems & Imaging, 2015, 9 (2) : 337-370. doi: 10.3934/ipi.2015.9.337

[16]

Eitan Tadmor, Prashant Athavale. Multiscale image representation using novel integro-differential equations. Inverse Problems & Imaging, 2009, 3 (4) : 693-710. doi: 10.3934/ipi.2009.3.693

[17]

Thierry Cazenave, Flávio Dickstein, Fred B. Weissler. Multi-scale multi-profile global solutions of parabolic equations in $\mathbb{R}^N $. Discrete & Continuous Dynamical Systems - S, 2012, 5 (3) : 449-472. doi: 10.3934/dcdss.2012.5.449

[18]

Emiliano Cristiani, Elisa Iacomini. An interface-free multi-scale multi-order model for traffic flow. Discrete & Continuous Dynamical Systems - B, 2019, 24 (11) : 6189-6207. doi: 10.3934/dcdsb.2019135

[19]

Rongliang Chen, Jizu Huang, Xiao-Chuan Cai. A parallel domain decomposition algorithm for large scale image denoising. Inverse Problems & Imaging, 2019, 13 (6) : 1259-1282. doi: 10.3934/ipi.2019055

[20]

Jianping Zhang, Ke Chen, Bo Yu, Derek A. Gould. A local information based variational model for selective image segmentation. Inverse Problems & Imaging, 2014, 8 (1) : 293-320. doi: 10.3934/ipi.2014.8.293

2020 Impact Factor: 1.639

Metrics

  • PDF downloads (53)
  • HTML views (66)
  • Cited by (0)

[Back to Top]