\`x^2+y_1+z_12^34\`
Advanced Search
Article Contents
Article Contents

Image fusion network for dual-modal restoration

  • *Corresponding author: Xiaoqun Zhang

    *Corresponding author: Xiaoqun Zhang
Abstract Full Text(HTML) Figure(6) / Table(2) Related Papers Cited by
  • In recent years multi-modal data processing methods have gained considerable research interest as technological advancements in imaging, computing, and data storage have made the collection of redundant, multi-modal data more commonplace. In this work we present an image restoration method tailored for scenarios where pre-existing, high-quality images from different modalities or contrasts are available in addition to the target image. Our method is based on a novel network architecture which combines the benefits of traditional multi-scale signal representation, such as wavelets, with more recent concepts from data fusion methods. Results from numerical simulations in which T1-weighted MRI images are used to restore noisy and undersampled T2-weighted images demonstrate that the proposed network successfully utilizes information from high-quality reference images to improve the restoration quality of the target image beyond that of existing popular methods.

    Mathematics Subject Classification: Primary: 68U10.

    Citation:

    \begin{equation} \\ \end{equation}
  • 加载中
  • Figure 1.  Proposed network architecture: An observed image $ \mathbf{f} $ and a reference image $ \mathbf{f}_\mathrm{ref} $ are passed as inputs into an encoding network consisting of a cascade of decomposition stages. The outputs of these stages are then passed as the input for a symmetric decoding stage which combines the information from $ \mathbf{f} $ and $ \mathbf{f}_\mathrm{ref} $. The structures labeled by $ \Psi_i $, $ \Psi_i^{'} $, and $ \Psi_i^{†} $ are convolutional layers representing decomposition and composition using learned kernels, while the wavelet transform layers represent decomposition and composition by a set of fixed kernels. The fusion of coefficients indicated by $ \bigoplus $ is multiplied by separated $ w $ and $ w^{'} $, and then performed with a multi-layer convolutional neural network (CNN) H, both of which need to be learned

    Figure 2.  Network inputs for three different denoising scenarios. In each column, the left panel is the target image to be denoised ($ \mathbf{f} $) and the right column is the corresponding reference image ($ \mathbf{f}_\mathrm{ref} $). (a) Denoising of a noisy T2W image with a high-SNR T1W reference image. (b) Denoising a noisy T2W image with an all-zero reference image. (c) "Denoising'' an all-zero image with a high-SNR T1W reference image. The scenarios in (b) and (c) were evaluated to observe how the network incorporates information from the reference image into the denoising process

    Figure 3.  T2W image denoising results for the scenarios depicted in Fig. 2. Top row: ground truth and denoising results of BM3D, DnCNN, and the proposed network, respectively. Bottom row: magnified results from the region indicated by the red box in the ground truth image in the top row. The PSNR and MSE over the entire image are given below each result

    Figure 4.  T2W image with high noise level denoising results. Top row: noisy T2W image and ground truth. Bottom row: denoising results of BM3D, DnCNN, and the proposed network, respectively

    Figure 5.  Reconstruction of sparsely sampled Fourier data. Top row: ground truth versions of several T2W images. Rows 2-4 show the corresponding reconstruction results for a sampling pattern with uniform undersampling in outer $ k $-space. Rows 5-7 show the corresponding reconstruction results for a sampling pattern with random under sampling. The MSE is given on each result

    Figure 6.  Reconstruction of random sampled Fourier data under different noise levels

    Table 1.  Specific parameters setting for e×periments shown in Fig. 5

    Encoding Decoding
    1th
    layer
    comb
    layer
    2th
    layer
    comb
    layer
    3th
    layer
    comb
    layer
    4th
    layer
    comb
    layer
    5th
    layer
    comb
    layer
    1th
    layer
    2th
    layer
    3th
    layer
    4th
    layer
    5th
    layer
    obj $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array} $ $\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered} $ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered} $ $\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered} $ \begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered}$ $ $ \begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered} $ $ \begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered} $ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered} $ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$ $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$
    ref $ \begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $ \begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$ $ \begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$
     | Show Table
    DownLoad: CSV

    Table 2.  The average MSE of several methods under different noise levels

    Noise level E-WTV DnCNN Proposed
    0.01 0.000956 0.000703 0.000657
    0.05 0.001600 0.000989 0.000926
    0.1 0.002700 0.004997 0.001533
     | Show Table
    DownLoad: CSV
  • [1] E. J. CandesJ. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2004), 489-509.  doi: 10.1109/TIT.2005.862083.
    [2] K. DabovA. FoiV. Katkovnik and K. Egiazarian, Image denoising by sparse 3D transform-domain collaborative filtering, IEEE Trans. Image Process, 16 (2007), 2080-2095.  doi: 10.1109/TIP.2007.901238.
    [3] L. DengM. Feng and X. Tai, The fusion of panchromatic and multispectral remote sensing images via tensor-based sparse modeling and hyper-Laplacian prior, Inform Fusion, 52 (2019), 76-89. 
    [4] S. Fujieda, K. Takayama and T. Hachisuka, Wavelet Convolutional Neural Networks, CoRR, 2018.
    [5] H. Huang, R. He, Z. Sun and T. Tan, Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution, 2017 IEEE International Conference on Computer Vision (ICCV), (2017), 1698–1706. doi: 10.1109/ICCV.2017.187.
    [6] C. M. Hyun, H. P. Kim, S. M. Lee, S. C. Lee and J. K. Seo, Deep learning for undersampled MRI reconstruction, Phys. Med. Biol., 63 (2018).
    [7] T. B. KaiM. Uecker and J. Frahm, Undersampled radial MRI with multiple coils. iterative image reconstruction using a total variation constraint, Magn. Reson. Med., 57 (2007), 1086-1098. 
    [8] D. P. Kingma and J. L. Ba, ADAM: A method for stochastic optimization, In Int Conf on Learning Representations, 2015.
    [9] H. LiB. S. Manjunath and S. K. Mitra, Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57 (1995), 235-245.  doi: 10.1006/gmip.1995.1022.
    [10] L. Li, B. Wang and G. Wang, A self-adaptive mask-enhanced dual-dictionary learning method for MRI-CT image reconstruction, In Nuclear Science Symposium & Medical Imaging Conf, 2016.
    [11] P. Liu, H. Zhang, K. Zhang, L. Lin and W. Zuo, Multi-level wavelet-cnn for image restoration, In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018. doi: 10.1109/CVPRW.2018.00121.
    [12] G. Pajares and J. M. D. L. Cruz, A wavelet-based image fusion tutorial, Pattern Recogn, 37 (2004), 1855-1872. 
    [13] Y. QuanH. Ji and Z. Shen, Data-driven multi-scale non-local wavelet frame construction and image recovery, J. Sci. Comput., 63 (2015), 307-329.  doi: 10.1007/s10915-014-9893-2.
    [14] R. Singh, M. Vatsa and A. Noore, Multimodal medical image fusion using redundant discrete wavelet transform, InInt. Conf. on Advances in Pattern Recognition, (2009), 232–235. doi: 10.1109/ICAPR.2009.97.
    [15] D. W. Townsend, Multimodality imaging of structure and function, Phys. Med. Biol., 53 (2008), 1-39.  doi: 10.1088/0031-9155/53/4/R01.
    [16] L. XiangY. ChenW. ChangY. ZhanW. LinQ. Wang and D. Shen, Ultra-fast t2-weighted mr reconstruction using complementary t1-weighted information, International Conference on Medical Image Computing and Computer-Assisted Intervention, 11070 (2018), 215-223.  doi: 10.1007/978-3-030-00928-1_25.
    [17] J. C. YeY. Han and E. Cha, Deep convolutional framelets: A general deep learning for inverse problems, SIAM J. Imaginig Sci., 11 (2017), 991-1048.  doi: 10.1137/17M1141771.
    [18] R. YinT. GaoY. M. Lu and I. Daubechies, A tale of two bases: Local-nonlocal regularization on image patches with convolution framelets, SIAM J. Imaginig Sci., 10 (2017), 711-750.  doi: 10.1137/16M1091447.
    [19] K. ZhangW. ZuoY. ChenD. Meng and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Transactions on Image Processing, 26 (2017), 3142-3155.  doi: 10.1109/TIP.2017.2662206.
    [20] Y. Zhang and X. Zhang, PET-MRI joint reconstruction with common edge weighted total variation regularization, Inverse Probl., 34 (2018), 76-89.  doi: 10.1088/1361-6420/aabce9.
  • 加载中
Open Access Under a Creative Commons license

Figures(6)

Tables(2)

SHARE

Article Metrics

HTML views(1414) PDF downloads(206) Cited by(0)

Access History

Catalog

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return