American Institute of Mathematical Sciences

December  2021, 15(6): 1409-1419. doi: 10.3934/ipi.2021067

Image fusion network for dual-modal restoration

 1 School of Statistics and Mathematics, Shanghai Lixin University of Accounting and Finance, Shanghai, 201209, China 2 School of Biomedical Engineering, Shanghai Jiao Tong University, China 3 Department of Electrical and Computer Engineering, Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, USA 4 School of Mathematical Sciences, MOE-LSC, Shanghai Jiao Tong University, China, Institute of Natural Sciences, Shanghai Jiao Tong University, China

*Corresponding author: Xiaoqun Zhang

Received  November 2021 Revised  August 2021 Published  December 2021 Early access  October 2021

In recent years multi-modal data processing methods have gained considerable research interest as technological advancements in imaging, computing, and data storage have made the collection of redundant, multi-modal data more commonplace. In this work we present an image restoration method tailored for scenarios where pre-existing, high-quality images from different modalities or contrasts are available in addition to the target image. Our method is based on a novel network architecture which combines the benefits of traditional multi-scale signal representation, such as wavelets, with more recent concepts from data fusion methods. Results from numerical simulations in which T1-weighted MRI images are used to restore noisy and undersampled T2-weighted images demonstrate that the proposed network successfully utilizes information from high-quality reference images to improve the restoration quality of the target image beyond that of existing popular methods.

Citation: Ying Zhang, Xuhua Ren, Bryan Alexander Clifford, Qian Wang, Xiaoqun Zhang. Image fusion network for dual-modal restoration. Inverse Problems & Imaging, 2021, 15 (6) : 1409-1419. doi: 10.3934/ipi.2021067
References:

show all references

References:
Proposed network architecture: An observed image $\mathbf{f}$ and a reference image $\mathbf{f}_\mathrm{ref}$ are passed as inputs into an encoding network consisting of a cascade of decomposition stages. The outputs of these stages are then passed as the input for a symmetric decoding stage which combines the information from $\mathbf{f}$ and $\mathbf{f}_\mathrm{ref}$. The structures labeled by $\Psi_i$, $\Psi_i^{'}$, and $\Psi_i^{†}$ are convolutional layers representing decomposition and composition using learned kernels, while the wavelet transform layers represent decomposition and composition by a set of fixed kernels. The fusion of coefficients indicated by $\bigoplus$ is multiplied by separated $w$ and $w^{'}$, and then performed with a multi-layer convolutional neural network (CNN) H, both of which need to be learned
Network inputs for three different denoising scenarios. In each column, the left panel is the target image to be denoised ($\mathbf{f}$) and the right column is the corresponding reference image ($\mathbf{f}_\mathrm{ref}$). (a) Denoising of a noisy T2W image with a high-SNR T1W reference image. (b) Denoising a noisy T2W image with an all-zero reference image. (c) "Denoising'' an all-zero image with a high-SNR T1W reference image. The scenarios in (b) and (c) were evaluated to observe how the network incorporates information from the reference image into the denoising process
. Top row: ground truth and denoising results of BM3D, DnCNN, and the proposed network, respectively. Bottom row: magnified results from the region indicated by the red box in the ground truth image in the top row. The PSNR and MSE over the entire image are given below each result">Figure 3.  T2W image denoising results for the scenarios depicted in Fig. 2. Top row: ground truth and denoising results of BM3D, DnCNN, and the proposed network, respectively. Bottom row: magnified results from the region indicated by the red box in the ground truth image in the top row. The PSNR and MSE over the entire image are given below each result
T2W image with high noise level denoising results. Top row: noisy T2W image and ground truth. Bottom row: denoising results of BM3D, DnCNN, and the proposed network, respectively
Reconstruction of sparsely sampled Fourier data. Top row: ground truth versions of several T2W images. Rows 2-4 show the corresponding reconstruction results for a sampling pattern with uniform undersampling in outer $k$-space. Rows 5-7 show the corresponding reconstruction results for a sampling pattern with random under sampling. The MSE is given on each result
Reconstruction of random sampled Fourier data under different noise levels
Specific parameters setting for e×periments shown in Fig. 5
 Encoding Decoding 1thlayer comb layer 2thlayer comb layer 3thlayer comb layer 4thlayer comb layer 5thlayer comb layer 1thlayer 2thlayer 3thlayer 4thlayer 5thlayer obj $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered}$ \begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered} $\begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$ $\begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered}$ $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$ $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$ ref $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$
 Encoding Decoding 1thlayer comb layer 2thlayer comb layer 3thlayer comb layer 4thlayer comb layer 5thlayer comb layer 1thlayer 2thlayer 3thlayer 4thlayer 5thlayer obj $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $\begin{gathered} 25 \times 2 \\ +3 \times 3 \\ \times 25, \\ 25 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $\begin{gathered} 50 \times 2 \\ +3 \times 3 \\ \times 50, \\ 50 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 100 \times 2 \\ +3 \times 3 \\ \times 100, \\ 100 \times 3 \end{gathered}$ \begin{gathered} 5 \times 5 \times \\ 100, \\ 150 \end{gathered} $\begin{gathered} 150 \times 2 \\ +3 \times 3 \\ \times 150, \\ 150 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$ $\begin{gathered} 200 \times 2 \\ +3 \times 3 \\ \times 200, \\ 200 \times 3 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 200, \\ 150 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100,50 \end{gathered}$ $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &50,25 \end{array}$ $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &25,1 \end{array}$ ref $\begin{array}{*{20}{c}} &5 \times 5 \times \\ &1,25 \end{array}$ $\begin{gathered} 5 \times 5 \times \\ 25,50 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 50,100 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 100 , \\ 150 \end{gathered}$ $\begin{gathered} 5 \times 5 \times \\ 150, \\ 200 \end{gathered}$
The average MSE of several methods under different noise levels
 Noise level E-WTV DnCNN Proposed 0.01 0.000956 0.000703 0.000657 0.05 0.001600 0.000989 0.000926 0.1 0.002700 0.004997 0.001533
 Noise level E-WTV DnCNN Proposed 0.01 0.000956 0.000703 0.000657 0.05 0.001600 0.000989 0.000926 0.1 0.002700 0.004997 0.001533
 [1] Nicolas Lermé, François Malgouyres, Dominique Hamoir, Emmanuelle Thouin. Bayesian image restoration for mosaic active imaging. Inverse Problems & Imaging, 2014, 8 (3) : 733-760. doi: 10.3934/ipi.2014.8.733 [2] Jiangchuan Fan, Xinyu Guo, Jianjun Du, Weiliang Wen, Xianju Lu, Brahmani Louiza. Analysis of the clustering fusion algorithm for multi-band color image. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1233-1249. doi: 10.3934/dcdss.2019085 [3] Yunmei Chen, Jiangli Shi, Murali Rao, Jin-Seop Lee. Deformable multi-modal image registration by maximizing Rényi's statistical dependence measure. Inverse Problems & Imaging, 2015, 9 (1) : 79-103. doi: 10.3934/ipi.2015.9.79 [4] Mehdi Bastani, Davod Khojasteh Salkuyeh. On the GSOR iteration method for image restoration. Numerical Algebra, Control & Optimization, 2021, 11 (1) : 27-43. doi: 10.3934/naco.2020013 [5] Eugene Kashdan, Svetlana Bunimovich-Mendrazitsky. Multi-scale model of bladder cancer development. Conference Publications, 2011, 2011 (Special) : 803-812. doi: 10.3934/proc.2011.2011.803 [6] Amir Averbuch, Pekka Neittaanmäki, Valery Zheludev. Periodic spline-based frames for image restoration. Inverse Problems & Imaging, 2015, 9 (3) : 661-707. doi: 10.3934/ipi.2015.9.661 [7] Ruiqiang He, Xiangchu Feng, Xiaolong Zhu, Hua Huang, Bingzhe Wei. RWRM: Residual Wasserstein regularization model for image restoration. Inverse Problems & Imaging, 2021, 15 (6) : 1307-1332. doi: 10.3934/ipi.2020069 [8] Ke Chen, Yiqiu Dong, Michael Hintermüller. A nonlinear multigrid solver with line Gauss-Seidel-semismooth-Newton smoother for the Fenchel pre-dual in total variation based image restoration. Inverse Problems & Imaging, 2011, 5 (2) : 323-339. doi: 10.3934/ipi.2011.5.323 [9] Yong Zheng Ong, Haizhao Yang. Generative imaging and image processing via generative encoder. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021060 [10] Thomas Blanc, Mihaï Bostan. Multi-scale analysis for highly anisotropic parabolic problems. Discrete & Continuous Dynamical Systems - B, 2020, 25 (1) : 335-399. doi: 10.3934/dcdsb.2019186 [11] Michel Potier-Ferry, Foudil Mohri, Fan Xu, Noureddine Damil, Bouazza Braikat, Khadija Mhada, Heng Hu, Qun Huang, Saeid Nezamabadi. Cellular instabilities analyzed by multi-scale Fourier series: A review. Discrete & Continuous Dynamical Systems - S, 2016, 9 (2) : 585-597. doi: 10.3934/dcdss.2016013 [12] Xiaoman Liu, Jijun Liu. Image restoration from noisy incomplete frequency data by alternative iteration scheme. Inverse Problems & Imaging, 2020, 14 (4) : 583-606. doi: 10.3934/ipi.2020027 [13] Alina Toma, Bruno Sixou, Françoise Peyrin. Iterative choice of the optimal regularization parameter in TV image restoration. Inverse Problems & Imaging, 2015, 9 (4) : 1171-1191. doi: 10.3934/ipi.2015.9.1171 [14] Jing Xu, Xue-Cheng Tai, Li-Lian Wang. A two-level domain decomposition method for image restoration. Inverse Problems & Imaging, 2010, 4 (3) : 523-545. doi: 10.3934/ipi.2010.4.523 [15] Bartomeu Coll, Joan Duran, Catalina Sbert. Half-linear regularization for nonconvex image restoration models. Inverse Problems & Imaging, 2015, 9 (2) : 337-370. doi: 10.3934/ipi.2015.9.337 [16] Eitan Tadmor, Prashant Athavale. Multiscale image representation using novel integro-differential equations. Inverse Problems & Imaging, 2009, 3 (4) : 693-710. doi: 10.3934/ipi.2009.3.693 [17] Thierry Cazenave, Flávio Dickstein, Fred B. Weissler. Multi-scale multi-profile global solutions of parabolic equations in $\mathbb{R}^N$. Discrete & Continuous Dynamical Systems - S, 2012, 5 (3) : 449-472. doi: 10.3934/dcdss.2012.5.449 [18] Emiliano Cristiani, Elisa Iacomini. An interface-free multi-scale multi-order model for traffic flow. Discrete & Continuous Dynamical Systems - B, 2019, 24 (11) : 6189-6207. doi: 10.3934/dcdsb.2019135 [19] Rongliang Chen, Jizu Huang, Xiao-Chuan Cai. A parallel domain decomposition algorithm for large scale image denoising. Inverse Problems & Imaging, 2019, 13 (6) : 1259-1282. doi: 10.3934/ipi.2019055 [20] Jianping Zhang, Ke Chen, Bo Yu, Derek A. Gould. A local information based variational model for selective image segmentation. Inverse Problems & Imaging, 2014, 8 (1) : 293-320. doi: 10.3934/ipi.2014.8.293

2020 Impact Factor: 1.639