How to best combine demosaicing and denoising?

Yu Guo; Qiyu Jin; Jean-Michel Morel; Gabriele Facciolo

doi:10.3934/ipi.2023044

Article Contents

2024, Volume 18, Issue 3: 571-599. Doi: 10.3934/ipi.2023044

This issue Previous Article Visibility, invisibility and unique recovery of inverse electromagnetic problems with conical singularities Next Article A Wasserstein distance and total variation regularized model for image reconstruction problems

How to best combine demosaicing and denoising?

1.
School of Mathematical Science, Inner Mongolia University, Hohhot 010020, China
2.
Department of Mathematics, City University of Hong Kong, Kowloon Tong, Hong Kong
3.
Centre Borelli, ENS Paris-Saclay, CNRS, 4, avenue des Sciences 91190 Gif-sur-Yvette, France

^*Corresponding author: Qiyu Jin
^*Corresponding author: Qiyu Jin

Received: November 2022

Revised: September 2023

Early access: October 16, 2023

Published online: October 16, 2023

Abstract / Introduction Full Text(HTML) Figure(14) / Table(13) Related Papers Cited by

Abstract

Image demosaicing and denoising play a critical role in the raw imaging pipeline. These processes have often been treated as independent, without considering their interactions. Indeed, most classic denoising methods handle noisy RGB images, not raw images. Conversely, most demosaicing methods address the demosaicing of noise free images. The real problem is to jointly denoise and demosaic noisy raw images. But the question of how to proceed is still not clarified. In this paper, we carry out extensive experiments and a mathematical analysis to tackle this problem by low complexity algorithms. Indeed, both problems have only been addressed jointly by end-to-end heavy-weight convolutional neural networks (CNNs), which are currently incompatible with low-power portable imaging devices and remain by nature domain (or device) dependent. Our study leads us to conclude that, with moderate noise, demosaicing should be applied first, followed by denoising. This requires a simple adaptation of classic denoising algorithms to demosaiced noise, which we justify and specify. Although our main conclusion is "demosaic first, then denoise, " we also discover that for high noise, there is a moderate PSNR gain by a more complex strategy: partial CFA denoising followed by demosaicing and by a second denoising on the RGB image. These surprising results are obtained by a black-box optimization of the pipeline, which could be applied to any other pipeline. We validate our results on simulated and real noisy CFA images obtained from several benchmarks.

Keywords:

Mathematics Subject Classification: Primary: 68U10; Secondary: 62H35.

Citation:

Full Text(HTML)

Figure 1. Image details at $ \sigma = 20 $. The lower row is the reconstructed image, and the upper row is the difference between the reconstructed image and ground truth. $ DN $: cfaBM3D or CBM3D denoising; $ DM $: RCNN demosaicing. $ 1.5 DN $ means that if the noise level is $ \sigma $, the input noise level parameter of denoising method $ DN $ is $ \sigma_2 = 1.5{{\sigma}} $

Download: Full-size image PowerPoint slide

Figure 2. The framework used for denoising before demosaicing using an RGB denoiser. The Bayer CFA image is split in two half resolution RGB images, each one with a different green. Both RGB images are denoised independently. Then the pixels of both results are recombined into a denoised Bayer CFA image. The last step consists in applying a demosaicing algorithm

Download: Full-size image PowerPoint slide

Figure 3. Generic raw image processing pipeline. This pipeline structure allows for an arbitrary order between $ DN $ and $ DM $ and sets free their parameters. We use the CMA-ES algorithm to optimize the parameter $ \alpha $, $ \beta $, $ \sigma_1 $, $ \sigma_2 $ in the pipeline

Download: Full-size image PowerPoint slide

Figure 4. A visual representation of the process in Figure 3, where the noise level is $ \sigma = 60 $. The parameter are $ \alpha = 0.90 $, $ \beta = 0.99 $, $ \sigma_1 = 34.50 $, $ \sigma_2 = 54.42 $. Since $ \beta $ is always close to 1 in the pipeline, the visual difference between Color Denoising and the $ \beta $ linear combination is not significant

Download: Full-size image PowerPoint slide

Figure 5. Evolution of the result of iterating CMA-ES when optimizing the parameters $ \alpha, \beta, \sigma_1, \sigma_2 $ of the processing pipeline

Download: Full-size image PowerPoint slide

Figure 6. First row: (a) Ground truth Imax 3, (b) its noisy version, (c) added white noise ($ {{\sigma}} = 20 $), (d) demosaiced version of (b) by RCNN, (e) the demosaiced noise, namely the difference (d)-(a). Second and third rows: $ 50\times 50 $ extracts from the first row

Download: Full-size image PowerPoint slide

Figure 7. AWGN image and demosaicing noise with standard deviation $ \sigma = 20 $ for respectively HA, MLRI, RCNN. Last row: the color spaces (in standard (R, G, B) Cartesian coordinates) of each noise, presented in their projection with maximal area. As expected, the AWG color space is isotropic, while the color space after demosaicing is elongated in the luminance direction $ {\bf{Y}} $ and squeezed in the others. This amounts to an increased noise standard deviation for $ {\bf{Y}} $ after demosaicing, and less noise in the chromatic directions. See table Table 4 for quantitative results

Download: Full-size image PowerPoint slide

Figure 8. Demosaicing and denoising results on an image from the Kodak dataset with $ \sigma = 20 $. We compare the two schemes of $ DN\& DM $, cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of $ DM\&1.5 DN $, MLRI+CBM3D and RCNN+CBM3D, and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method

Download: Full-size image PowerPoint slide

Figure 9. Demosaicing and denoising results on an image from the Kodak dataset with σ = 10. We compare the two schemes of DN & DM, cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of DM & 1.5DN, MLRI+CBM3D and RCNN+CBM3D, and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method

Download: Full-size image PowerPoint slide

Figure 10. Demosaicing and denoising results on an image from the Imax dataset with $ \sigma = 20 $. We compare the two schemes of $ DN\& DM $, cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of $ DM\&1.5 DN $, MLRI+CBM3D and RCNN+CBM3D, and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method

Download: Full-size image PowerPoint slide

Figure 11. Demosaicing and denoising results on an image from the Imax dataset with $ \sigma = 60 $. We compare the two schemes of $ DN\& DM $, cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of $ DM\&1.5 DN $, MLRI+CBM3D and RCNN+CBM3D and the MLRI+CBM3D schemes optimized by the CMA-ES algorithm. As a reference we also include the result of JCNN, a joint CNN method

Download: Full-size image PowerPoint slide

Figure 12. Demosaicing and denoising results on an image from the SIDD dataset. We compare the two schemes of $ DN\& DM $, cfaBM3D+MLRI and cfaBM3D+RCNN, the two schemes of $ DM\&1.5 DN $, MLRI+CBM3D and RCNN+CBM3D. As a reference we also include the result of JCNN, a joint CNN method

Download: Full-size image PowerPoint slide

Figure 13. The flowchart of raw image denoising under $ DM\&1.5 DN $ scheme. The dashed VST/IVST blocks are active in just one of the pipeline variants

Download: Full-size image PowerPoint slide

Figure 14. Denoising results on an image from the DND dataset. We compare the $ DM\&1.5 DN $ scheme (MLRI+CBM3D and RCNN+CBM3D), TNRD [10], EPLL [80], WNNM [27] and BM3D [16] (results as reported on the benchmark website)

Download: Full-size image PowerPoint slide

Table 1. Advantages and drawbacks of the three types of pipelines

	$ DN\& DM $	$ DM\& DN $	Joint $ DM DN $
Advantages	The noise is maintained AWGN	Richer details	Better imaging quality
Drawbacks	Detail loss and checkerboard artifacts	Spatial and chromaticity-related structural noise	High computational complexity and generalization concerns

| Show Table

DownLoad: CSV

Table 2. The optimization result of CMA-ES for the pipeline $ DN_{1}\& DM\& DN_{2} $ (see Eq. (3)), where $ \sigma, \sigma_1, \sigma_2 \in [0,255] $ and $ \alpha, \beta \in [0, 1] $. In this experiment $ DM $ is always MLRI and $ DN $ is CBM3D or cfaBM3D depending on the input data

$\sigma$	Method	$\alpha$	$\beta$	$\sigma_1$	$\sigma_2$	CPSNR Imax	CPSNR Kodak
5	$DN\&DM$	1.00	0.00	5.00	0	34.20	35.08
	$DM\&DN$	0.00	1.00	0	5.00	34.18	35.03
	$DM\&1.5DN$	0.00	1.00	0	7.50	34.64	35.77
	CMA-ES	0.02	0.90	0	7.83	34.66	35.78
10	$DN\&DM$	1.00	0.00	10.00	0	31.68	32.15
	$DM\&DN$	0.00	1.00	0	10.00	31.55	31.62
	$DM\&1.5DN$	0.00	1.00	0	15.00	32.35	32.99
	CMA-ES	0.51	0.92	6.81	12.98	32.43	33.02
20	$DN\&DM$	1.00	0.00	20.00	0	28.48	28.91
	$DM\&DN$	0.00	1.00	0	20.00	28.07	27.75
	$DM\&1.5DN$	0.00	1.00	0	30.00	29.30	29.85
	CMA-ES	0.52	0.95	10.58	30.63	29.36	29.91
40	$DN\&DM$	1.00	0.00	40.00	0	24.90	25.84
	$DM\&DN$	0.00	1.00	0	40.00	24.16	24.05
	$DM\&1.5DN$	0.00	1.00	0	60.00	25.46	26.53
	CMA-ES	0.82	0.98	23.46	41.79	25.74	26.72
50	$DN\&DM$	1.00	0.00	50.00	0	23.62	24.83
	$DM\&DN$	0.00	1.00	0	50.00	22.87	23.00
	$DM\&1.5DN$	0.00	1.00	0	75.00	24.01	25.33
	CMA-ES	0.72	1.00	30.55	49.75	24.36	25.61
60	$DN\&DM$	1.00	0.00	60.00	0	22.49	23.90
	$DM\&DN$	0.00	1.00	0	60.00	21.83	22.24
	$DM\&1.5DN$	0.00	1.00	0	90.00	22.76	24.26
	CMA-ES	0.90	0.99	34.50	54.42	23.16	24.60

| Show Table

DownLoad: CSV

Table 3. RMSE between ground truth and demosaicked image for different demosaicking algorithms in presence of noise of standard deviation $ \sigma $

$ {{\sigma}} $	HA	GBTF	RI	MLRI	RCNN
$ 1 $	5.04	5.10	4.17	4.06	3.21
$ 3 $	5.70	5.79	4.97	4.88	4.17
$ 5 $	6.78	6.87	6.12	6.10	5.59
$ 10 $	10.18	10.27	9.53	9.74	9.65
$ 15 $	13.93	14.01	13.15	13.64	13.87
$ 20 $	17.75	17.83	16.77	17.56	18.04
$ 30 $	25.36	25.42	23.94	25.30	26.21
$ 40 $	32.67	32.76	30.77	32.64	33.98
$ 50 $	39.58	39.71	37.25	39.55	41.21
$ 60 $	46.14	46.35	43.43	46.11	47.95

| Show Table

DownLoad: CSV

Table 4. Noise intensity. Variance and covariance of $ ({\bf{R}}, {\bf{G}}, {\bf{B}}) $ and $ ({\bf{Y}}, {\bf{C}}_{1}, {\bf{C}}_{2}) $ between pixels $ (i, j) $ and $ (i+s, j+t) $, $ s, t = 0, 1, 2 $ first for AWGN (a) with standard deviation $ \sigma = 20 $, then for its demosaiced versions by HA (b), RI (c), MLRI (d) and RCNN (e)

| Show Table

DownLoad: CSV

Table 5. Correlation between pixels. The corresponding correlations of $({\bf{R}}, {\bf{G}}, {\bf{B}}) $ and $ ({\bf{Y}}, {\bf{C}}_{1}, {\bf{C}}_{2})$ between pixels (i, j) and (i+s, j+t), s, t = 0, 1, 2 first for AWGN (a) with standard deviation $\sigma = 20 $, then for its demosaiced versions by HA (b), RI (c), MLRI (d) and RCNN (e)

| Show Table

DownLoad: CSV

Table 6. Correlation between channels. Covariance (each first row) and corresponding correlation (each second row) of the three color channels (R, G, and B) of the demosaicing noise when the initial CFA white noise satisfies $ \sigma = 20 $. See Figure 7 for an illustration

| Show Table

DownLoad: CSV

Table 7. The results of different combinations of denoising and demosaicing methods for the Imax image dataset. The best result for each row is red, the second best result is brown and the third best result is blue

| Show Table

DownLoad: CSV

Table 8. The results of different combinations of denoising and demosaicing methods for the Kodak image dataset. The best result for each row is red, the second best result is brown and the third best result is blue

| Show Table

DownLoad: CSV

Table 9. Average CPSNR results on the SIDD dataset. Note that for each camera, images with different noise levels are being considered. The noise range is $ \sigma \in [3.28, 38.12] $. The proposed $ DM\&1.5 DN $ schemes outperforms the $ DN\& DM $ ones. The best result is in red and the second best one is in brown

| Show Table

DownLoad: CSV

Table 10. Validation of the $ DM\&1.5 DN $ scheme on the SIDD dataset. Note that for each camera, images with different noise levels are being considered. The noise range is $ \sigma \in [0.48, 22.59] $ without VST and $ \sigma\in [0.38, 13.00] $ with VST. The best result is in red and the second best one is in brown

| Show Table

DownLoad: CSV

Table 11. Comparison results of the $ DM\&1.5 DN $ scheme on the SIDD and DND benchmarks (results as reported on the corresponding websites). * indicates the use of the variance stabilizing transform (VST). The best result is in red and the second best one is in brown

| Show Table

DownLoad: CSV

Table 12. Time consumption. The average running time (CPU) of the three strategies in processing 10 images on a PC with an Intel Core i7-9750H 2.60GHz CPU and 16GB memory. Note that we do not use the deep learning methods and only compared the traditional methods

$ DN\& DM $			$ DM\&1.5 DN $			CMA-ES
cfaBM3D+HA	cfaBM3D+RI	cfaBM3D+MLRI	HA+CBM3D	RI+CBM3D	MLRI+CBM3D	cfaBM3D+MLRI+CBM3D
7.41 s	7.64 s	7.85 s	16.16 s	16.66 s	16.72 s	23.93 s

| Show Table

DownLoad: CSV

Table 13. Generalizability of CMA-ES optimal parameters to different noise levels. Evaluation of noise levels with $ \sigma = 50 $ proximity (selected as 46 to 54) using two generalization schemes

$ \sigma $	$ DN\& DM $	$ DM\&1.5 DN $	CMA-ES image transformation	CMA-ES $ \sigma $ transformation
46	24.10	24.60	24.83	24.90
47	23.98	24.46	24.74	24.78
48	23.85	24.32	24.63	24.64
49	23.74	24.19	24.52	24.52
51	23.50	23.91	24.26	24.26
52	23.35	23.77	24.13	24.12
53	23.24	23.64	24.00	24.00
54	23.14	23.52	23.90	23.89

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	A. Abdelhamed, S. Lin and M. S. Brown, A high-quality denoising dataset for smartphone cameras, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., (2018), 1692-1700. doi: 10.1109/CVPR.2018.00182.
[2]	H. Akiyama, M. Tanaka and M. Okutomi, Pseudo four-channel image denoising for noisy cfa raw data, in Proc. IEEE Int. Conf. Image Process., (2015), 4778-4782. doi: 10.1109/ICIP.2015.7351714.
[3]	D. Alleysson, S. Susstrunk and J. Herault, Linear demosaicing inspired by the human visual system, IEEE Trans. Image Process., 14 (2005), 439-449. doi: 10.1109/TIP.2004.841200.
[4]	F. J. Anscombe, The transformation of poisson, binomial and negative-binomial data, Biometrika, 35 (1948), 246-254. doi: 10.1093/biomet/35.3-4.246.
[5]	B. E. Bayer, Color Imaging Array, 1976, US Patent 3,971,065.
[6]	A. Buades, B. Coll and J. M. Morel, A review of image denoising algorithms, with a new one, Multiscale Model. Simul., 4 (2005), 490-530. doi: 10.1137/040616024.
[7]	A. Buades, B. Coll, J.-M. Morel and C. Sbert, Self-similarity driven demosaicking, Image Processing On Line, 1 (2011), 51-56. doi: 10.5201/ipol.2011.bcms-ssdd.
[8]	P. Chatterjee, N. Joshi, S. B. Kang and Y. Matsushita, Noise suppression in low-light images through joint denoising and demosaicing, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2011), 321-328. doi: 10.1109/CVPR.2011.5995371.
[9]	G. Chen, F. Zhu and P. A. Heng, An efficient statistical method for image noise level estimation, in Proc. IEEE Int. Conf. Comput. Vis., (2015), 477-485. doi: 10.1109/ICCV.2015.62.
[10]	Y. Chen, W. Yu and T. Pock, On learning optimized reaction diffusion processes for effective image restoration, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2015), 5261-5269. doi: 10.1109/CVPR.2015.7299163.
[11]	M. R. Chowdhury, J. Zhang, J. Qin and Y. Lou, Poisson image denoising based on fractional-order total variation, Inverse Probl. Imaging, 14 (2020), 77-96. doi: 10.3934/ipi.2019064.
[12]	L. Condat, A simple, fast and efficient approach to denoisaicking: Joint demosaicking and denoising, in Proc. IEEE Int. Conf. Image Process., (2010), 905-908. doi: 10.1109/ICIP.2010.5652196.
[13]	L. Condat, A generic proximal algorithm for convex optimization-application to total variation minimization, IEEE Signal Process. Lett., 21 (2014), 985-989. doi: 10.1109/LSP.2014.2322123.
[14]	L. Condat and S. Mosaddegh, Joint demosaicking and denoising by total variation minimization, in Proc. IEEE Int. Conf. Image Process., (2012), 2781-2784. doi: 10.1109/ICIP.2012.6467476.
[15]	K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space, in Proc. IEEE Int. Conf. Image Process., 1 (2007), I - 313-I - 316. doi: 10.1109/ICIP.2007.4378954.
[16]	K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Image denoising by sparse 3-d transform-domain collaborative filtering, IEEE Trans. Image Process., 16 (2007), 2080-2095. doi: 10.1109/TIP.2007.901238.
[17]	A. Danielyan, M. Vehvilainen, A. Foi, V. Katkovnik and K. Egiazarian, Cross-color bm3d filtering of noisy raw data, in Proc. Int. Workshop Local Non-Local Approx. Image Process., (2009), 125-129. doi: 10.1109/LNLA.2009.5278395.
[18]	A. Davy and T. Ehret, Gpu acceleration of nl-means, bm3d and vbm3d, J. Real-Time Image Process., 18 (2021), 57-74. doi: 10.1007/s11554-020-00945-4.
[19]	W. Dong, M. Yuan, X. Li and G. Shi, Joint demosaicing and denoising with perceptual optimization on a generative adversarial network, arXiv: 1802.04723.
[20]	E. Dubois, Frequency-domain methods for demosaicking of bayer-sampled color images, IEEE Signal Process. Lett., 12 (2005), 847-850. doi: 10.1109/LSP.2005.859503.
[21]	T. Ehret, A. Davy, P. Arias and G. Facciolo, Joint demosaicking and denoising by fine-tuning of bursts of raw images, in Proc. IEEE/CVF Int. Conf. Comput. Vis., (2019), 8867-8876. doi: 10.1109/ICCV.2019.00896.
[22]	T. Ehret and G. Facciolo, A study of two CNN demosaicking algorithms, Image Processing On Line, 9 (2019), 220-230. doi: 10.5201/ipol.2019.274.
[23]	O. A. Elgendy, A. Gnanasambandam, S. H. Chan and J. Ma, Low-light demosaicking and denoising for small pixels using learned frequency selection, IEEE Trans. Comput. Imaging, 7 (2021), 137-150. doi: 10.1109/TCI.2021.3052694.
[24]	F. Fang, J. Li, Y. Yuan, T. Zeng and G. Zhang, Multilevel edge features guided network for image denoising, IEEE Trans. Neural Netw. Learn. Syst., 32 (2021), 3956-3970. doi: 10.1109/TNNLS.2020.3016321.
[25]	R. Franzen, Kodak lossless true color image suite, source: http://r0k.us/graphics/kodak/, 4.
[26]	M. Gharbi, G. Chaurasia, S. Paris and F. Durand, Deep joint demosaicking and denoising, ACM Trans. Graph., 35 (2016), 1-12. doi: 10.1145/2980179.2982399.
[27]	S. Gu, L. Zhang, W. Zuo and X. Feng, Weighted nuclear norm minimization with application to image denoising, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2014), 2862-2869. doi: 10.1109/CVPR.2014.366.
[28]	J. Guan, R. Lai, Y. Lu, Y. Li, H. Li, L. Feng, Y. Yang and L. Gu, Memory-efficient deformable convolution based joint denoising and demosaicing for uhd images, IEEE Trans. Circuits Syst. Video Technol., 32 (2022), 7346-7358. doi: 10.1109/TCSVT.2022.3182990.
[29]	J. Guo, Y. Guo, Q. Jin, M. Kwok-Po Ng and S. Wang, Gaussian patch mixture model guided low-rank covariance matrix minimization for image denoising, SIAM J. Imaging Sci., 15 (2022), 1601-1622. doi: 10.1137/21M1454262.
[30]	S. Guo, Z. Liang and L. Zhang, Joint denoising and demosaicking with green channel prior for real-world burst images, IEEE Trans. Image Process., 30 (2021), 6930-6942. doi: 10.1109/TIP.2021.3100312.
[31]	S. Guo, Z. Yan, K. Zhang, W. Zuo and L. Zhang, Toward convolutional blind denoising of real photographs, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., (2019), 1712-1722. doi: 10.1109/CVPR.2019.00181.
[32]	Y. Guo, A. Davy, G. Facciolo, J.-M. Morel and Q. Jin, Fast, nonlocal and neural: A lightweight high quality solution to image denoising, IEEE Signal Process. Lett., 28 (2021), 1515-1519. doi: 10.1109/LSP.2021.3099963.
[33]	Y. Guo, Q. Jin, J.-M. Morel, T. Zeng and G. Facciolo, Joint demosaicking and denoising benefits from a two-stage training strategy, J. Comput. Appl. Math., 434 (2023), Paper No. 115330, 16 pp. doi: 10.1016/j.cam.2023.115330.
[34]	J. F. Hamilton Jr and J. E. Adams Jr, Adaptive color plan interpolation in single sensor color electronic camera, 1997, US Patent 5,629,734.
[35]	N. Hansen and A. Ostermeier, Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation, in Proc. IEEE Int. Conf. Evol. Comput., (1996), 312-317. doi: 10.1109/ICEC.1996.542381.
[36]	M. K. Heris, Implementation of covariance matrix adaptation evolution strategy (cma-es) in matlab, https://yarpiz.com/235/ypea108-cma-es, (2015).
[37]	M. Hintermüller and M. Rincon-Camacho, An adaptive finite element method in $ l^2$-tv-based image denoising, Inverse Probl. Imaging, 8 (2014), 685-711. doi: 10.3934/ipi.2014.8.685.
[38]	K. Hirakawa and T. W. Parks, Joint demosaicing and denoising, IEEE Trans. Image Process., 15 (2006), 2146-2157. doi: 10.1109/ICIP.2005.1530390.
[39]	H. Hu, J. Froment, B. Wang and X. Fan, Spatial-frequency domain nonlocal total variation for image denoising, Inverse Probl. Imaging, 14 (2020), 1157-1184. doi: 10.3934/ipi.2020059.
[40]	Q. Jin, G. Facciolo and J.-M. Morel, A review of an old dilemma: Demosaicking first, or denoising first?, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops, (2020), 2169-2179. doi: 10.1109/CVPRW50498.2020.00265.
[41]	Q. Jin, I. Grama, C. Kervrann and Q. Liu, Nonlocal means and optimal weights for noise removal, SIAM J. Imaging Sci., 10 (2017), 1878-1920. doi: 10.1137/16M1080781.
[42]	Q. Jin, I. Grama and Q. Liu, Convergence theorems for the non-local means filter, Inverse Probl. Imaging, 12 (2018), 853-881. doi: 10.3934/ipi.2018036.
[43]	Q. Jin, Y. Guo, J.-M. Morel and G. Facciolo, A mathematical analysis and implementation of residual interpolation demosaicking algorithms, Image Processing On Line, 11 (2021), 234-283. doi: 10.5201/ipol.2021.358.
[44]	Y. Jin, J. Jost and G. Wang, A new nonlocal variational setting for image processing, Inverse Probl. Imaging, 9 (2015), 415-430. doi: 10.3934/ipi.2015.9.415.
[45]	O. Kalevo and H. Rantanen, Noise reduction techniques for bayer-matrix images, Proc. Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications Ⅲ, 4669 (2002), 348-359.
[46]	D. Khashabi, S. Nowozin, J. Jancsary and A. W. Fitzgibbon, Joint demosaicing and denoising via learned nonparametric random fields, IEEE Trans. Image Process., 23 (2014), 4968-4981. doi: 10.1109/TIP.2014.2359774.
[47]	D. Kiku, Y. Monno, M. Tanaka and M. Okutomi, Residual interpolation for color image demosaicking, in Proc. IEEE Int. Conf. Image Process., (2013), 2304-2308. doi: 10.1109/ICIP.2013.6738475.
[48]	D. Kiku, Y. Monno, M. Tanaka and M. Okutomi, Minimized-laplacian residual interpolation for color image demosaicking, in Proc. Digital Photography X, 9023 (2014), 90230L. doi: 10.1117/12.2038425.
[49]	D. Kiku, Y. Monno, M. Tanaka and M. Okutomi, Beyond color difference: Residual interpolation for color image demosaicking, IEEE Trans. Image Process., 25 (2016), 1288-1300. doi: 10.1109/TIP.2016.2518082.
[50]	F. Kokkinos and S. Lefkimmiatis, Iterative joint image demosaicking and denoising using a residual denoising network, IEEE Trans. Image Process., 28 (2019), 4177-4188. doi: 10.1109/TIP.2019.2905991.
[51]	M. Lebrun, A. Buades and J. M. Morel, A nonlocal bayesian image denoising algorithm, SIAM J. Imaging Sci., 6 (2013), 1665-1688. doi: 10.1137/120874989.
[52]	M. Lebrun, M. Colom, A. Buades and J. M. Morel, Secrets of image denoising cuisine, Acta Numer., 21 (2012), 475-576. doi: 10.1017/S0962492912000062.
[53]	M. S. Lee, S. W. Park and M. G. Kang, Denoising algorithm for cfa image sensors considering inter-channel correlation, Sensors, 17 (2017), 1236. doi: 10.3390/s17061236.
[54]	J. Liang, J. Li, Z. Shen and X. Zhang, Wavelet frame based color image demosaicing, Inverse Probl. Imaging, 7 (2013), 777-794. doi: 10.3934/ipi.2013.7.777.
[55]	L. Liu, X. Jia, J. Liu and Q. Tian, Joint demosaicing and denoising with self guidance, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., (2020), 2237-2246. doi: 10.1109/CVPR42600.2020.00231.
[56]	J. Mairal, F. Bach, J. Ponce, G. Sapiro and A. Zisserman, Non-local sparse models for image restoration, in Proc. IEEE Int. Conf. Comput. Vis., (2009), 2272-2279. doi: 10.1109/ICCV.2009.5459452.
[57]	H. S. Malvar, L.-w. He and R. Cutler, High-quality linear interpolation for demosaicing of bayer-patterned color images, in Proc. IEEE Int. Conf. Acoust. Speech. Signal. Process., 3 (2004), ⅲ-485. doi: 10.1109/ICASSP.2004.1326587.
[58]	Y. Monno, D. Kiku, M. Tanaka and M. Okutomi, Adaptive residual interpolation for color and multispectral image demosaicking, Sensors, 17 (2017), 2787. doi: 10.3390/s17122787.
[59]	A. Mosleh, A. Sharma, E. Onzon, F. Mannan, N. Robidoux and F. Heide, Hardware-in-the-loop end-to-end optimization of camera image processing pipelines, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2020), 7529-7538. doi: 10.1109/CVPR42600.2020.00755.
[60]	Y.-I. Ohta, T. Kanade and T. Sakai, Color information for region segmentation, Computer Graphics & Image Processing, 13 (1980), 222-241. doi: 10.1016/0146-664X(80)90047-7.
[61]	D. Paliy, M. Trimeche, V. Katkovnik and S. Alenius, Demosaicing of noisy data: Spatially adaptive approach, Proc. Image Processing: Algorithms and Systems V, 6497 (2007), 179-190. doi: 10.1117/12.713595.
[62]	S. H. Park, H. S. Kim, S. Lansel, M. Parmar and B. A. Wandell, A case for denoising before demosaicking color filter array data, in Proc. Conf. Rec. Asilomar Conf. Signals Syst. Comput., (2009), 860-864. doi: 10.1109/ACSSC.2009.5469990.
[63]	S. Patil and A. Rajwade, Poisson noise removal for image demosaicing, in Proc. Br. Mach. Vis. Conf., (2016), 33.1-33.10. doi: 10.5244/C.30.33.
[64]	I. Pekkucuksen and Y. Altunbasak, Gradient based threshold free color filter array interpolation, in Proc. IEEE Int. Conf. Image Process., (2010), 137-140. doi: 10.1109/ICIP.2010.5654327.
[65]	T. Plötz and S. Roth, Benchmarking denoising algorithms with real photographs, in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., (2017), 2750-2759. doi: 10.1109/CVPR.2017.294.
[66]	J. Portilla, V. Strela, M. J. Wainwright and E. P. Simoncelli, Image denoising using scale mixtures of gaussians in the wavelet domain, IEEE Trans. Image Process., 12 (2003), 1338-1351. doi: 10.1109/TIP.2003.818640.
[67]	L. I. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268. doi: 10.1016/0167-2789(92)90242-F.
[68]	N.-S. Syu, Y.-S. Chen and Y.-Y. Chuang, Learning deep convolutional networks for demosaicing, arXiv: 1802.03769.
[69]	R. Tan, K. Zhang, W. Zuo and L. Zhang, Color image demosaicking via deep residual learning, in Proc. IEEE Int. Conf. Multimedia Expo, (2017), 793-798.
[70]	J. Wu, R. Timofte and L. Van Gool, Demosaicing based on directional difference regression and efficient regression priors, IEEE Trans. Image Process., 25 (2016), 3862-3874. doi: 10.1109/TIP.2016.2574984.
[71]	X. Wu and L. Zhang, Temporal color video demosaicking via motion estimation and data fusion, IEEE Trans. Circuits Syst. Video Technol., 16 (2006), 231-240. doi: 10.1109/TCSVT.2005.860938.
[72]	W. Xing and K. Egiazarian, End-to-end learning for joint image demosaicing, denoising and super-resolution, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., (2021), 3507-3516. doi: 10.1109/CVPR46437.2021.00351.
[73]	S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, M.-H. Yang and L. Shao, CycleISP: Real image restoration via improved data synthesis, in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., (2020), 2693-2702. doi: 10.1109/CVPR42600.2020.00277.
[74]	K. Zhang, W. Zuo, Y. Chen, D. Meng and L. Zhang, Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising, IEEE Trans. Image Process., 26 (2017), 3142-3155. doi: 10.1109/TIP.2017.2662206.
[75]	K. Zhang, W. Zuo and L. Zhang, FFDNet: Toward a fast and flexible solution for cnn-based image denoising, IEEE Trans. Image Process., 27 (2018), 4608-4622. doi: 10.1109/TIP.2018.2839891.
[76]	L. Zhang, R. Lukac, X. Wu and D. Zhang, PCA-based spatially adaptive denoising of cfa images for single-sensor digital cameras, IEEE Trans. Image Process., 18 (2009), 797-812. doi: 10.1109/TIP.2008.2011384.
[77]	L. Zhang and X. Wu, Color demosaicking via directional linear minimum mean square-error estimation, IEEE Trans. Image Process., 14 (2005), 2167-2178. doi: 10.1109/TIP.2005.857260.
[78]	L. Zhang, X. Wu, A. Buades and X. Li, Color demosaicking by local directional interpolation and nonlocal adaptive thresholding, J. Electron. Imaging, 20 (2011), 023016. doi: 10.1117/1.3600632.
[79]	X. Zhang, M.-T. Sun, L. Fang and O. C. Au, Joint denoising and demosaicking of noisy cfa images based on inter-color correlation, in Proc. IEEE Int. Conf. Acoust. Speech. Signal. Process., (2014), 5784-5788. doi: 10.1109/ICASSP.2014.6854712.
[80]	D. Zoran and Y. Weiss, From learning models of natural image patches to whole image restoration, in Proc. Int. Conf. Comput. Vis., (2011), 479-486. doi: 10.1109/ICCV.2011.6126278.