Generative imaging and image processing via generative encoder

Yong Zheng Ong; Haizhao Yang

doi:10.3934/ipi.2021060

Previous Article
An inverse source problem for the stochastic wave equation
IPI Home
This Issue
Next Article
A stable non-iterative reconstruction algorithm for the acoustic inverse boundary value problem

doi: 10.3934/ipi.2021060

Online First

Online First articles are published articles within a journal that have not yet been assigned to a formal issue. This means they do not yet have a volume number, issue number, or page numbers assigned to them, however, they can still be found and cited using their DOI (Digital Object Identifier). Online First publication benefits the research community by making new scientific discoveries known as quickly as possible.

Readers can access Online First articles via the “Online First” tab for the selected journal.

Generative imaging and image processing via generative encoder

Yong Zheng Ong ^1, and Haizhao Yang ^2,

1.	National University of Singapore, 21 Lower Kent Ridge Rd, Singapore, Singapore 119077
2.	Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, USA

Received January 2021 Revised July 2021 Early access October 2021

This paper introduces a novel generative encoder (GE) framework for generative imaging and image processing tasks like image reconstruction, compression, denoising, inpainting, deblurring, and super-resolution. GE unifies the generative capacity of GANs and the stability of AEs in an optimization framework instead of stacking GANs and AEs into a single network or combining their loss functions as in existing literature. GE provides a novel approach to visualizing relationships between latent spaces and the data space. The GE framework is made up of a pre-training phase and a solving phase. In the former, a GAN with generator

$ G $

capturing the data distribution of a given image set, and an AE network with encoder

$ E $

that compresses images following the estimated distribution by

$ G $

are trained separately, resulting in two latent representations of the data, denoted as the generative and encoding latent space respectively. In the solving phase, given noisy image

$ x = \mathcal{P}(x^*) $

, where

$ x^* $

is the target unknown image,

$ \mathcal{P} $

is an operator adding an addictive, or multiplicative, or convolutional noise, or equivalently given such an image

$ x $

in the compressed domain, i.e., given

$ m = E(x) $

, the two latent spaces are unified via solving the optimization problem

$ z^* = \underset{z}{\mathrm{argmin}} \|E(G(z))-m\|_2^2+\lambda\|z\|_2^2 $

and the image

$ x^* $

is recovered in a generative way via

$ \hat{x}: = G(z^*)\approx x^* $

, where

$ \lambda>0 $

is a hyperparameter. The unification of the two spaces allows improved performance against corresponding GAN and AE networks while visualizing interesting properties in each latent space.

Keywords: Generative adversarial network, autoencoder, inverse problem, latent space, image reconstruction, denoising, deblurring, super resolution, inpainting.

Mathematics Subject Classification: Primary: 68U10; Secondary: 68T07.

Citation: Yong Zheng Ong, Haizhao Yang. Generative imaging and image processing via generative encoder. Inverse Problems & Imaging, doi: 10.3934/ipi.2021060

References:

[1]	M. Aharon, M. Elad and A. Bruckstein, K-svd: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on Signal Processing, 54 (2006), 4311-4322. Google Scholar
[2]	M. Arjovsky, S. Chintala and L. Bottou, Wasserstein generative adversarial networks, In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, PMLR, International Convention Centre, Sydney, Australia, 70 (2017), 214–223, http://proceedings.mlr.press/v70/arjovsky17a.html. Google Scholar
[3]	D. Bau, J.-Y. Zhu, J. Wulff, W. Peebles, H. Strobelt, B. Zhou and A. Torralba, Seeing what a GAN cannot generate, arXiv: 1910.11626 doi: 10.1109/ICCV.2019.00460. Google Scholar
[4]	D. Berthelot, T. Schumm and L. Metz, BEGAN: Boundary equilibrium generative adversarial networks, Computer Science, http://arXiv.org/abs/1703.10717. Google Scholar
[5]	A. Bora, A. Jalal, E. Price and A. G. Dimakis, Compressed sensing using generative models, ICML'17 Proceedings of the 34th International Conference on Machine Learning, 70 (2017), 537-546. Google Scholar
[6]	C. Bowles, L. J. Chen, R. Guerrero, P. Bentley, R. N. Gunn, A. Hammers, D. A. Dickie, M. del C. Valdés Hernández, J. M. Wardlaw and D. Rueckert, Gan augmentation: Augmenting training data using generative adversarial networks, arXiv: 1810.10863. Google Scholar
[7]	A. Buades, B. Coll and J. .Morel, A non-local algorithm for image denoising, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 2 (2005), 60-65. Google Scholar
[8]	E. J. Candes, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2006), 489-509. doi: 10.1109/TIT.2005.862083. Google Scholar
[9]	J. Chen, J. Chen, H. Chao and M. Yang, Image blind denoising with generative adversarial network based noise modeling, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi: 10.1109/CVPR.2018.00333. Google Scholar
[10]	T. Chen, X. Zhai, M. Ritter, M. Lucic and N. Houlsby, Self-supervised gans via auxiliary rotation loss, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 12146–12155. doi: 10.1109/CVPR.2019.01243. Google Scholar
[11]	A. Creswell and A. A. Bharath, Inverting the generator of A generative adversarial network, IEEE Transactions on Neural Networks and Learning Systems, 30 (2019), http://arXiv.org/abs/1611.05644. doi: 10.1109/TNNLS.2018.2875194. Google Scholar
[12]	K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Bm3d image denoising with shape-adaptive principal component analysis, Proc. Workshop on Signal Processing with Adaptive Sparse Structured Representations (SPARS'09). Google Scholar
[13]	D. Ulyanov, A. Vedaldi and V. Lempitsky, Deep image prior, arXiv: 1711.10925. Google Scholar
[14]	J. Donahue, P. Krähenbühl and T. Darrell, Adversarial feature learning, Computer Science, http://arXiv.org/abs/1605.09782. Google Scholar
[15]	W. Dong, L. Zhang, G. Shi and X. Wu, Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization, IEEE Trans. Image Process., 20 (2011), 1838-1857. doi: 10.1109/TIP.2011.2108306. Google Scholar
[16]	D. L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, 52 (2006), 1289-1306. doi: 10.1109/TIT.2006.871582. Google Scholar
[17]	V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro and A. C. Courville, Adversarially learned inference, arXiv: 1606.00704. Google Scholar
[18]	P. Getreuer, Total variation inpainting using split bregman, Image Processing On Line, 2 (2012), 147-157. Google Scholar
[19]	I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, Generative adversarial nets, Advances in Neural Information Processing Systems, 27 (NIPS 2014). Google Scholar
[20]	M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer and S. Hochreiter, Gans trained by a two time-scale update rule converge to a nash equilibrium, Computer Science, http://arXiv.org/abs/1706.08500. Google Scholar
[21]	S.-W. Huang, C.-T. Lin, S.-P. Chen, Y.-Y. Wu, P.-H. Hsu and S.-H. Lai, Auggan: Cross domain adaptation with gan-based data augmentation, In Computer Vision – ECCV 2018, (eds. V. Ferrari, M. Hebert, C. Sminchisescu and Y. Weiss), Springer International Publishing, Cham, (2018), 731–744. Google Scholar
[22]	T. Karras, T. Aila, S. Laine and J. Lehtinen, Progressive growing of gans for improved quality, stability, and variation, Computer Science, http://arXiv.org/abs/1710.10196. Google Scholar
[23]	D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, Published as A Conference Paper at the 3rd International Conference for Learning Representations, San Diego, 2015, 2014, http://arXiv.org/abs/1412.6980. Google Scholar
[24]	D. P. Kingma and M. Welling, Auto-encoding variational bayes, 2013. Google Scholar
[25]	O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin and J. Matas, Deblurgan: Blind motion deblurring using conditional adversarial networks, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2018), 8183–8192. Google Scholar
[26]	A. B. L. Larsen, S. K. Sønderby and O. Winther, Autoencoding beyond pixels using a learned similarity metric, Computer Science, http://arXiv.org/abs/1512.09300. Google Scholar
[27]	C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, Photo-realistic single image super-resolution using a generative adversarial network, (2017), 105–114. Google Scholar
[28]	Q. Lei, A. Jalal, I. S. Dhillon and A. G. Dimakis, Inverting deep generative models, one layer at a time, Computer Science, 2019, http://arXiv.org/abs/1906.07437. Google Scholar
[29]	X. Liang, H. Zhang and E. P. Xing, Generative semantic manipulation with mask-contrasting GAN, Lecture Notes in Computer Science, 11217 (2018), 574–590, http://arXiv.org/abs/1708.00315. doi: 10.1007/978-3-030-01261-8_34. Google Scholar
[30]	Z. C. Lipton and S. Tripathi, Precise recovery of latent vectors from generative adversarial networks, Computer Science, http://arXiv.org/abs/1702.04782. Google Scholar
[31]	Z. Liu, P. Luo, X. Wang and X. Tang, Deep learning face attributes in the wild, In IEEE International Conference on Computer Vision (ICCV), 2015. doi: 10.1109/ICCV.2015.425. Google Scholar
[32]	S. Menon, A. Damian, S. Hu, N. Ravi and C. Rudin, Pulse: Self-supervised photo upsampling via latent space exploration of generative models, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2002), 2434–2442. Google Scholar
[33]	A. Nguyen, J. Yosinski, Y. Bengio, A. Dosovitskiy and J. Clune, Plug & play generative networks: Conditional iterative generation of images in latent space, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, http://arXiv.org/abs/1612.00005. doi: 10.1109/CVPR.2017.374. Google Scholar
[34]	D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell and A. Efros, Context encoders: Feature learning by inpainting, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi: 10.1109/CVPR.2016.278. Google Scholar
[35]	A. Radford, L. Metz and S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, In ICLR, 2016. Google Scholar
[36]	T. Ramstad, Bentheimer micro-ct with waterflood, 2018, http://www.digitalrocksportal.org/projects/172. Google Scholar
[37]	M. Rosca, B. Lakshminarayanan, D. Warde-Farley and S. Mohamed, Variational approaches for auto-encoding generative adversarial networks, arXiv: 1706.04987. Google Scholar
[38]	L. I. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268. doi: 10.1016/0167-2789(92)90242-F. Google Scholar
[39]	J. Schlemper, J. Caballero, J. V. Hajnal, A. Price and D. Rueckert, A deep cascade of convolutional neural networks for mr image reconstruction, Information Processing in Medical Imaging, 10265 (2017), 647-658. doi: 10.1007/978-3-319-59050-9_51. Google Scholar
[40]	V. Shah and C. Hegde, Solving linear inverse problems using gan priors: An algorithm with provable guarantees, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2018), 4609–4613. doi: 10.1109/ICASSP.2018.8462233. Google Scholar
[41]	Y. Shen, J. Gu, X. Tang and B. Zhou, Interpreting the latent space of gans for semantic face editing, Computer Science, http://arXiv.org/abs/1907.10786. Google Scholar
[42]	D. Ulyanov, A. Vedaldi and V. S. Lempitsky, Adversarial generator-encoder networks, Computer Science, http://arXiv.org/abs/1704.02304. Google Scholar
[43]	P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P. Manzagol, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, J. Mach. Learn. Res., 11 (2010), 3371–3408, http://dl.acm.org/citation.cfm?id=1756006.1953039. Google Scholar
[44]	P. Vincent, H. Larochelle, Y. Bengio and P.-A. Manzagol, Extracting and composing robust features with denoising autoencoders, In ICML '08, (2008), 1096–1103. doi: 10.1145/1390156.1390294. Google Scholar
[45]	Z. Wang, A. Bovik, H. Sheikh and E. Simoncelli, Image quality assessment: From error visibility to structural similarity, Image Processing, IEEE Transactions on, 13 (2004), 600-612. doi: 10.1109/TIP.2003.819861. Google Scholar
[46]	D. Warde-Farley and Y. Bengio, Improving generative adversarial networks with denoising feature matching, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017, https://openreview.net/forum?id=S1X7nhsxl. Google Scholar
[47]	L. Xu and J. Jia, Two-phase kernel estimation for robust motion deblurring, Lecture Notes in Computer Science, 6311 (2010), 157-170. doi: 10.1007/978-3-642-15549-9_12. Google Scholar
[48]	Q. Yan and W. Wang, DCGANsfor image super-resolution, denoising and debluring., Google Scholar
[49]	R. Yan and L. Shao, Blind image blur estimation via deep learning, IEEE Trans Image Process, 25 (2016), 1910-1921. Google Scholar
[50]	R. A. Yeh, C. Chen, T. Lim, A. G. Schwing, M. Hasegawa-Johnson and M. N. Do, Semantic image inpainting with deep generative models, In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR 2017), Honolulu, HI, USA,, (2017), 6882–6890 doi: 10.1109/CVPR.2017.728. Google Scholar
[51]	F. Yu, Y. Zhang, S. Song, A. Seff, T. Funkhouser and J. Xiao, LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop, Computer Science, http://arXiv.org/abs/1506.03365. Google Scholar
[52]	J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu and T. S. Huang, Generative image inpainting with contextual attention, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. doi: 10.1109/CVPR.2018.00577. Google Scholar
[53]	H. Zhang, Z. Zhang, A. Odena and H. Lee, Consistency regularization for generative adversarial networks, arXiv: 1910.12027. Google Scholar
[54]	J. Zhang, D. Zhao and W. Gao, Group-based sparse representation for image restoration, IEEE Trans. Image Process., 23 (2014), 3336-3351. doi: 10.1109/TIP.2014.2323127. Google Scholar
[55]	J. J. Zhao, M. Mathieu and Y. LeCun, Energy-based generative adversarial networks, In ICLR, 2017. Google Scholar

show all references

References:

[1]	M. Aharon, M. Elad and A. Bruckstein, K-svd: An algorithm for designing overcomplete dictionaries for sparse representation, IEEE Transactions on Signal Processing, 54 (2006), 4311-4322. Google Scholar
[2]	M. Arjovsky, S. Chintala and L. Bottou, Wasserstein generative adversarial networks, In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, PMLR, International Convention Centre, Sydney, Australia, 70 (2017), 214–223, http://proceedings.mlr.press/v70/arjovsky17a.html. Google Scholar
[3]	D. Bau, J.-Y. Zhu, J. Wulff, W. Peebles, H. Strobelt, B. Zhou and A. Torralba, Seeing what a GAN cannot generate, arXiv: 1910.11626 doi: 10.1109/ICCV.2019.00460. Google Scholar
[4]	D. Berthelot, T. Schumm and L. Metz, BEGAN: Boundary equilibrium generative adversarial networks, Computer Science, http://arXiv.org/abs/1703.10717. Google Scholar
[5]	A. Bora, A. Jalal, E. Price and A. G. Dimakis, Compressed sensing using generative models, ICML'17 Proceedings of the 34th International Conference on Machine Learning, 70 (2017), 537-546. Google Scholar
[6]	C. Bowles, L. J. Chen, R. Guerrero, P. Bentley, R. N. Gunn, A. Hammers, D. A. Dickie, M. del C. Valdés Hernández, J. M. Wardlaw and D. Rueckert, Gan augmentation: Augmenting training data using generative adversarial networks, arXiv: 1810.10863. Google Scholar
[7]	A. Buades, B. Coll and J. .Morel, A non-local algorithm for image denoising, IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 2 (2005), 60-65. Google Scholar
[8]	E. J. Candes, J. Romberg and T. Tao, Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information, IEEE Trans. Inform. Theory, 52 (2006), 489-509. doi: 10.1109/TIT.2005.862083. Google Scholar
[9]	J. Chen, J. Chen, H. Chao and M. Yang, Image blind denoising with generative adversarial network based noise modeling, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi: 10.1109/CVPR.2018.00333. Google Scholar
[10]	T. Chen, X. Zhai, M. Ritter, M. Lucic and N. Houlsby, Self-supervised gans via auxiliary rotation loss, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2019), 12146–12155. doi: 10.1109/CVPR.2019.01243. Google Scholar
[11]	A. Creswell and A. A. Bharath, Inverting the generator of A generative adversarial network, IEEE Transactions on Neural Networks and Learning Systems, 30 (2019), http://arXiv.org/abs/1611.05644. doi: 10.1109/TNNLS.2018.2875194. Google Scholar
[12]	K. Dabov, A. Foi, V. Katkovnik and K. Egiazarian, Bm3d image denoising with shape-adaptive principal component analysis, Proc. Workshop on Signal Processing with Adaptive Sparse Structured Representations (SPARS'09). Google Scholar
[13]	D. Ulyanov, A. Vedaldi and V. Lempitsky, Deep image prior, arXiv: 1711.10925. Google Scholar
[14]	J. Donahue, P. Krähenbühl and T. Darrell, Adversarial feature learning, Computer Science, http://arXiv.org/abs/1605.09782. Google Scholar
[15]	W. Dong, L. Zhang, G. Shi and X. Wu, Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization, IEEE Trans. Image Process., 20 (2011), 1838-1857. doi: 10.1109/TIP.2011.2108306. Google Scholar
[16]	D. L. Donoho, Compressed sensing, IEEE Trans. Inform. Theory, 52 (2006), 1289-1306. doi: 10.1109/TIT.2006.871582. Google Scholar
[17]	V. Dumoulin, I. Belghazi, B. Poole, A. Lamb, M. Arjovsky, O. Mastropietro and A. C. Courville, Adversarially learned inference, arXiv: 1606.00704. Google Scholar
[18]	P. Getreuer, Total variation inpainting using split bregman, Image Processing On Line, 2 (2012), 147-157. Google Scholar
[19]	I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville and Y. Bengio, Generative adversarial nets, Advances in Neural Information Processing Systems, 27 (NIPS 2014). Google Scholar
[20]	M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, G. Klambauer and S. Hochreiter, Gans trained by a two time-scale update rule converge to a nash equilibrium, Computer Science, http://arXiv.org/abs/1706.08500. Google Scholar
[21]	S.-W. Huang, C.-T. Lin, S.-P. Chen, Y.-Y. Wu, P.-H. Hsu and S.-H. Lai, Auggan: Cross domain adaptation with gan-based data augmentation, In Computer Vision – ECCV 2018, (eds. V. Ferrari, M. Hebert, C. Sminchisescu and Y. Weiss), Springer International Publishing, Cham, (2018), 731–744. Google Scholar
[22]	T. Karras, T. Aila, S. Laine and J. Lehtinen, Progressive growing of gans for improved quality, stability, and variation, Computer Science, http://arXiv.org/abs/1710.10196. Google Scholar
[23]	D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, Published as A Conference Paper at the 3rd International Conference for Learning Representations, San Diego, 2015, 2014, http://arXiv.org/abs/1412.6980. Google Scholar
[24]	D. P. Kingma and M. Welling, Auto-encoding variational bayes, 2013. Google Scholar
[25]	O. Kupyn, V. Budzan, M. Mykhailych, D. Mishkin and J. Matas, Deblurgan: Blind motion deblurring using conditional adversarial networks, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, (2018), 8183–8192. Google Scholar
[26]	A. B. L. Larsen, S. K. Sønderby and O. Winther, Autoencoding beyond pixels using a learned similarity metric, Computer Science, http://arXiv.org/abs/1512.09300. Google Scholar
[27]	C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, Photo-realistic single image super-resolution using a generative adversarial network, (2017), 105–114. Google Scholar
[28]	Q. Lei, A. Jalal, I. S. Dhillon and A. G. Dimakis, Inverting deep generative models, one layer at a time, Computer Science, 2019, http://arXiv.org/abs/1906.07437. Google Scholar
[29]	X. Liang, H. Zhang and E. P. Xing, Generative semantic manipulation with mask-contrasting GAN, Lecture Notes in Computer Science, 11217 (2018), 574–590, http://arXiv.org/abs/1708.00315. doi: 10.1007/978-3-030-01261-8_34. Google Scholar
[30]	Z. C. Lipton and S. Tripathi, Precise recovery of latent vectors from generative adversarial networks, Computer Science, http://arXiv.org/abs/1702.04782. Google Scholar
[31]	Z. Liu, P. Luo, X. Wang and X. Tang, Deep learning face attributes in the wild, In IEEE International Conference on Computer Vision (ICCV), 2015. doi: 10.1109/ICCV.2015.425. Google Scholar
[32]	S. Menon, A. Damian, S. Hu, N. Ravi and C. Rudin, Pulse: Self-supervised photo upsampling via latent space exploration of generative models, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), (2002), 2434–2442. Google Scholar
[33]	A. Nguyen, J. Yosinski, Y. Bengio, A. Dosovitskiy and J. Clune, Plug & play generative networks: Conditional iterative generation of images in latent space, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, http://arXiv.org/abs/1612.00005. doi: 10.1109/CVPR.2017.374. Google Scholar
[34]	D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell and A. Efros, Context encoders: Feature learning by inpainting, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi: 10.1109/CVPR.2016.278. Google Scholar
[35]	A. Radford, L. Metz and S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, In ICLR, 2016. Google Scholar
[36]	T. Ramstad, Bentheimer micro-ct with waterflood, 2018, http://www.digitalrocksportal.org/projects/172. Google Scholar
[37]	M. Rosca, B. Lakshminarayanan, D. Warde-Farley and S. Mohamed, Variational approaches for auto-encoding generative adversarial networks, arXiv: 1706.04987. Google Scholar
[38]	L. I. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992), 259-268. doi: 10.1016/0167-2789(92)90242-F. Google Scholar
[39]	J. Schlemper, J. Caballero, J. V. Hajnal, A. Price and D. Rueckert, A deep cascade of convolutional neural networks for mr image reconstruction, Information Processing in Medical Imaging, 10265 (2017), 647-658. doi: 10.1007/978-3-319-59050-9_51. Google Scholar
[40]	V. Shah and C. Hegde, Solving linear inverse problems using gan priors: An algorithm with provable guarantees, IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), (2018), 4609–4613. doi: 10.1109/ICASSP.2018.8462233. Google Scholar
[41]	Y. Shen, J. Gu, X. Tang and B. Zhou, Interpreting the latent space of gans for semantic face editing, Computer Science, http://arXiv.org/abs/1907.10786. Google Scholar
[42]	D. Ulyanov, A. Vedaldi and V. S. Lempitsky, Adversarial generator-encoder networks, Computer Science, http://arXiv.org/abs/1704.02304. Google Scholar
[43]	P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio and P. Manzagol, Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, J. Mach. Learn. Res., 11 (2010), 3371–3408, http://dl.acm.org/citation.cfm?id=1756006.1953039. Google Scholar
[44]	P. Vincent, H. Larochelle, Y. Bengio and P.-A. Manzagol, Extracting and composing robust features with denoising autoencoders, In ICML '08, (2008), 1096–1103. doi: 10.1145/1390156.1390294. Google Scholar
[45]	Z. Wang, A. Bovik, H. Sheikh and E. Simoncelli, Image quality assessment: From error visibility to structural similarity, Image Processing, IEEE Transactions on, 13 (2004), 600-612. doi: 10.1109/TIP.2003.819861. Google Scholar
[46]	D. Warde-Farley and Y. Bengio, Improving generative adversarial networks with denoising feature matching, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017, https://openreview.net/forum?id=S1X7nhsxl. Google Scholar
[47]	L. Xu and J. Jia, Two-phase kernel estimation for robust motion deblurring, Lecture Notes in Computer Science, 6311 (2010), 157-170. doi: 10.1007/978-3-642-15549-9_12. Google Scholar
[48]	Q. Yan and W. Wang, DCGANsfor image super-resolution, denoising and debluring., Google Scholar
[49]	R. Yan and L. Shao, Blind image blur estimation via deep learning, IEEE Trans Image Process, 25 (2016), 1910-1921. Google Scholar
[50]	R. A. Yeh, C. Chen, T. Lim, A. G. Schwing, M. Hasegawa-Johnson and M. N. Do, Semantic image inpainting with deep generative models, In IEEE Conference on Computer Vision and Pattern Recognition, (CVPR 2017), Honolulu, HI, USA,, (2017), 6882–6890 doi: 10.1109/CVPR.2017.728. Google Scholar
[51]	F. Yu, Y. Zhang, S. Song, A. Seff, T. Funkhouser and J. Xiao, LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop, Computer Science, http://arXiv.org/abs/1506.03365. Google Scholar
[52]	J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu and T. S. Huang, Generative image inpainting with contextual attention, IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018. doi: 10.1109/CVPR.2018.00577. Google Scholar
[53]	H. Zhang, Z. Zhang, A. Odena and H. Lee, Consistency regularization for generative adversarial networks, arXiv: 1910.12027. Google Scholar
[54]	J. Zhang, D. Zhao and W. Gao, Group-based sparse representation for image restoration, IEEE Trans. Image Process., 23 (2014), 3336-3351. doi: 10.1109/TIP.2014.2323127. Google Scholar
[55]	J. J. Zhao, M. Mathieu and Y. LeCun, Energy-based generative adversarial networks, In ICLR, 2017. Google Scholar

Figure 1. Flow of training process in GE. Step 1 and 2 forms the pre-training phase, while the remaining form the solving phase

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 2. Reconstruction results on CelebA dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 3. Reconstruction results on Digital Rock dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 4. Reconstruction results on LSUN church dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 5. Denoising results on CelebA dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 6. Deblurring results on CelebA dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 7. Super-resolution results on CelebA dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 8. Inpainting results on CelebA dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 9. Plot of log of average MSE based on number of iterations in the solving phase

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 10. Comparison of image reconstruction of detail region (red box) for original image (left). In order of comparison, from left to right, we have Original, GE, invertGAN, ConvAE

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 11. Comparison of image reconstruction of detail region (red box) for original image (left). In order of comparison, from top to bottom, we have Original, GE, invertGAN, ConvAE

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 12. Additional pore sample result on Digital Rock dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 13. Missing spectacles sample results on CelebA dataset

Figure Options

Download full-size image

Download as PowerPoint slide

Figure 14. Reconstruction results for $ 64\times 64\times 3 $ images in CelebA with GE using BEGAN instead of pGAN

Figure Options

Download full-size image

Download as PowerPoint slide

Table 1. Structure of $ E $. The decoder $ DC $ is a mirror of $ E $ using conv_transpose and upsample

layer type	layer
conv2d	$ k=[3,3,3,f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,f,2*f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,2f,4f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,4f,8f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,8f,16f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,16f,32f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
fullyconnected	$ h=256 $

Table Options

Download as excel

layer type	layer
conv2d	$ k=[3,3,3,f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,f,2*f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,2f,4f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,4f,8f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,8f,16f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
conv2d	$ k=[3,3,16f,32f],s=[1,1],a=ReLU $
maxpool2d	$ k=[1,2,2,1],s=[2,2] $
fullyconnected	$ h=256 $

Table 2. Quantitative results comparing models for CelebA. Additionally, some FID scores reported by recent GAN papers that used CelebA $ 128\times128\times3 $ images are also presented for comparison, labelled with *

Model	MSE	SSIM	FID
CRGAN*			16.97
SSGAN*			24.36
Our pGAN			22.13
ConvAE	0.03386	0.6823$ \pm $0.051	87.71
AEGAN	0.03317	0.6907$ \pm $0.050	34.53
invertGAN	0.03529	0.7203$ \pm $0.038	19.19
GE	0.03262	0.7329$ \pm $0.025	17.42

Table Options

Download as excel

Model	MSE	SSIM	FID
CRGAN*			16.97
SSGAN*			24.36
Our pGAN			22.13
ConvAE	0.03386	0.6823$ \pm $0.051	87.71
AEGAN	0.03317	0.6907$ \pm $0.050	34.53
invertGAN	0.03529	0.7203$ \pm $0.038	19.19
GE	0.03262	0.7329$ \pm $0.025	17.42

Table 3. Quantitative results comparing models for digital rocks. The number in brackets show the size of the latent vector in pGAN that the model is trained on. Models with same latent sizes are solved with the same pGAN weights. The same AE is used for all models

Model	MSE	PSNR
ConvAE	0.009271	20.32
invertGAN (512)	0.008185	20.86
GE (512)	0.007470	21.26
GE (256)	0.007741	21.11
GE (128)	0.007839	21.05
GE (64)	0.008499	20.70

Table Options

Download as excel

Model	MSE	PSNR
ConvAE	0.009271	20.32
invertGAN (512)	0.008185	20.86
GE (512)	0.007470	21.26
GE (256)	0.007741	21.11
GE (128)	0.007839	21.05
GE (64)	0.008499	20.70

Table 4. Results of invertGAN, GE on spectacles. T refers to samples which produced spectacles, F refers to samples which did not. Remaining are invalid reconstructions

	InvertGAN, F	InvertGAN, T
GE, F	289	32
GE, T	157	469

Table Options

Download as excel

	InvertGAN, F	InvertGAN, T
GE, F	289	32
GE, T	157	469

[1]	Susu Zhang, Jiancheng Ni, Lijun Hou, Zili Zhou, Jie Hou, Feng Gao. Global-Affine and Local-Specific Generative Adversarial Network for semantic-guided image generation. Mathematical Foundations of Computing, 2021, 4 (3) : 145-165. doi: 10.3934/mfc.2021009
[2]	Changming Song, Yun Wang. Nonlocal latent low rank sparse representation for single image super resolution via self-similarity learning. Inverse Problems & Imaging, 2021, 15 (6) : 1347-1362. doi: 10.3934/ipi.2021017
[3]	Lacramioara Grecu, Constantin Popa. Constrained SART algorithm for inverse problems in image reconstruction. Inverse Problems & Imaging, 2013, 7 (1) : 199-216. doi: 10.3934/ipi.2013.7.199
[4]	Amine Laghrib, Abdelkrim Chakib, Aissam Hadri, Abdelilah Hakim. A nonlinear fourth-order PDE for multi-frame image super-resolution enhancement. Discrete & Continuous Dynamical Systems - B, 2020, 25 (1) : 415-442. doi: 10.3934/dcdsb.2019188
[5]	Wei Wan, Weihong Guo, Jun Liu, Haiyang Huang. Non-local blind hyperspectral image super-resolution via 4d sparse tensor factorization and low-rank. Inverse Problems & Imaging, 2020, 14 (2) : 339-361. doi: 10.3934/ipi.2020015
[6]	Moez Kallel, Maher Moakher, Anis Theljani. The Cauchy problem for a nonlinear elliptic equation: Nash-game approach and application to image inpainting. Inverse Problems & Imaging, 2015, 9 (3) : 853-874. doi: 10.3934/ipi.2015.9.853
[7]	Jie Huang, Marco Donatelli, Raymond H. Chan. Nonstationary iterated thresholding algorithms for image deblurring. Inverse Problems & Imaging, 2013, 7 (3) : 717-736. doi: 10.3934/ipi.2013.7.717
[8]	Weihong Guo, Jing Qin. A geometry guided image denoising scheme. Inverse Problems & Imaging, 2013, 7 (2) : 499-521. doi: 10.3934/ipi.2013.7.499
[9]	Nam-Yong Lee, Bradley J. Lucier. Preconditioned conjugate gradient method for boundary artifact-free image deblurring. Inverse Problems & Imaging, 2016, 10 (1) : 195-225. doi: 10.3934/ipi.2016.10.195
[10]	Weishi Yin, Jiawei Ge, Pinchao Meng, Fuheng Qu. A neural network method for the inverse scattering problem of impenetrable cavities. Electronic Research Archive, 2020, 28 (2) : 1123-1142. doi: 10.3934/era.2020062
[11]	Lucie Baudouin, Emmanuelle Crépeau, Julie Valein. Global Carleman estimate on a network for the wave equation and application to an inverse problem. Mathematical Control & Related Fields, 2011, 1 (3) : 307-330. doi: 10.3934/mcrf.2011.1.307
[12]	Xiangtuan Xiong, Jinmei Li, Jin Wen. Some novel linear regularization methods for a deblurring problem. Inverse Problems & Imaging, 2017, 11 (2) : 403-426. doi: 10.3934/ipi.2017019
[13]	Nils Dabrock, Yves van Gennip. A note on "Anisotropic total variation regularized $L^1$-approximation and denoising/deblurring of 2D bar codes". Inverse Problems & Imaging, 2018, 12 (2) : 525-526. doi: 10.3934/ipi.2018022
[14]	Rustum Choksi, Yves van Gennip, Adam Oberman. Anisotropic total variation regularized $L^1$ approximation and denoising/deblurring of 2D bar codes. Inverse Problems & Imaging, 2011, 5 (3) : 591-617. doi: 10.3934/ipi.2011.5.591
[15]	Yuan Shen, Lei Ji. Partial convolution for total variation deblurring and denoising by new linearized alternating direction method of multipliers with extension step. Journal of Industrial & Management Optimization, 2019, 15 (1) : 159-175. doi: 10.3934/jimo.2018037
[16]	Wei Wan, Haiyang Huang, Jun Liu. Local block operators and TV regularization based image inpainting. Inverse Problems & Imaging, 2018, 12 (6) : 1389-1410. doi: 10.3934/ipi.2018058
[17]	Marko Filipović, Ivica Kopriva. A comparison of dictionary based approaches to inpainting and denoising with an emphasis to independent component analysis learned dictionaries. Inverse Problems & Imaging, 2011, 5 (4) : 815-841. doi: 10.3934/ipi.2011.5.815
[18]	Tianyu Yang, Yang Yang. A stable non-iterative reconstruction algorithm for the acoustic inverse boundary value problem. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021038
[19]	Valter Pohjola. An inverse problem for the magnetic Schrödinger operator on a half space with partial data. Inverse Problems & Imaging, 2014, 8 (4) : 1169-1189. doi: 10.3934/ipi.2014.8.1169
[20]	Fatimzehrae Ait Bella, Aissam Hadri, Abdelilah Hakim, Amine Laghrib. A nonlocal Weickert type PDE applied to multi-frame super-resolution. Evolution Equations & Control Theory, 2021, 10 (3) : 633-655. doi: 10.3934/eect.2020084

2020 Impact Factor: 1.639

Tools

Metrics

Tools

Article outline

Figures and Tables

2020 Impact Factor: 1.639

Tools

Figures and Tables

2020 Impact Factor: 1.639

American Institute of Mathematical Sciences

Generative imaging and image processing via generative encoder

References:

References:

Tools

Metrics

Other articles
by authors

Tools

Tools

Tools

Metrics

Other articles
by authors

American Institute of Mathematical Sciences

Generative imaging and image processing via generative encoder

References:

References:

Tools

Metrics

Other articlesby authors

Tools

Tools

Tools

Metrics

Other articlesby authors

Other articles
by authors

Other articles
by authors