Video stabilization of atmospheric turbulence distortion

Previous Article
A conformal approach for surface inpainting
IPI Home
This Issue
Next Article
How to explore the patch space

August 2013, 7(3): 839-861. doi: 10.3934/ipi.2013.7.839

Video stabilization of atmospheric turbulence distortion

Yifei Lou ^1, , Sung Ha Kang ^2, , Stefano Soatto ^3, and Andrea L. Bertozzi ^1,

1.	Department of Mathematics, University of California Los Angeles, Los Angeles, CA, 90095, United States, United States
2.	School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332-0160
3.	Computer Science Department, University of California Los Angeles, Los Angeles, CA, 90095, United States

Received April 2012 Revised March 2013 Published September 2013

Abstract
Related Papers
Under a Creative Commons license

We present a method to enhance the quality of a video sequence captured through a turbulent atmospheric medium, and give an estimate of the radiance of the distant scene, represented as a ``latent image,'' which is assumed to be static throughout the video. Due to atmospheric turbulence, temporal averaging produces a blurred version of the scene's radiance. We propose a method combining Sobolev gradient and Laplacian to stabilize the video sequence, and a latent image is further found utilizing the ``lucky region" method. The video sequence is stabilized while keeping sharp details, and the latent image shows more consistent straight edges. We analyze the well-posedness for the stabilizing PDE and the linear stability of the numerical scheme.

Keywords: spectral methods, Imaging through turbulence, anisotropic diffusion, Sobolev gradient sharpening, image fusion..

Mathematics Subject Classification: Primary: 68U10; Secondary: 35B65, 35B3.

Citation: Yifei Lou, Sung Ha Kang, Stefano Soatto, Andrea L. Bertozzi. Video stabilization of atmospheric turbulence distortion. Inverse Problems and Imaging, 2013, 7 (3) : 839-861. doi: 10.3934/ipi.2013.7.839

References:

[1]	L. Alvarez and L. Mazorra, Signal and image restoration using shock filters and anisotropic diffusion, SIAM Journal on Numerical Analysis, 31 (1994), 590-065. doi: 10.1137/0731032.
[2]	M. Aubailly, M. A. Vorontsov, G. W. Carhat and M. T. Valley, Automated video enhancement from a stream of atmospherically-distorted images: The lucky-region fusion approach, In "Proceedings of SPIE," volume 7463, (2009). doi: 10.1117/12.828332.
[3]	A. Buades, B. Coll and J. M. Morel, A review of image denoising algorithms, with a new one, Multiscale Modeling and Simulation, 4 (2005), 490-530. doi: 10.1137/040616024.
[4]	A. Buades, B. Coll and J.M. Morel, Nonlocal image and movie denoising, International Journal of Computer Vision, 76 (2008), 123-139.
[5]	K. Buskila, S. Towito, E. Shmuel, R. Levi, N. Kopeika, K. Krapels, R. Driggers, R. Vollmerhausen and C. Halford, Atmospheric modulation transfer function in the infrared, Applied Optics, 43 (2004), 471-482. doi: 10.1364/AO.43.000471.
[6]	J. Calder, A. Mansouri and A. Yezzi, Image sharpening via sobolev gradient flows, SIAM Journal on Imaging Sciences, 3 (2010), 981-1014. doi: 10.1137/090771260.
[7]	J. Caviedes and S. Gurbuz, No-reference sharpness metric based on local edge kurtosis, In "IEEE Int. Conf. on Image Processing," 3 (2002), 53-56. doi: 10.1109/ICIP.2002.1038901.
[8]	L. C. Evans, "Partial Differential Equations," Graduate Studies in Mathematics, 19. American Mathematical Society, Providence, RI, 1998.
[9]	D. Frakes, J. Monaco and M. Smith, Suppression of atmospheric turbulence in video using an adaptive control grid interpolation approach, In "IEEE Int. Conf. on Acoustics, Speech and Signal processing (ICASSP)," 3 (2001), 1881-1884. doi: 10.1109/ICASSP.2001.941311.
[10]	D. L. Fried, Optical resolution through a randomly inhomogeneous medium for very long and very short exposures, Journal of the Optical Society of America, 56 (1966), 1372-1379. doi: 10.1364/JOSA.56.001372.
[11]	S. Gepshtein, A. Shtainman, B. Fishbain and L. P. Yaroslavsky, Restoration of atmospheric turbulent video containing real motion using rank filtering and elastic image registration, In "Proceeding of the Eusipco," 2004.
[12]	J. Gilles and S. Osher, "Fried Deconvolution," UCLA CAM Report 11-62, 2011. doi: 10.1117/12.917234.
[13]	P. Hartman, "Ordinary Differential Equations," Corrected reprint. S. M. Hartman, Baltimore, Md., 1973.
[14]	M. Hirsch, S. Sra, B. Scholkopf and S. Harmeling, Efficient filter flow for space-variant multiframe blind deconvolution, IEEE Computer Vision and Pattern Recognition (CVPR), pages 607-614, (2010). doi: 10.1109/CVPR.2010.5540158.
[15]	R. E. Hufnagel and N. R. Stanley, Modulation transfer function associated with image transmission through turbulence media, Journal of the Optical Society of America, 54 (1964), 52-60. doi: 10.1364/JOSA.54.000052.
[16]	J.Gilles, T. Dagobert and C. De Franchis, Atmospheric turbulence restoration by diffeomorphism image registration and blind deconvolution, In "Advanced Concepts for Intelligent Vision Systems" (ACIVS), Oct 2008.
[17]	D. Li, R. Mersereau and S. Simske, Atmospheric turbulence degraded image restoration using principal components analysis, IEEE Geoscience and Remote Sensing Letters, 4 (2007), 340-344. doi: 10.1109/LGRS.2007.895691.
[18]	D. Li and S. Simske, Atmospheric turbulence degraded-image restoration by kurtosis minimization, IEEE Geoscience and Remote Sensing Letters, 6 (2009), 244-247.
[19]	Y. Mao and J. Gilles, Non rigid geometric distortions correction - application to atmospheric turbulence stabilization, Inverse Problems and Imaging, 6 (2012), 531-546. doi: 10.3934/ipi.2012.6.531.
[20]	A. Marquina, Nonlinear inverse scale space methods for total variation blind deconvolution, SIAM Journal on Imaging Sciences, 2 (2009), 64-83. doi: 10.1137/080724289.
[21]	M. Micheli, Y. Lou, S. Soatto and Andrea L. Bertozzi, A linear systems approach to imaging through turbulence, Journal of Mathematical Imaging and Vision July 2013. doi: 10.1007/s10851-012-0410-7.
[22]	A. V. Oppenheim and R. W. Schafer, "Discrete-Time Signal Processing," 2nd Edition Prentice Hall, Upper Saddle River, NJ, 1999.
[23]	P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Trans. on Pattern Analysis and Machine Intelligence, 12 (1990), 629-639. doi: 10.1109/34.56205.
[24]	J. Ricklin and F. Davidson, Atmospheric turbulence effects on a partially coherent gaussian beam: implications for free-space laser communication, Journal of the Optical Society of America A, pages 1794-1802, (2002). doi: 10.1364/JOSAA.19.001794.
[25]	M. Roggemann and B. Welsh, "Imaging Through Turbulence," CRC Press, Boca Raton, FL, 1996. doi: 10.1117/1.601043.
[26]	M. Shimizu, S. Yoshimura, M. Tanaka and M. Okutomi, Super-resolution from image sequence under influence of hot-air optical turbulence, In "IEEE Conference on Computer Vision and Pattern Recognition" (CVPR), pages 1-8, (2008).
[27]	G. Sundaramoorthi, A. Yezzi and A. C. Mennucci, Sobolev active contours, International Journal of Computer Vision, 73 (2007), 345-366.
[28]	D. Tofsted, Reanalysis of turbulence effects on short-exposure passive imaging, Optical Engineering, 50 (2011), 016001. doi: 10.1117/1.3532999.
[29]	M. A. Vorontsov and G. W. Carhart, Anisoplanatic imaging through turbulent media: Image recovery by local information fusion from a set of short-exposure images, Journal of the Optical Society of America A, 18 (2001), 1312-1324. doi: 10.1364/JOSAA.18.001312.
[30]	P. Zhang, W. Gong, X. Shen and S. Han, Correlated imaging through atmospheric turbulence, Physical Review A, 82 (2010), 033817. doi: 10.1103/PhysRevA.82.033817.
[31]	X. Zhu and P. Milanfar, Image reconstruction from videos distorted by atmospheric turbulence, In "SPIE Electronic Imaging, Conference on Visual Information Processing and Communication," 2010. doi: 10.1117/12.840127.
[32]	X. Zhu and P. Milanfar, Stabilizing and deblurring atmospheric turbulence, In "IEEE Int. Conf. on Computational Photography (ICCP)," pages 1-8, (2011). doi: 10.1109/ICCPHOT.2011.5753122.
[33]	X. Zhu and P. Milanfar, Removing atmospheric turbulence via space-invariant deconvolution, IEEE Trans. on Pattern Analysis and Machine Intelligence, 35 (2012), 157-170.

show all references

References:

[1]	L. Alvarez and L. Mazorra, Signal and image restoration using shock filters and anisotropic diffusion, SIAM Journal on Numerical Analysis, 31 (1994), 590-065. doi: 10.1137/0731032.
[2]	M. Aubailly, M. A. Vorontsov, G. W. Carhat and M. T. Valley, Automated video enhancement from a stream of atmospherically-distorted images: The lucky-region fusion approach, In "Proceedings of SPIE," volume 7463, (2009). doi: 10.1117/12.828332.
[3]	A. Buades, B. Coll and J. M. Morel, A review of image denoising algorithms, with a new one, Multiscale Modeling and Simulation, 4 (2005), 490-530. doi: 10.1137/040616024.
[4]	A. Buades, B. Coll and J.M. Morel, Nonlocal image and movie denoising, International Journal of Computer Vision, 76 (2008), 123-139.
[5]	K. Buskila, S. Towito, E. Shmuel, R. Levi, N. Kopeika, K. Krapels, R. Driggers, R. Vollmerhausen and C. Halford, Atmospheric modulation transfer function in the infrared, Applied Optics, 43 (2004), 471-482. doi: 10.1364/AO.43.000471.
[6]	J. Calder, A. Mansouri and A. Yezzi, Image sharpening via sobolev gradient flows, SIAM Journal on Imaging Sciences, 3 (2010), 981-1014. doi: 10.1137/090771260.
[7]	J. Caviedes and S. Gurbuz, No-reference sharpness metric based on local edge kurtosis, In "IEEE Int. Conf. on Image Processing," 3 (2002), 53-56. doi: 10.1109/ICIP.2002.1038901.
[8]	L. C. Evans, "Partial Differential Equations," Graduate Studies in Mathematics, 19. American Mathematical Society, Providence, RI, 1998.
[9]	D. Frakes, J. Monaco and M. Smith, Suppression of atmospheric turbulence in video using an adaptive control grid interpolation approach, In "IEEE Int. Conf. on Acoustics, Speech and Signal processing (ICASSP)," 3 (2001), 1881-1884. doi: 10.1109/ICASSP.2001.941311.
[10]	D. L. Fried, Optical resolution through a randomly inhomogeneous medium for very long and very short exposures, Journal of the Optical Society of America, 56 (1966), 1372-1379. doi: 10.1364/JOSA.56.001372.
[11]	S. Gepshtein, A. Shtainman, B. Fishbain and L. P. Yaroslavsky, Restoration of atmospheric turbulent video containing real motion using rank filtering and elastic image registration, In "Proceeding of the Eusipco," 2004.
[12]	J. Gilles and S. Osher, "Fried Deconvolution," UCLA CAM Report 11-62, 2011. doi: 10.1117/12.917234.
[13]	P. Hartman, "Ordinary Differential Equations," Corrected reprint. S. M. Hartman, Baltimore, Md., 1973.
[14]	M. Hirsch, S. Sra, B. Scholkopf and S. Harmeling, Efficient filter flow for space-variant multiframe blind deconvolution, IEEE Computer Vision and Pattern Recognition (CVPR), pages 607-614, (2010). doi: 10.1109/CVPR.2010.5540158.
[15]	R. E. Hufnagel and N. R. Stanley, Modulation transfer function associated with image transmission through turbulence media, Journal of the Optical Society of America, 54 (1964), 52-60. doi: 10.1364/JOSA.54.000052.
[16]	J.Gilles, T. Dagobert and C. De Franchis, Atmospheric turbulence restoration by diffeomorphism image registration and blind deconvolution, In "Advanced Concepts for Intelligent Vision Systems" (ACIVS), Oct 2008.
[17]	D. Li, R. Mersereau and S. Simske, Atmospheric turbulence degraded image restoration using principal components analysis, IEEE Geoscience and Remote Sensing Letters, 4 (2007), 340-344. doi: 10.1109/LGRS.2007.895691.
[18]	D. Li and S. Simske, Atmospheric turbulence degraded-image restoration by kurtosis minimization, IEEE Geoscience and Remote Sensing Letters, 6 (2009), 244-247.
[19]	Y. Mao and J. Gilles, Non rigid geometric distortions correction - application to atmospheric turbulence stabilization, Inverse Problems and Imaging, 6 (2012), 531-546. doi: 10.3934/ipi.2012.6.531.
[20]	A. Marquina, Nonlinear inverse scale space methods for total variation blind deconvolution, SIAM Journal on Imaging Sciences, 2 (2009), 64-83. doi: 10.1137/080724289.
[21]	M. Micheli, Y. Lou, S. Soatto and Andrea L. Bertozzi, A linear systems approach to imaging through turbulence, Journal of Mathematical Imaging and Vision July 2013. doi: 10.1007/s10851-012-0410-7.
[22]	A. V. Oppenheim and R. W. Schafer, "Discrete-Time Signal Processing," 2nd Edition Prentice Hall, Upper Saddle River, NJ, 1999.
[23]	P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Trans. on Pattern Analysis and Machine Intelligence, 12 (1990), 629-639. doi: 10.1109/34.56205.
[24]	J. Ricklin and F. Davidson, Atmospheric turbulence effects on a partially coherent gaussian beam: implications for free-space laser communication, Journal of the Optical Society of America A, pages 1794-1802, (2002). doi: 10.1364/JOSAA.19.001794.
[25]	M. Roggemann and B. Welsh, "Imaging Through Turbulence," CRC Press, Boca Raton, FL, 1996. doi: 10.1117/1.601043.
[26]	M. Shimizu, S. Yoshimura, M. Tanaka and M. Okutomi, Super-resolution from image sequence under influence of hot-air optical turbulence, In "IEEE Conference on Computer Vision and Pattern Recognition" (CVPR), pages 1-8, (2008).
[27]	G. Sundaramoorthi, A. Yezzi and A. C. Mennucci, Sobolev active contours, International Journal of Computer Vision, 73 (2007), 345-366.
[28]	D. Tofsted, Reanalysis of turbulence effects on short-exposure passive imaging, Optical Engineering, 50 (2011), 016001. doi: 10.1117/1.3532999.
[29]	M. A. Vorontsov and G. W. Carhart, Anisoplanatic imaging through turbulent media: Image recovery by local information fusion from a set of short-exposure images, Journal of the Optical Society of America A, 18 (2001), 1312-1324. doi: 10.1364/JOSAA.18.001312.
[30]	P. Zhang, W. Gong, X. Shen and S. Han, Correlated imaging through atmospheric turbulence, Physical Review A, 82 (2010), 033817. doi: 10.1103/PhysRevA.82.033817.
[31]	X. Zhu and P. Milanfar, Image reconstruction from videos distorted by atmospheric turbulence, In "SPIE Electronic Imaging, Conference on Visual Information Processing and Communication," 2010. doi: 10.1117/12.840127.
[32]	X. Zhu and P. Milanfar, Stabilizing and deblurring atmospheric turbulence, In "IEEE Int. Conf. on Computational Photography (ICCP)," pages 1-8, (2011). doi: 10.1109/ICCPHOT.2011.5753122.
[33]	X. Zhu and P. Milanfar, Removing atmospheric turbulence via space-invariant deconvolution, IEEE Trans. on Pattern Analysis and Machine Intelligence, 35 (2012), 157-170.

[1]	Zhiguang Zhang, Qiang Liu, Tianling Gao. A fast explicit diffusion algorithm of fractional order anisotropic diffusion for image denoising. Inverse Problems and Imaging, 2021, 15 (6) : 1451-1469. doi: 10.3934/ipi.2021018
[2]	Ying Zhang, Xuhua Ren, Bryan Alexander Clifford, Qian Wang, Xiaoqun Zhang. Image fusion network for dual-modal restoration. Inverse Problems and Imaging, 2021, 15 (6) : 1409-1419. doi: 10.3934/ipi.2021067
[3]	Ying Wen, Jiebao Sun, Zhichang Guo. A new anisotropic fourth-order diffusion equation model based on image features for image denoising. Inverse Problems and Imaging, , () : -. doi: 10.3934/ipi.2022004
[4]	Kaitlyn (Voccola) Muller. SAR correlation imaging and anisotropic scattering. Inverse Problems and Imaging, 2018, 12 (3) : 697-731. doi: 10.3934/ipi.2018030
[5]	Hong Seng Sim, Wah June Leong, Chuei Yee Chen, Siti Nur Iqmal Ibrahim. Multi-step spectral gradient methods with modified weak secant relation for large scale unconstrained optimization. Numerical Algebra, Control and Optimization, 2018, 8 (3) : 377-387. doi: 10.3934/naco.2018024
[6]	Zhong Wan, Chaoming Hu, Zhanlu Yang. A spectral PRP conjugate gradient methods for nonconvex optimization problem based on modified line search. Discrete and Continuous Dynamical Systems - B, 2011, 16 (4) : 1157-1169. doi: 10.3934/dcdsb.2011.16.1157
[7]	Jiangchuan Fan, Xinyu Guo, Jianjun Du, Weiliang Wen, Xianju Lu, Brahmani Louiza. Analysis of the clustering fusion algorithm for multi-band color image. Discrete and Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1233-1249. doi: 10.3934/dcdss.2019085
[8]	Moulay Rchid Sidi Ammi, Ismail Jamiai. Finite difference and Legendre spectral method for a time-fractional diffusion-convection equation for image restoration. Discrete and Continuous Dynamical Systems - S, 2018, 11 (1) : 103-117. doi: 10.3934/dcdss.2018007
[9]	Nicolas Lermé, François Malgouyres, Dominique Hamoir, Emmanuelle Thouin. Bayesian image restoration for mosaic active imaging. Inverse Problems and Imaging, 2014, 8 (3) : 733-760. doi: 10.3934/ipi.2014.8.733
[10]	Yong Zheng Ong, Haizhao Yang. Generative imaging and image processing via generative encoder. Inverse Problems and Imaging, 2022, 16 (3) : 525-545. doi: 10.3934/ipi.2021060
[11]	Yuhong Dai, Ya-xiang Yuan. Analysis of monotone gradient methods. Journal of Industrial and Management Optimization, 2005, 1 (2) : 181-192. doi: 10.3934/jimo.2005.1.181
[12]	Shenglong Hu, Zheng-Hai Huang, Hong-Yan Ni, Liqun Qi. Positive definiteness of Diffusion Kurtosis Imaging. Inverse Problems and Imaging, 2012, 6 (1) : 57-75. doi: 10.3934/ipi.2012.6.57
[13]	Antoni Buades, Bartomeu Coll, Jose-Luis Lisani, Catalina Sbert. Conditional image diffusion. Inverse Problems and Imaging, 2007, 1 (4) : 593-608. doi: 10.3934/ipi.2007.1.593
[14]	Jianjun Zhang, Yunyi Hu, James G. Nagy. A scaled gradient method for digital tomographic image reconstruction. Inverse Problems and Imaging, 2018, 12 (1) : 239-259. doi: 10.3934/ipi.2018010
[15]	Wanyou Cheng, Zixin Chen, Donghui Li. Nomonotone spectral gradient method for sparse recovery. Inverse Problems and Imaging, 2015, 9 (3) : 815-833. doi: 10.3934/ipi.2015.9.815
[16]	Timothy Blass, Rafael De La Llave, Enrico Valdinoci. A comparison principle for a Sobolev gradient semi-flow. Communications on Pure and Applied Analysis, 2011, 10 (1) : 69-91. doi: 10.3934/cpaa.2011.10.69
[17]	Takeshi Saito, Kazuyuki Yagasaki. Chebyshev spectral methods for computing center manifolds. Journal of Computational Dynamics, 2021, 8 (2) : 165-181. doi: 10.3934/jcd.2021008
[18]	Robert I. McLachlan, G. R. W. Quispel. Discrete gradient methods have an energy conservation law. Discrete and Continuous Dynamical Systems, 2014, 34 (3) : 1099-1104. doi: 10.3934/dcds.2014.34.1099
[19]	Predrag S. Stanimirović, Branislav Ivanov, Haifeng Ma, Dijana Mosić. A survey of gradient methods for solving nonlinear optimization. Electronic Research Archive, 2020, 28 (4) : 1573-1624. doi: 10.3934/era.2020115
[20]	Giacomo Frassoldati, Luca Zanni, Gaetano Zanghirati. New adaptive stepsize selections in gradient methods. Journal of Industrial and Management Optimization, 2008, 4 (2) : 299-312. doi: 10.3934/jimo.2008.4.299

2020 Impact Factor: 1.639

Tools

Metrics

Other articles
by authors

on AIMS
Yifei Lou Sung Ha Kang Stefano Soatto Andrea L. Bertozzi
on Google Scholar
Yifei Lou Sung Ha Kang Stefano Soatto Andrea L. Bertozzi

[Back to Top]