American Institute of Mathematical Sciences

Advanced
August  2013, 7(3): 839-861. doi: 10.3934/ipi.2013.7.839

Video stabilization of atmospheric turbulence distortion

Yifei Lou 1, , Sung Ha Kang 2, , Stefano Soatto 3, and  Andrea L. Bertozzi 1,

1. 

Department of Mathematics, University of California Los Angeles, Los Angeles, CA, 90095, United States, United States
2. 

School of Mathematics, Georgia Institute of Technology, Atlanta, GA 30332-0160
3. 

Computer Science Department, University of California Los Angeles, Los Angeles, CA, 90095, United States

Received  April 2012 Revised  March 2013 Published  September 2013

We present a method to enhance the quality of a video sequence captured through a turbulent atmospheric medium, and give an estimate of the radiance of the distant scene, represented as a ``latent image,'' which is assumed to be static throughout the video. Due to atmospheric turbulence, temporal averaging produces a blurred version of the scene's radiance. We propose a method combining Sobolev gradient and Laplacian to stabilize the video sequence, and a latent image is further found utilizing the ``lucky region" method. The video sequence is stabilized while keeping sharp details, and the latent image shows more consistent straight edges. We analyze the well-posedness for the stabilizing PDE and the linear stability of the numerical scheme.
Keywords: spectral methods, Imaging through turbulence, anisotropic diffusion, Sobolev gradient sharpening, image fusion..
Mathematics Subject Classification: Primary: 68U10; Secondary: 35B65, 35B3.
Citation: Yifei Lou, Sung Ha Kang, Stefano Soatto, Andrea L. Bertozzi. Video stabilization of atmospheric turbulence distortion. Inverse Problems & Imaging, 2013, 7 (3) : 839-861. doi: 10.3934/ipi.2013.7.839
References:
[1]

L. Alvarez and L. Mazorra, Signal and image restoration using shock filters and anisotropic diffusion, SIAM Journal on Numerical Analysis, 31 (1994), 590-065. doi: 10.1137/0731032.  Google Scholar

[2]

M. Aubailly, M. A. Vorontsov, G. W. Carhat and M. T. Valley, Automated video enhancement from a stream of atmospherically-distorted images: The lucky-region fusion approach, In "Proceedings of SPIE," volume 7463, (2009). doi: 10.1117/12.828332.  Google Scholar

[3]

A. Buades, B. Coll and J. M. Morel, A review of image denoising algorithms, with a new one, Multiscale Modeling and Simulation, 4 (2005), 490-530. doi: 10.1137/040616024.  Google Scholar

[4]

A. Buades, B. Coll and J.M. Morel, Nonlocal image and movie denoising, International Journal of Computer Vision, 76 (2008), 123-139. Google Scholar

[5]

K. Buskila, S. Towito, E. Shmuel, R. Levi, N. Kopeika, K. Krapels, R. Driggers, R. Vollmerhausen and C. Halford, Atmospheric modulation transfer function in the infrared, Applied Optics, 43 (2004), 471-482. doi: 10.1364/AO.43.000471.  Google Scholar

[6]

J. Calder, A. Mansouri and A. Yezzi, Image sharpening via sobolev gradient flows, SIAM Journal on Imaging Sciences, 3 (2010), 981-1014. doi: 10.1137/090771260.  Google Scholar

[7]

J. Caviedes and S. Gurbuz, No-reference sharpness metric based on local edge kurtosis, In "IEEE Int. Conf. on Image Processing," 3 (2002), 53-56. doi: 10.1109/ICIP.2002.1038901.  Google Scholar

[8]

L. C. Evans, "Partial Differential Equations," Graduate Studies in Mathematics, 19. American Mathematical Society, Providence, RI, 1998.  Google Scholar

[9]

D. Frakes, J. Monaco and M. Smith, Suppression of atmospheric turbulence in video using an adaptive control grid interpolation approach, In "IEEE Int. Conf. on Acoustics, Speech and Signal processing (ICASSP)," 3 (2001), 1881-1884. doi: 10.1109/ICASSP.2001.941311.  Google Scholar

[10]

D. L. Fried, Optical resolution through a randomly inhomogeneous medium for very long and very short exposures, Journal of the Optical Society of America, 56 (1966), 1372-1379. doi: 10.1364/JOSA.56.001372.  Google Scholar

[11]

S. Gepshtein, A. Shtainman, B. Fishbain and L. P. Yaroslavsky, Restoration of atmospheric turbulent video containing real motion using rank filtering and elastic image registration, In "Proceeding of the Eusipco," 2004. Google Scholar

[12]

J. Gilles and S. Osher, "Fried Deconvolution," UCLA CAM Report 11-62, 2011. doi: 10.1117/12.917234.  Google Scholar

[13]

P. Hartman, "Ordinary Differential Equations," Corrected reprint. S. M. Hartman, Baltimore, Md., 1973.  Google Scholar

[14]

M. Hirsch, S. Sra, B. Scholkopf and S. Harmeling, Efficient filter flow for space-variant multiframe blind deconvolution, IEEE Computer Vision and Pattern Recognition (CVPR), pages 607-614, (2010). doi: 10.1109/CVPR.2010.5540158.  Google Scholar

[15]

R. E. Hufnagel and N. R. Stanley, Modulation transfer function associated with image transmission through turbulence media, Journal of the Optical Society of America, 54 (1964), 52-60. doi: 10.1364/JOSA.54.000052.  Google Scholar

[16]

J.Gilles, T. Dagobert and C. De Franchis, Atmospheric turbulence restoration by diffeomorphism image registration and blind deconvolution, In "Advanced Concepts for Intelligent Vision Systems" (ACIVS), Oct 2008. Google Scholar

[17]

D. Li, R. Mersereau and S. Simske, Atmospheric turbulence degraded image restoration using principal components analysis, IEEE Geoscience and Remote Sensing Letters, 4 (2007), 340-344. doi: 10.1109/LGRS.2007.895691.  Google Scholar

[18]

D. Li and S. Simske, Atmospheric turbulence degraded-image restoration by kurtosis minimization, IEEE Geoscience and Remote Sensing Letters, 6 (2009), 244-247. Google Scholar

[19]

Y. Mao and J. Gilles, Non rigid geometric distortions correction - application to atmospheric turbulence stabilization, Inverse Problems and Imaging, 6 (2012), 531-546. doi: 10.3934/ipi.2012.6.531.  Google Scholar

[20]

A. Marquina, Nonlinear inverse scale space methods for total variation blind deconvolution, SIAM Journal on Imaging Sciences, 2 (2009), 64-83. doi: 10.1137/080724289.  Google Scholar

[21]

M. Micheli, Y. Lou, S. Soatto and Andrea L. Bertozzi, A linear systems approach to imaging through turbulence, Journal of Mathematical Imaging and Vision July 2013. doi: 10.1007/s10851-012-0410-7.  Google Scholar

[22]

A. V. Oppenheim and R. W. Schafer, "Discrete-Time Signal Processing," 2nd Edition Prentice Hall, Upper Saddle River, NJ, 1999. Google Scholar

[23]

P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Trans. on Pattern Analysis and Machine Intelligence, 12 (1990), 629-639. doi: 10.1109/34.56205.  Google Scholar

[24]

J. Ricklin and F. Davidson, Atmospheric turbulence effects on a partially coherent gaussian beam: implications for free-space laser communication, Journal of the Optical Society of America A, pages 1794-1802, (2002). doi: 10.1364/JOSAA.19.001794.  Google Scholar

[25]

M. Roggemann and B. Welsh, "Imaging Through Turbulence," CRC Press, Boca Raton, FL, 1996. doi: 10.1117/1.601043.  Google Scholar

[26]

M. Shimizu, S. Yoshimura, M. Tanaka and M. Okutomi, Super-resolution from image sequence under influence of hot-air optical turbulence, In "IEEE Conference on Computer Vision and Pattern Recognition" (CVPR), pages 1-8, (2008). Google Scholar

[27]

G. Sundaramoorthi, A. Yezzi and A. C. Mennucci, Sobolev active contours, International Journal of Computer Vision, 73 (2007), 345-366. Google Scholar

[28]

D. Tofsted, Reanalysis of turbulence effects on short-exposure passive imaging, Optical Engineering, 50 (2011), 016001. doi: 10.1117/1.3532999.  Google Scholar

[29]

M. A. Vorontsov and G. W. Carhart, Anisoplanatic imaging through turbulent media: Image recovery by local information fusion from a set of short-exposure images, Journal of the Optical Society of America A, 18 (2001), 1312-1324. doi: 10.1364/JOSAA.18.001312.  Google Scholar

[30]

P. Zhang, W. Gong, X. Shen and S. Han, Correlated imaging through atmospheric turbulence, Physical Review A, 82 (2010), 033817. doi: 10.1103/PhysRevA.82.033817.  Google Scholar

[31]

X. Zhu and P. Milanfar, Image reconstruction from videos distorted by atmospheric turbulence, In "SPIE Electronic Imaging, Conference on Visual Information Processing and Communication," 2010. doi: 10.1117/12.840127.  Google Scholar

[32]

X. Zhu and P. Milanfar, Stabilizing and deblurring atmospheric turbulence, In "IEEE Int. Conf. on Computational Photography (ICCP)," pages 1-8, (2011). doi: 10.1109/ICCPHOT.2011.5753122.  Google Scholar

[33]

X. Zhu and P. Milanfar, Removing atmospheric turbulence via space-invariant deconvolution, IEEE Trans. on Pattern Analysis and Machine Intelligence, 35 (2012), 157-170. Google Scholar

show all references

References:
[1]

L. Alvarez and L. Mazorra, Signal and image restoration using shock filters and anisotropic diffusion, SIAM Journal on Numerical Analysis, 31 (1994), 590-065. doi: 10.1137/0731032.  Google Scholar

[2]

M. Aubailly, M. A. Vorontsov, G. W. Carhat and M. T. Valley, Automated video enhancement from a stream of atmospherically-distorted images: The lucky-region fusion approach, In "Proceedings of SPIE," volume 7463, (2009). doi: 10.1117/12.828332.  Google Scholar

[3]

A. Buades, B. Coll and J. M. Morel, A review of image denoising algorithms, with a new one, Multiscale Modeling and Simulation, 4 (2005), 490-530. doi: 10.1137/040616024.  Google Scholar

[4]

A. Buades, B. Coll and J.M. Morel, Nonlocal image and movie denoising, International Journal of Computer Vision, 76 (2008), 123-139. Google Scholar

[5]

K. Buskila, S. Towito, E. Shmuel, R. Levi, N. Kopeika, K. Krapels, R. Driggers, R. Vollmerhausen and C. Halford, Atmospheric modulation transfer function in the infrared, Applied Optics, 43 (2004), 471-482. doi: 10.1364/AO.43.000471.  Google Scholar

[6]

J. Calder, A. Mansouri and A. Yezzi, Image sharpening via sobolev gradient flows, SIAM Journal on Imaging Sciences, 3 (2010), 981-1014. doi: 10.1137/090771260.  Google Scholar

[7]

J. Caviedes and S. Gurbuz, No-reference sharpness metric based on local edge kurtosis, In "IEEE Int. Conf. on Image Processing," 3 (2002), 53-56. doi: 10.1109/ICIP.2002.1038901.  Google Scholar

[8]

L. C. Evans, "Partial Differential Equations," Graduate Studies in Mathematics, 19. American Mathematical Society, Providence, RI, 1998.  Google Scholar

[9]

D. Frakes, J. Monaco and M. Smith, Suppression of atmospheric turbulence in video using an adaptive control grid interpolation approach, In "IEEE Int. Conf. on Acoustics, Speech and Signal processing (ICASSP)," 3 (2001), 1881-1884. doi: 10.1109/ICASSP.2001.941311.  Google Scholar

[10]

D. L. Fried, Optical resolution through a randomly inhomogeneous medium for very long and very short exposures, Journal of the Optical Society of America, 56 (1966), 1372-1379. doi: 10.1364/JOSA.56.001372.  Google Scholar

[11]

S. Gepshtein, A. Shtainman, B. Fishbain and L. P. Yaroslavsky, Restoration of atmospheric turbulent video containing real motion using rank filtering and elastic image registration, In "Proceeding of the Eusipco," 2004. Google Scholar

[12]

J. Gilles and S. Osher, "Fried Deconvolution," UCLA CAM Report 11-62, 2011. doi: 10.1117/12.917234.  Google Scholar

[13]

P. Hartman, "Ordinary Differential Equations," Corrected reprint. S. M. Hartman, Baltimore, Md., 1973.  Google Scholar

[14]

M. Hirsch, S. Sra, B. Scholkopf and S. Harmeling, Efficient filter flow for space-variant multiframe blind deconvolution, IEEE Computer Vision and Pattern Recognition (CVPR), pages 607-614, (2010). doi: 10.1109/CVPR.2010.5540158.  Google Scholar

[15]

R. E. Hufnagel and N. R. Stanley, Modulation transfer function associated with image transmission through turbulence media, Journal of the Optical Society of America, 54 (1964), 52-60. doi: 10.1364/JOSA.54.000052.  Google Scholar

[16]

J.Gilles, T. Dagobert and C. De Franchis, Atmospheric turbulence restoration by diffeomorphism image registration and blind deconvolution, In "Advanced Concepts for Intelligent Vision Systems" (ACIVS), Oct 2008. Google Scholar

[17]

D. Li, R. Mersereau and S. Simske, Atmospheric turbulence degraded image restoration using principal components analysis, IEEE Geoscience and Remote Sensing Letters, 4 (2007), 340-344. doi: 10.1109/LGRS.2007.895691.  Google Scholar

[18]

D. Li and S. Simske, Atmospheric turbulence degraded-image restoration by kurtosis minimization, IEEE Geoscience and Remote Sensing Letters, 6 (2009), 244-247. Google Scholar

[19]

Y. Mao and J. Gilles, Non rigid geometric distortions correction - application to atmospheric turbulence stabilization, Inverse Problems and Imaging, 6 (2012), 531-546. doi: 10.3934/ipi.2012.6.531.  Google Scholar

[20]

A. Marquina, Nonlinear inverse scale space methods for total variation blind deconvolution, SIAM Journal on Imaging Sciences, 2 (2009), 64-83. doi: 10.1137/080724289.  Google Scholar

[21]

M. Micheli, Y. Lou, S. Soatto and Andrea L. Bertozzi, A linear systems approach to imaging through turbulence, Journal of Mathematical Imaging and Vision July 2013. doi: 10.1007/s10851-012-0410-7.  Google Scholar

[22]

A. V. Oppenheim and R. W. Schafer, "Discrete-Time Signal Processing," 2nd Edition Prentice Hall, Upper Saddle River, NJ, 1999. Google Scholar

[23]

P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Trans. on Pattern Analysis and Machine Intelligence, 12 (1990), 629-639. doi: 10.1109/34.56205.  Google Scholar

[24]

J. Ricklin and F. Davidson, Atmospheric turbulence effects on a partially coherent gaussian beam: implications for free-space laser communication, Journal of the Optical Society of America A, pages 1794-1802, (2002). doi: 10.1364/JOSAA.19.001794.  Google Scholar

[25]

M. Roggemann and B. Welsh, "Imaging Through Turbulence," CRC Press, Boca Raton, FL, 1996. doi: 10.1117/1.601043.  Google Scholar

[26]

M. Shimizu, S. Yoshimura, M. Tanaka and M. Okutomi, Super-resolution from image sequence under influence of hot-air optical turbulence, In "IEEE Conference on Computer Vision and Pattern Recognition" (CVPR), pages 1-8, (2008). Google Scholar

[27]

G. Sundaramoorthi, A. Yezzi and A. C. Mennucci, Sobolev active contours, International Journal of Computer Vision, 73 (2007), 345-366. Google Scholar

[28]

D. Tofsted, Reanalysis of turbulence effects on short-exposure passive imaging, Optical Engineering, 50 (2011), 016001. doi: 10.1117/1.3532999.  Google Scholar

[29]

M. A. Vorontsov and G. W. Carhart, Anisoplanatic imaging through turbulent media: Image recovery by local information fusion from a set of short-exposure images, Journal of the Optical Society of America A, 18 (2001), 1312-1324. doi: 10.1364/JOSAA.18.001312.  Google Scholar

[30]

P. Zhang, W. Gong, X. Shen and S. Han, Correlated imaging through atmospheric turbulence, Physical Review A, 82 (2010), 033817. doi: 10.1103/PhysRevA.82.033817.  Google Scholar

[31]

X. Zhu and P. Milanfar, Image reconstruction from videos distorted by atmospheric turbulence, In "SPIE Electronic Imaging, Conference on Visual Information Processing and Communication," 2010. doi: 10.1117/12.840127.  Google Scholar

[32]

X. Zhu and P. Milanfar, Stabilizing and deblurring atmospheric turbulence, In "IEEE Int. Conf. on Computational Photography (ICCP)," pages 1-8, (2011). doi: 10.1109/ICCPHOT.2011.5753122.  Google Scholar

[33]

X. Zhu and P. Milanfar, Removing atmospheric turbulence via space-invariant deconvolution, IEEE Trans. on Pattern Analysis and Machine Intelligence, 35 (2012), 157-170. Google Scholar

[1]

Zhiguang Zhang, Qiang Liu, Tianling Gao. A fast explicit diffusion algorithm of fractional order anisotropic diffusion for image denoising. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021018
[2]

Kaitlyn (Voccola) Muller. SAR correlation imaging and anisotropic scattering. Inverse Problems & Imaging, 2018, 12 (3) : 697-731. doi: 10.3934/ipi.2018030
[3]

Hong Seng Sim, Wah June Leong, Chuei Yee Chen, Siti Nur Iqmal Ibrahim. Multi-step spectral gradient methods with modified weak secant relation for large scale unconstrained optimization. Numerical Algebra, Control & Optimization, 2018, 8 (3) : 377-387. doi: 10.3934/naco.2018024
[4]

Zhong Wan, Chaoming Hu, Zhanlu Yang. A spectral PRP conjugate gradient methods for nonconvex optimization problem based on modified line search. Discrete & Continuous Dynamical Systems - B, 2011, 16 (4) : 1157-1169. doi: 10.3934/dcdsb.2011.16.1157
[5]

Jiangchuan Fan, Xinyu Guo, Jianjun Du, Weiliang Wen, Xianju Lu, Brahmani Louiza. Analysis of the clustering fusion algorithm for multi-band color image. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1233-1249. doi: 10.3934/dcdss.2019085
[6]

Moulay Rchid Sidi Ammi, Ismail Jamiai. Finite difference and Legendre spectral method for a time-fractional diffusion-convection equation for image restoration. Discrete & Continuous Dynamical Systems - S, 2018, 11 (1) : 103-117. doi: 10.3934/dcdss.2018007
[7]

Nicolas Lermé, François Malgouyres, Dominique Hamoir, Emmanuelle Thouin. Bayesian image restoration for mosaic active imaging. Inverse Problems & Imaging, 2014, 8 (3) : 733-760. doi: 10.3934/ipi.2014.8.733
[8]

Yong Zheng Ong, Haizhao Yang. Generative imaging and image processing via generative encoder. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021060
[9]

Yuhong Dai, Ya-xiang Yuan. Analysis of monotone gradient methods. Journal of Industrial & Management Optimization, 2005, 1 (2) : 181-192. doi: 10.3934/jimo.2005.1.181
[10]

Shenglong Hu, Zheng-Hai Huang, Hong-Yan Ni, Liqun Qi. Positive definiteness of Diffusion Kurtosis Imaging. Inverse Problems & Imaging, 2012, 6 (1) : 57-75. doi: 10.3934/ipi.2012.6.57
[11]

Antoni Buades, Bartomeu Coll, Jose-Luis Lisani, Catalina Sbert. Conditional image diffusion. Inverse Problems & Imaging, 2007, 1 (4) : 593-608. doi: 10.3934/ipi.2007.1.593
[12]

Jianjun Zhang, Yunyi Hu, James G. Nagy. A scaled gradient method for digital tomographic image reconstruction. Inverse Problems & Imaging, 2018, 12 (1) : 239-259. doi: 10.3934/ipi.2018010
[13]

Timothy Blass, Rafael De La Llave, Enrico Valdinoci. A comparison principle for a Sobolev gradient semi-flow. Communications on Pure & Applied Analysis, 2011, 10 (1) : 69-91. doi: 10.3934/cpaa.2011.10.69
[14]

Wanyou Cheng, Zixin Chen, Donghui Li. Nomonotone spectral gradient method for sparse recovery. Inverse Problems & Imaging, 2015, 9 (3) : 815-833. doi: 10.3934/ipi.2015.9.815
[15]

Takeshi Saito, Kazuyuki Yagasaki. Chebyshev spectral methods for computing center manifolds. Journal of Computational Dynamics, 2021, 8 (2) : 165-181. doi: 10.3934/jcd.2021008
[16]

Robert I. McLachlan, G. R. W. Quispel. Discrete gradient methods have an energy conservation law. Discrete & Continuous Dynamical Systems, 2014, 34 (3) : 1099-1104. doi: 10.3934/dcds.2014.34.1099
[17]

Predrag S. Stanimirović, Branislav Ivanov, Haifeng Ma, Dijana Mosić. A survey of gradient methods for solving nonlinear optimization. Electronic Research Archive, 2020, 28 (4) : 1573-1624. doi: 10.3934/era.2020115
[18]

Giacomo Frassoldati, Luca Zanni, Gaetano Zanghirati. New adaptive stepsize selections in gradient methods. Journal of Industrial & Management Optimization, 2008, 4 (2) : 299-312. doi: 10.3934/jimo.2008.4.299
[19]

Richard A. Norton, David I. McLaren, G. R. W. Quispel, Ari Stern, Antonella Zanna. Projection methods and discrete gradient methods for preserving first integrals of ODEs. Discrete & Continuous Dynamical Systems, 2015, 35 (5) : 2079-2098. doi: 10.3934/dcds.2015.35.2079
[20]

Jianqing Chen. Best constant of 3D Anisotropic Sobolev inequality and its applications. Communications on Pure & Applied Analysis, 2010, 9 (3) : 655-666. doi: 10.3934/cpaa.2010.9.655

2020 Impact Factor: 1.639

Tools

Metrics

  • PDF downloads (257)
  • HTML views (0)
  • Cited by (14)

Other articles
by authors

[Back to Top]

Copyright © 2021 American Institute of Mathematical Sciences