doi: 10.3934/ipi.2021016

Inbetweening auto-animation via Fokker-Planck dynamics and thresholding

1. 

Department of Mathematics, Duke University, Durham, NC 27708, USA

2. 

College of Marine Ecology and Environment, Shanghai Ocean University, China

3. 

Southern marine science and engineering Guangdong laboratory, Zhuhai, China

4. 

Department of Mathematics and Department of Physics, Duke University, Durham, NC 27708, USA

* Corresponding author: Guangzhen Jin

Received  May 2020 Revised  October 2020 Published  February 2021

Fund Project: J.-G. Liu was supported in part by the National Science Foundation (NSF) under award DMS-1812573. G. Jin was supported in part by the the Natural Science Foundation of Guangdong Province under award 2019A1515011487, the China Postdoctoral Science Foundation under award 2020M681269 and the Fundamental Research Funds for the Central Universities under award 20184200031610059

We propose an equilibrium-driven deformation algorithm (EDDA) to simulate the inbetweening transformations starting from an initial image to an equilibrium image, which covers images varying from a greyscale type to a colorful type on planes or manifolds. The algorithm is based on the Fokker-Planck dynamics on manifold, which automatically incorporates the manifold structure suggested by dataset and satisfies positivity, unconditional stability, mass conservation law and exponentially convergence. The thresholding scheme is adapted for the sharp interface dynamics and is used to achieve the finite time convergence. Using EDDA, three challenging examples, (I) facial aging process, (II) coronavirus disease 2019 (COVID-19) pneumonia invading/fading process, and (III) continental evolution process are computed efficiently.

Citation: Yuan Gao, Guangzhen Jin, Jian-Guo Liu. Inbetweening auto-animation via Fokker-Planck dynamics and thresholding. Inverse Problems & Imaging, doi: 10.3934/ipi.2021016
References:
[1]

M. Cong, M. Bao, Jane L. E., K. S. Bhat and R. Fedkiw, Fully automatic generation of anatomical face simulation models, In Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, (2015), 175–183. doi: 10.1145/2786784.2786786.  Google Scholar

[2]

Q. DuM. D. Gunzburger and L. Ju, Voronoi-based finite volume methods, optimal voronoi meshes, and pdes on the sphere, Computer Methods in Applied Mechanics and Engineering, 192 (2003), 3933-3957.  doi: 10.1016/S0045-7825(03)00394-3.  Google Scholar

[3]

R. Eymard, T. Gallouët and R. Herbin, Finite volume methods, Handbook of numerical analysis, Vol. VII, Handb. Numer. Anal., VII, North-Holland, Amsterdam, 2000, 713–1020. doi: 10.1086/phos.67.4.188705.  Google Scholar

[4]

Y. Gao, T. Li, X. Li and J.-G. Liu, Transition path theory for langevin dynamics on manifold: optimal control and data-driven solver, arXiv: 2010.09988, 2020. Google Scholar

[5]

Y. Gao, J.-G. Liu and N. Wu, Data-driven efficient solvers and predictions of conformational transitions for langevin dynamics on manifold in high dimensions, arXiv: 2005.12787, 2020. Google Scholar

[6]

Z. Leng, R. Zhu, W. Hou, Y. Feng, Y. Yang, Q. Han, G. Shan, F. Meng, D. Du, S. Wang, J. Fan, W. Wang, L. Deng, H. Shi, H.Li, Z. Hu, F. Zhang, J. Gao, H. Liu, X. Li, Y. Zhao, K. Yin, X. He, Z. Gao, Y. Wang, B. Yang, R. Jin, I. Stambler, L. W. Lim, H. Su, A. Moskalev, A. Cano, S. Chakrabarti, K.-. Min, G. Ellison-Hughes, C. Caruso, K. Jin and R. C. Zhao, Transplantation of ACE2-mesenchymal stem cells improves the outcome of patients with COVID-19 pneumonia, Aging and disease, 11 (2020), 216. Google Scholar

[7]

The Sydney Morning Herald, July 4, 2014, 'Can your face reveal how long you'll live?', http://www.dailylife.com.au/health-and-fitness/dl-fitness/can-your-face-reveal-how-long-youll-live-20140708-3bjsk.html. Google Scholar

[8]

R. J. Renka, Algorithm 772: Stripack: Delaunay triangulation and voronoi diagram on the surface of a sphere, ACM Transactions on Mathematical Software (TOMS), 23 (1997), 416-434.  doi: 10.1145/275323.275329.  Google Scholar

[9]

M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner and C. Theobalt, State of the art on monocular 3d face reconstruction, tracking, and applications, In Computer Graphics Forum, volume 37. Wiley Online Library, 2018, 523–550. Google Scholar

show all references

References:
[1]

M. Cong, M. Bao, Jane L. E., K. S. Bhat and R. Fedkiw, Fully automatic generation of anatomical face simulation models, In Proceedings of the 14th ACM SIGGRAPH/Eurographics Symposium on Computer Animation, (2015), 175–183. doi: 10.1145/2786784.2786786.  Google Scholar

[2]

Q. DuM. D. Gunzburger and L. Ju, Voronoi-based finite volume methods, optimal voronoi meshes, and pdes on the sphere, Computer Methods in Applied Mechanics and Engineering, 192 (2003), 3933-3957.  doi: 10.1016/S0045-7825(03)00394-3.  Google Scholar

[3]

R. Eymard, T. Gallouët and R. Herbin, Finite volume methods, Handbook of numerical analysis, Vol. VII, Handb. Numer. Anal., VII, North-Holland, Amsterdam, 2000, 713–1020. doi: 10.1086/phos.67.4.188705.  Google Scholar

[4]

Y. Gao, T. Li, X. Li and J.-G. Liu, Transition path theory for langevin dynamics on manifold: optimal control and data-driven solver, arXiv: 2010.09988, 2020. Google Scholar

[5]

Y. Gao, J.-G. Liu and N. Wu, Data-driven efficient solvers and predictions of conformational transitions for langevin dynamics on manifold in high dimensions, arXiv: 2005.12787, 2020. Google Scholar

[6]

Z. Leng, R. Zhu, W. Hou, Y. Feng, Y. Yang, Q. Han, G. Shan, F. Meng, D. Du, S. Wang, J. Fan, W. Wang, L. Deng, H. Shi, H.Li, Z. Hu, F. Zhang, J. Gao, H. Liu, X. Li, Y. Zhao, K. Yin, X. He, Z. Gao, Y. Wang, B. Yang, R. Jin, I. Stambler, L. W. Lim, H. Su, A. Moskalev, A. Cano, S. Chakrabarti, K.-. Min, G. Ellison-Hughes, C. Caruso, K. Jin and R. C. Zhao, Transplantation of ACE2-mesenchymal stem cells improves the outcome of patients with COVID-19 pneumonia, Aging and disease, 11 (2020), 216. Google Scholar

[7]

The Sydney Morning Herald, July 4, 2014, 'Can your face reveal how long you'll live?', http://www.dailylife.com.au/health-and-fitness/dl-fitness/can-your-face-reveal-how-long-youll-live-20140708-3bjsk.html. Google Scholar

[8]

R. J. Renka, Algorithm 772: Stripack: Delaunay triangulation and voronoi diagram on the surface of a sphere, ACM Transactions on Mathematical Software (TOMS), 23 (1997), 416-434.  doi: 10.1145/275323.275329.  Google Scholar

[9]

M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner and C. Theobalt, State of the art on monocular 3d face reconstruction, tracking, and applications, In Computer Graphics Forum, volume 37. Wiley Online Library, 2018, 523–550. Google Scholar

Figure 1.  The semilog plot of temporal evolution of relative root mean square errors for the RGB facial aging transformation with parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. The red, green and blue lines represent the relative errors of the corresponding color modes
Figure 2.  Facial aging transformation from initial to equilibrium with parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. The updated results after time step $ 40,100 $, $ 200 $, 400, $ 1000, 2000 $, $ 4000 $, $ 10000 $ are shown and compared to the initial and equilibrium images
6]. The left column of figures illustrates the evading of pneumonia from January 23th to February 2nd and the right column illustrates the fading away of pneumonia from February 2nd to February 15th after treatment. Red circles indicate the significant COVID-19 pneumonia invading areas and blue circles indicate the significant pneumonia fading away areas">Figure 3.  Chest CT images of the critically severe COVID-19 patient [6]. The left column of figures illustrates the evading of pneumonia from January 23th to February 2nd and the right column illustrates the fading away of pneumonia from February 2nd to February 15th after treatment. Red circles indicate the significant COVID-19 pneumonia invading areas and blue circles indicate the significant pneumonia fading away areas
Figure 4.  The semilog plot of temporal evolution of the relative root mean square errors for COVID-19 pneumonia invading and fading away on CT scan images with the parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. (up) The error evolution for the pneumonia invading process simulation. (down) The error evolution for the pneumonia fading away process simulation
Figure 5.  The simulated COVID-19 pneumonia invading and fading away process on CT scan images with the parameters $ \Delta t = 0.01, T = 100 $ and $ \Delta x = \Delta y = 10^{-4} $. Results after the step 20, 50, 100, 200, 500, 1000, 2000, 5000 are illustrated and compared to the initial and equilibrium scan images. The white part inside the lungs shown on images indicates the evidence of pneumonia. (up) The pneumonia invades into the patient's lungs caused by COVID-19. (down) The pneumonia fades away from the lungs after a stem cell treatment is applied to the patient. Red circles indicate the significant COVID-19 pneumonia invading areas and blue circles indicate the significant pneumonia fading away areas
Figure 6.  The unit sphere and the Voronoi cells on it. There are totally 3000 cells on the sphere. The black dots indicate locations of the point clouds on the sphere. The polygons with black edges are the Voronoi polygons generated with CVT algorithm
Figure 7.  The semilog plot of temporal evolution of relative root mean square errors for the continental evolution process with thresholding for shape dynamics with $ \Delta t = 0.05 $ and the total number of the thresholding adjustments is $ N_t = 50 $. The linear iterations before the $ (k+1) $th thresholding adjustment is $ 2k $. The red line indicates the error of simulations with only the linear Fokker-Planck algorithm while the blue line is the error of the linear algorithm combined with the thresholding scheme. The black dotted lines indicate the time steps when the thresholding adjustments are applied
Figure 8.  The evolutions of continental movements on the unit sphere with the parameter $ \Delta t = 0.05 $ and the total number of the thresholding adjustments is $ N_t = 50 $. The continental evolution on the sphere after the 2th, 5th, 10th, 30th thresholding adjustment are illustrated and compared with the initial and equilibrium states. The black dots and polygons in each subplot illustrate the point clouds and the Voronoi cells, respectively. The orange and blue patch indicate the land and ocean, respectively. 'TH' is short for 'thresholding step'. The formation of the Antarctic is revealed at the bottom (southern part) of the globe (black arrow in TH 5). Note that the globes are shown in the same view angle so the Antarctic continental is out of view in the last two subplots
Table 1.  Comparison of the time steps needed to reach the torenlence between linear method and threshold method
Tolerance $ 10^{-3} $ $ 10^{-4} $ $ 10^{-5} $ $ 10^{-6} $
Steps (Linear) 1737 4026 6304 8581
Steps (Threshold) 960 960 960 960
Tolerance $ 10^{-3} $ $ 10^{-4} $ $ 10^{-5} $ $ 10^{-6} $
Steps (Linear) 1737 4026 6304 8581
Steps (Threshold) 960 960 960 960
[1]

Anton Schiela, Julian Ortiz. Second order directional shape derivatives of integrals on submanifolds. Mathematical Control & Related Fields, 2021  doi: 10.3934/mcrf.2021017

[2]

Andreu Ferré Moragues. Properties of multicorrelation sequences and large returns under some ergodicity assumptions. Discrete & Continuous Dynamical Systems, 2021, 41 (6) : 2809-2828. doi: 10.3934/dcds.2020386

[3]

Caifang Wang, Tie Zhou. The order of convergence for Landweber Scheme with $\alpha,\beta$-rule. Inverse Problems & Imaging, 2012, 6 (1) : 133-146. doi: 10.3934/ipi.2012.6.133

[4]

Roberto Civino, Riccardo Longo. Formal security proof for a scheme on a topological network. Advances in Mathematics of Communications, 2021  doi: 10.3934/amc.2021009

[5]

Misha Perepelitsa. A model of cultural evolution in the context of strategic conflict. Kinetic & Related Models, , () : -. doi: 10.3934/krm.2021014

[6]

Pengyu Chen. Non-autonomous stochastic evolution equations with nonlinear noise and nonlocal conditions governed by noncompact evolution families. Discrete & Continuous Dynamical Systems, 2021, 41 (6) : 2725-3737. doi: 10.3934/dcds.2020383

[7]

Wolf-Jüergen Beyn, Janosch Rieger. The implicit Euler scheme for one-sided Lipschitz differential inclusions. Discrete & Continuous Dynamical Systems - B, 2010, 14 (2) : 409-428. doi: 10.3934/dcdsb.2010.14.409

[8]

Alina Chertock, Alexander Kurganov, Mária Lukáčová-Medvi${\rm{\check{d}}}$ová, Șeyma Nur Özcan. An asymptotic preserving scheme for kinetic chemotaxis models in two space dimensions. Kinetic & Related Models, 2019, 12 (1) : 195-216. doi: 10.3934/krm.2019009

[9]

Pengyu Chen. Periodic solutions to non-autonomous evolution equations with multi-delays. Discrete & Continuous Dynamical Systems - B, 2021, 26 (6) : 2921-2939. doi: 10.3934/dcdsb.2020211

[10]

Caichun Chai, Tiaojun Xiao, Zhangwei Feng. Evolution of revenue preference for competing firms with nonlinear inverse demand. Journal of Industrial & Management Optimization, 2021  doi: 10.3934/jimo.2021071

[11]

Matheus C. Bortolan, José Manuel Uzal. Upper and weak-lower semicontinuity of pullback attractors to impulsive evolution processes. Discrete & Continuous Dynamical Systems - B, 2021, 26 (7) : 3667-3692. doi: 10.3934/dcdsb.2020252

[12]

K. Ravikumar, Manil T. Mohan, A. Anguraj. Approximate controllability of a non-autonomous evolution equation in Banach spaces. Numerical Algebra, Control & Optimization, 2021, 11 (3) : 461-485. doi: 10.3934/naco.2020038

[13]

Zhaoqiang Ge. Controllability and observability of stochastic implicit systems and stochastic GE-evolution operator. Numerical Algebra, Control & Optimization, 2021  doi: 10.3934/naco.2021009

[14]

Tomáš Roubíček. An energy-conserving time-discretisation scheme for poroelastic media with phase-field fracture emitting waves and heat. Discrete & Continuous Dynamical Systems - S, 2017, 10 (4) : 867-893. doi: 10.3934/dcdss.2017044

[15]

Chih-Chiang Fang. Bayesian decision making in determining optimal leased term and preventive maintenance scheme for leased facilities. Journal of Industrial & Management Optimization, 2020  doi: 10.3934/jimo.2020127

[16]

Rabiaa Ouahabi, Nasr-Eddine Hamri. Design of new scheme adaptive generalized hybrid projective synchronization for two different chaotic systems with uncertain parameters. Discrete & Continuous Dynamical Systems - B, 2021, 26 (5) : 2361-2370. doi: 10.3934/dcdsb.2020182

[17]

Mario Bukal. Well-posedness and convergence of a numerical scheme for the corrected Derrida-Lebowitz-Speer-Spohn equation using the Hellinger distance. Discrete & Continuous Dynamical Systems, 2021, 41 (7) : 3389-3414. doi: 10.3934/dcds.2021001

[18]

Tôn Việt Tạ. Strict solutions to stochastic semilinear evolution equations in M-type 2 Banach spaces. Communications on Pure & Applied Analysis, , () : -. doi: 10.3934/cpaa.2021050

[19]

Siqi Chen, Yong-Kui Chang, Yanyan Wei. Pseudo $ S $-asymptotically Bloch type periodic solutions to a damped evolution equation. Evolution Equations & Control Theory, 2021  doi: 10.3934/eect.2021017

[20]

Qi Deng, Zhipeng Qiu, Ting Guo, Libin Rong. Modeling within-host viral dynamics: The role of CTL immune responses in the evolution of drug resistance. Discrete & Continuous Dynamical Systems - B, 2021, 26 (7) : 3543-3562. doi: 10.3934/dcdsb.2020245

2019 Impact Factor: 1.373

Article outline

Figures and Tables

[Back to Top]