# American Institute of Mathematical Sciences

September  2017, 12(3): 417-459. doi: 10.3934/nhm.2017019

## Optimal control of infinite-dimensional piecewise deterministic Markov processes and application to the control of neuronal dynamics via Optogenetics

 1 Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7599, Laboratoire de Probabilités et Modèles Aléatoires, F-75005, Paris, France 2 Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7598, Laboratoire Jacques-Louis Lions, F-75005, Paris, France

Received  July 2016 Revised  May 2017 Published  September 2017

In this paper we define an infinite-dimensional controlled piecewise deterministic Markov process (PDMP) and we study an optimal control problem with finite time horizon and unbounded cost. This process is a coupling between a continuous time Markov Chain and a set of semilinear parabolic partial differential equations, both processes depending on the control. We apply dynamic programming to the embedded Markov decision process to obtain existence of optimal relaxed controls and we give some sufficient conditions ensuring the existence of an optimal ordinary control. This study, which constitutes an extension of controlled PDMPs to infinite dimension, is motivated by the control that provides Optogenetics on neuron models such as the Hodgkin-Huxley model. We define an infinite-dimensional controlled Hodgkin-Huxley model as an infinite-dimensional controlled piecewise deterministic Markov process and apply the previous results to prove the existence of optimal ordinary controls for a tracking problem.

Citation: Vincent Renault, Michèle Thieullen, Emmanuel Trélat. Optimal control of infinite-dimensional piecewise deterministic Markov processes and application to the control of neuronal dynamics via Optogenetics. Networks & Heterogeneous Media, 2017, 12 (3) : 417-459. doi: 10.3934/nhm.2017019
##### References:

show all references

##### References:
Simplified four states ChR2 channel : $\varepsilon_1$, $\varepsilon_2$, $e_{12}$, $e_{21}$, $K_{d1}$, $K_{d2}$ and $K_r$ are positive constants
Simplified ChR2 three states model
ChR2 three states model
ChR2 channel : $K_{a1}$, $K_{a2}$, and $K_{d2}$ are positive constants defined by:
Expression of the individual jump rate functions and the Hodgkin-Huxley model
 $\underline {{\rm{In}}\;\;{D_1} = \left\{ {{n_0},{n_1},{n_2},{n_3},{n_4}} \right\}} :$ $\sigma_{n_0,n_1}(v,u) = 4\alpha_n(v)$, $\sigma_{n_1,n_2}(v,u) = 3\alpha_n(v)$, $\sigma_{n_2,n_3}(v,u) = 2\alpha_n(v)$, $\sigma_{n_3,n_4}(v,u) = \alpha_n(v)$ $\sigma_{n_4,n_3}(v,u) = 4\beta_n(v)$, $\sigma_{n_3,n_2}(v,u) = 3\beta_n(v)$, $\sigma_{n_2,n_1}(v,u) = 2\beta_n(v)$, $\sigma_{n_1,n_0}(v,u) = \beta_n(v)$. $\underline {{\rm{In}}\;\;{D_2} = \left\{ {{m_0}{h_1},{m_1}{h_1},{m_2}{h_1},{m_3}{h_1},{m_0}{h_0},{m_1}{h_0},{m_2}{h_0},{m_3}{h_0}} \right\}} :$ : $\sigma_{m_0h_1,m_1h_1}(v,u)=\sigma_{m_0h_0,m_1h_0}(v,u) = 3\alpha_m(v)$, $\sigma_{m_1h_1,m_2h_1}(v,u) =\sigma_{m_1h_0,m_2h_0}(v,u) = 2\alpha_m(v)$, $\sigma_{m_2h_1,m_3h_1}(v,u) = \sigma_{m_2h_0,m_3h_0}(v,u) = \alpha_m(v)$, $\sigma_{m_3h_1,m_2h_1}(v,u) = \sigma_{m_3h_0,m_2h_0}(v,u) = 3\beta_m(v)$, $\sigma_{m_2h_1,m_1h_1}(v,u) =\sigma_{m_2h_0,m_1h_0}(v,u) = 2\beta_m(v)$, $\sigma_{m_1h_1,m_0h_1}(v,u) = \sigma_{m_1h_0,m_0h_0}(v,u) = \beta_m(v)$. $\underline {{\rm{In}}\;\;{D_{ChR2}} = \left\{ {{o_1},{o_2},{c_1},{c_2}} \right\}} :$ $\sigma_{c_1,o_1}(v,u) = \varepsilon_1 u$, $\sigma_{o_1,c_1}(v,u) = K_{d1}$, $\sigma_{o_1,o_2}(v,u) = e_{12}$, $\sigma_{o_2,o_1}(v,u) = e_{21}$ $\sigma_{o_2,c_2}(v,u) = K_{d2}$, $\sigma_{c_2,o_2}(v,u) = \varepsilon_2 u$, $\sigma_{c_2,c_1}(v,u) = K_r$. $\alpha_n(v)=\frac{0.1-0.01v}{e^{1-0.1v}-1}$, $\beta_n(v)=0.125e^{-\frac{v}{80}}$, $\alpha_m(v)=\frac{2.5-0.1v}{e^{2.5-0.1v}-1}$, $\beta_m(v)=4e^{-\frac{v}{18}}$, $\alpha_h(v)=0.07e^{-\frac{v}{20}}$, $\beta_h(v)=\frac{1}{e^{3-0.1v}+1}$. (HH)\left\{ \begin{aligned} C \dot{V}(t)&= \bar{g}_Kn^4(t)(E_K - V(t)) +\bar{g}_{Na}m^3(t)h(t)(E_{Na}-V(t))\\ & \ \ \ \ \ \ \ \ \ + g_L(E_L-V(t)) + I_{ext}(t),\\ \dot{n}(t)&= \alpha_n(V(t))(1-n(t)) - \beta_n(V(t))n(t),\\ \dot{m}(t)&= \alpha_m(V(t))(1-m(t)) - \beta_m(V(t))m(t),\\ \dot{h}(t)&= \alpha_h(V(t))(1-h(t)) - \beta_h(V(t))h(t). \end{aligned} \right.
 $\underline {{\rm{In}}\;\;{D_1} = \left\{ {{n_0},{n_1},{n_2},{n_3},{n_4}} \right\}} :$ $\sigma_{n_0,n_1}(v,u) = 4\alpha_n(v)$, $\sigma_{n_1,n_2}(v,u) = 3\alpha_n(v)$, $\sigma_{n_2,n_3}(v,u) = 2\alpha_n(v)$, $\sigma_{n_3,n_4}(v,u) = \alpha_n(v)$ $\sigma_{n_4,n_3}(v,u) = 4\beta_n(v)$, $\sigma_{n_3,n_2}(v,u) = 3\beta_n(v)$, $\sigma_{n_2,n_1}(v,u) = 2\beta_n(v)$, $\sigma_{n_1,n_0}(v,u) = \beta_n(v)$. $\underline {{\rm{In}}\;\;{D_2} = \left\{ {{m_0}{h_1},{m_1}{h_1},{m_2}{h_1},{m_3}{h_1},{m_0}{h_0},{m_1}{h_0},{m_2}{h_0},{m_3}{h_0}} \right\}} :$ : $\sigma_{m_0h_1,m_1h_1}(v,u)=\sigma_{m_0h_0,m_1h_0}(v,u) = 3\alpha_m(v)$, $\sigma_{m_1h_1,m_2h_1}(v,u) =\sigma_{m_1h_0,m_2h_0}(v,u) = 2\alpha_m(v)$, $\sigma_{m_2h_1,m_3h_1}(v,u) = \sigma_{m_2h_0,m_3h_0}(v,u) = \alpha_m(v)$, $\sigma_{m_3h_1,m_2h_1}(v,u) = \sigma_{m_3h_0,m_2h_0}(v,u) = 3\beta_m(v)$, $\sigma_{m_2h_1,m_1h_1}(v,u) =\sigma_{m_2h_0,m_1h_0}(v,u) = 2\beta_m(v)$, $\sigma_{m_1h_1,m_0h_1}(v,u) = \sigma_{m_1h_0,m_0h_0}(v,u) = \beta_m(v)$. $\underline {{\rm{In}}\;\;{D_{ChR2}} = \left\{ {{o_1},{o_2},{c_1},{c_2}} \right\}} :$ $\sigma_{c_1,o_1}(v,u) = \varepsilon_1 u$, $\sigma_{o_1,c_1}(v,u) = K_{d1}$, $\sigma_{o_1,o_2}(v,u) = e_{12}$, $\sigma_{o_2,o_1}(v,u) = e_{21}$ $\sigma_{o_2,c_2}(v,u) = K_{d2}$, $\sigma_{c_2,o_2}(v,u) = \varepsilon_2 u$, $\sigma_{c_2,c_1}(v,u) = K_r$. $\alpha_n(v)=\frac{0.1-0.01v}{e^{1-0.1v}-1}$, $\beta_n(v)=0.125e^{-\frac{v}{80}}$, $\alpha_m(v)=\frac{2.5-0.1v}{e^{2.5-0.1v}-1}$, $\beta_m(v)=4e^{-\frac{v}{18}}$, $\alpha_h(v)=0.07e^{-\frac{v}{20}}$, $\beta_h(v)=\frac{1}{e^{3-0.1v}+1}$. (HH)\left\{ \begin{aligned} C \dot{V}(t)&= \bar{g}_Kn^4(t)(E_K - V(t)) +\bar{g}_{Na}m^3(t)h(t)(E_{Na}-V(t))\\ & \ \ \ \ \ \ \ \ \ + g_L(E_L-V(t)) + I_{ext}(t),\\ \dot{n}(t)&= \alpha_n(V(t))(1-n(t)) - \beta_n(V(t))n(t),\\ \dot{m}(t)&= \alpha_m(V(t))(1-m(t)) - \beta_m(V(t))m(t),\\ \dot{h}(t)&= \alpha_h(V(t))(1-h(t)) - \beta_h(V(t))h(t). \end{aligned} \right.
 [1] Mathias Staudigl. A limit theorem for Markov decision processes. Journal of Dynamics & Games, 2014, 1 (4) : 639-659. doi: 10.3934/jdg.2014.1.639 [2] Artur Stephan, Holger Stephan. Memory equations as reduced Markov processes. Discrete & Continuous Dynamical Systems, 2019, 39 (4) : 2133-2155. doi: 10.3934/dcds.2019089 [3] A. Mittal, N. Hemachandra. Learning algorithms for finite horizon constrained Markov decision processes. Journal of Industrial & Management Optimization, 2007, 3 (3) : 429-444. doi: 10.3934/jimo.2007.3.429 [4] Wael Bahsoun, Paweł Góra. SRB measures for certain Markov processes. Discrete & Continuous Dynamical Systems, 2011, 30 (1) : 17-37. doi: 10.3934/dcds.2011.30.17 [5] Qiuli Liu, Xiaolong Zou. A risk minimization problem for finite horizon semi-Markov decision processes with loss rates. Journal of Dynamics & Games, 2018, 5 (2) : 143-163. doi: 10.3934/jdg.2018009 [6] H.Thomas Banks, Shuhua Hu. Nonlinear stochastic Markov processes and modeling uncertainty in populations. Mathematical Biosciences & Engineering, 2012, 9 (1) : 1-25. doi: 10.3934/mbe.2012.9.1 [7] Angelica Pachon, Federico Polito, Costantino Ricciuti. On discrete-time semi-Markov processes. Discrete & Continuous Dynamical Systems - B, 2021, 26 (3) : 1499-1529. doi: 10.3934/dcdsb.2020170 [8] Thomas Kruse, Mikhail Urusov. Approximating exit times of continuous Markov processes. Discrete & Continuous Dynamical Systems - B, 2020, 25 (9) : 3631-3650. doi: 10.3934/dcdsb.2020076 [9] Xian Chen, Zhi-Ming Ma. A transformation of Markov jump processes and applications in genetic study. Discrete & Continuous Dynamical Systems, 2014, 34 (12) : 5061-5084. doi: 10.3934/dcds.2014.34.5061 [10] A. M. Vershik. Polymorphisms, Markov processes, quasi-similarity. Discrete & Continuous Dynamical Systems, 2005, 13 (5) : 1305-1324. doi: 10.3934/dcds.2005.13.1305 [11] Vladimir Kazakov. Sampling - reconstruction procedure with jitter of markov continuous processes formed by stochastic differential equations of the first order. Conference Publications, 2009, 2009 (Special) : 433-441. doi: 10.3934/proc.2009.2009.433 [12] Oliver Knill. A deterministic displacement theorem for Poisson processes. Electronic Research Announcements, 1997, 3: 110-113. [13] Marzia Bisi, Maria Groppi, Giorgio Martalò, Romina Travaglini. Optimal control of leachate recirculation for anaerobic processes in landfills. Discrete & Continuous Dynamical Systems - B, 2021, 26 (6) : 2957-2976. doi: 10.3934/dcdsb.2020215 [14] Roberto C. Cabrales, Gema Camacho, Enrique Fernández-Cara. Analysis and optimal control of some solidification processes. Discrete & Continuous Dynamical Systems, 2014, 34 (10) : 3985-4017. doi: 10.3934/dcds.2014.34.3985 [15] Zhenzhong Zhang, Enhua Zhang, Jinying Tong. Necessary and sufficient conditions for ergodicity of CIR model driven by stable processes with Markov switching. Discrete & Continuous Dynamical Systems - B, 2018, 23 (6) : 2433-2455. doi: 10.3934/dcdsb.2018053 [16] Armen Shirikyan. Ergodicity for a class of Markov processes and applications to randomly forced PDE'S. II. Discrete & Continuous Dynamical Systems - B, 2006, 6 (4) : 911-926. doi: 10.3934/dcdsb.2006.6.911 [17] Gábor Horváth, Zsolt Saffer, Miklós Telek. Queue length analysis of a Markov-modulated vacation queue with dependent arrival and service processes and exhaustive service policy. Journal of Industrial & Management Optimization, 2017, 13 (3) : 1365-1381. doi: 10.3934/jimo.2016077 [18] Linyi Qian, Wei Wang, Rongming Wang. Risk-minimizing portfolio selection for insurance payment processes under a Markov-modulated model. Journal of Industrial & Management Optimization, 2013, 9 (2) : 411-429. doi: 10.3934/jimo.2013.9.411 [19] Rein Luus. Optimal control of oscillatory systems by iterative dynamic programming. Journal of Industrial & Management Optimization, 2008, 4 (1) : 1-15. doi: 10.3934/jimo.2008.4.1 [20] Sébastien Court, Karl Kunisch, Laurent Pfeiffer. Hybrid optimal control problems for a class of semilinear parabolic equations. Discrete & Continuous Dynamical Systems - S, 2018, 11 (6) : 1031-1060. doi: 10.3934/dcdss.2018060

2019 Impact Factor: 1.053