Optimal control of infinite-dimensional piecewise deterministic Markov processes and application to the control of neuronal dynamics via Optogenetics

Vincent Renault; Michèle Thieullen; Emmanuel Trélat

doi:10.3934/nhm.2017019

Article Contents

2017, Volume 12, Issue 3: 417-459. Doi: 10.3934/nhm.2017019

This issue Previous Article The Wigner-Lohe model for quantum synchronization and its emergent dynamics Next Article Decay rates for elastic-thermoelastic star-shaped networks

Optimal control of infinite-dimensional piecewise deterministic Markov processes and application to the control of neuronal dynamics via Optogenetics

1.
Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7599, Laboratoire de Probabilités et Modèles Aléatoires, F-75005, Paris, France
2.
Sorbonne Universités, UPMC Univ Paris 06, CNRS UMR 7598, Laboratoire Jacques-Louis Lions, F-75005, Paris, France

Received: June 30, 2016

Revised: April 30, 2017

Published: October 2017

Abstract Full Text(HTML) Figure(4) / Table(1) Related Papers Cited by

Abstract

In this paper we define an infinite-dimensional controlled piecewise deterministic Markov process (PDMP) and we study an optimal control problem with finite time horizon and unbounded cost. This process is a coupling between a continuous time Markov Chain and a set of semilinear parabolic partial differential equations, both processes depending on the control. We apply dynamic programming to the embedded Markov decision process to obtain existence of optimal relaxed controls and we give some sufficient conditions ensuring the existence of an optimal ordinary control. This study, which constitutes an extension of controlled PDMPs to infinite dimension, is motivated by the control that provides Optogenetics on neuron models such as the Hodgkin-Huxley model. We define an infinite-dimensional controlled Hodgkin-Huxley model as an infinite-dimensional controlled piecewise deterministic Markov process and apply the previous results to prove the existence of optimal ordinary controls for a tracking problem.

Keywords:

Piecewise Deterministic Markov Processes,
optimal control,
semilinear parabolic equations,
dynamic programming,
Markov Decision Processes,
Optogenetics.

Mathematics Subject Classification: Primary: 93E20, 60J25, 35K58; Secondary: 49L20, 92C20, 92C45.

Citation:

FullText(HTML)

Figure 1. Simplified four states ChR2 channel : $\varepsilon_1$, $\varepsilon_2$, $e_{12}$, $e_{21}$, $K_{d1}$, $K_{d2}$ and $K_r$ are positive constants

Download: Full-size image PowerPoint slide

Figure 2. Simplified ChR2 three states model

Download: Full-size image PowerPoint slide

Figure 3. ChR2 three states model

Download: Full-size image PowerPoint slide

Figure 4. ChR2 channel : $K_{a1}$, $K_{a2}$, and $K_{d2}$ are positive constants defined by:

Download: Full-size image PowerPoint slide

Table 1. Expression of the individual jump rate functions and the Hodgkin-Huxley model

$\underline {{\rm{In}}\;\;{D_1} = \left\{ {{n_0},{n_1},{n_2},{n_3},{n_4}} \right\}} :$
$\sigma_{n_0,n_1}(v,u) = 4\alpha_n(v)$,	$\sigma_{n_1,n_2}(v,u) = 3\alpha_n(v)$,
$\sigma_{n_2,n_3}(v,u) = 2\alpha_n(v)$,	$\sigma_{n_3,n_4}(v,u) = \alpha_n(v)$
$\sigma_{n_4,n_3}(v,u) = 4\beta_n(v)$,	$\sigma_{n_3,n_2}(v,u) = 3\beta_n(v)$,
$\sigma_{n_2,n_1}(v,u) = 2\beta_n(v)$,	$\sigma_{n_1,n_0}(v,u) = \beta_n(v)$.
$\underline {{\rm{In}}\;\;{D_2} = \left\{ {{m_0}{h_1},{m_1}{h_1},{m_2}{h_1},{m_3}{h_1},{m_0}{h_0},{m_1}{h_0},{m_2}{h_0},{m_3}{h_0}} \right\}} :$ :
$\sigma_{m_0h_1,m_1h_1}(v,u)=\sigma_{m_0h_0,m_1h_0}(v,u) = 3\alpha_m(v)$,
$\sigma_{m_1h_1,m_2h_1}(v,u) =\sigma_{m_1h_0,m_2h_0}(v,u) = 2\alpha_m(v)$,
$\sigma_{m_2h_1,m_3h_1}(v,u) = \sigma_{m_2h_0,m_3h_0}(v,u) = \alpha_m(v)$,
$\sigma_{m_3h_1,m_2h_1}(v,u) = \sigma_{m_3h_0,m_2h_0}(v,u) = 3\beta_m(v)$,
$\sigma_{m_2h_1,m_1h_1}(v,u) =\sigma_{m_2h_0,m_1h_0}(v,u) = 2\beta_m(v)$,
$\sigma_{m_1h_1,m_0h_1}(v,u) = \sigma_{m_1h_0,m_0h_0}(v,u) = \beta_m(v)$.
$\underline {{\rm{In}}\;\;{D_{ChR2}} = \left\{ {{o_1},{o_2},{c_1},{c_2}} \right\}} :$
$\sigma_{c_1,o_1}(v,u) = \varepsilon_1 u $,	$\sigma_{o_1,c_1}(v,u) = K_{d1}$,
$\sigma_{o_1,o_2}(v,u) = e_{12} $,	$\sigma_{o_2,o_1}(v,u) = e_{21}$
$\sigma_{o_2,c_2}(v,u) = K_{d2} $,	$\sigma_{c_2,o_2}(v,u) = \varepsilon_2 u $,
$\sigma_{c_2,c_1}(v,u) = K_r$.
$\alpha_n(v)=\frac{0.1-0.01v}{e^{1-0.1v}-1}$,	$\beta_n(v)=0.125e^{-\frac{v}{80}}$,
$\alpha_m(v)=\frac{2.5-0.1v}{e^{2.5-0.1v}-1}$,	$\beta_m(v)=4e^{-\frac{v}{18}}$,
$\alpha_h(v)=0.07e^{-\frac{v}{20}}$,	$\beta_h(v)=\frac{1}{e^{3-0.1v}+1}$.
$(HH)\left\{ \begin{aligned} C \dot{V}(t)&= \bar{g}_Kn^4(t)(E_K - V(t)) +\bar{g}_{Na}m^3(t)h(t)(E_{Na}-V(t))\\ & \ \ \ \ \ \ \ \ \ + g_L(E_L-V(t)) + I_{ext}(t),\\ \dot{n}(t)&= \alpha_n(V(t))(1-n(t)) - \beta_n(V(t))n(t),\\ \dot{m}(t)&= \alpha_m(V(t))(1-m(t)) - \beta_m(V(t))m(t),\\ \dot{h}(t)&= \alpha_h(V(t))(1-h(t)) - \beta_h(V(t))h(t). \end{aligned} \right.$

| Show Table

DownLoad: CSV

Related Papers

Cited by

References

[1]	N. U. Ahmed, Properties of relaxed trajectories for a class of nonlinear evolution equations on a Banach space, SIAM J. Control Optim., 21 (1983), 953-957. doi: 10.1137/0321058.
[2]	N. U. Ahmed and K. L. Teo, Optimal control of systems governed by a class of nonlinear evolution equations in a reflexive Banach space, Journal of Optimization Theory and Applications, 25 (1978), 57-81.
[3]	N. U. Ahmed and X. Xiang, Properties of relaxed trajectories of evolution equations and optimal control, SIAM J. Control Optim., 31 (1993), 1135-1142. doi: 10.1137/0331053.
[4]	T. D. Austin, The emergence of the deterministic Hodgkin-Huxley equations as a limit from the underlying stochastic ion-channel mechanism, Ann. Appl. Probab., 18 (2008), 1279-1325. doi: 10.1214/07-AAP494.
[5]	E. J. Balder, A general denseness result for relaxed control theory, Bull. Austral. Math. Soc., 30 (1984), 463-475. doi: 10.1017/S0004972700002185.
[6]	D. Bertsekas and S. Shreve, Stochastic Optimal Control: The Discrete-Time Case, Academic Press, 1978.
[7]	P. Billingsley, Convergence Of Probability Measures, John Wiley & Sons, New York, 1968.
[8]	E. S. Boyden, F. Zhang, E. Bamberg, G. Nagel and K. Deisseroth, Millisecond-timescale, genetically targeted optical control of neural activity, Nature Neuroscience, 8 (2005), 1263-1268. doi: 10.1038/nn1525.
[9]	A. Brandejsky, B. de Saporta and F. Dufour, Numerical methods for the exit time of a Piecewise Deterministic Markov Process, Adv. in Appl. Probab., 44 (2012), 196-225. doi: 10.1017/S0001867800005504.
[10]	E. Buckwar and M. Riedler, An exact stochastic hybrid model of excitable membranes including spatio-temporal evolution, J. Math. Biol., 63 (2011), 1051-1093. doi: 10.1007/s00285-010-0395-z.
[11]	N. Bäuerle and U. Rieder, Optimal control of Piecewise Deterministic Markov Processes with finite time horizon, Modern Trends of Controlled Stochastic Processes: Theory and Applications, (2010), 144-160.
[12]	N. Bäuerle and U. Rieder, AMDP algorithms for portfolio optimization problems in pure jump markets, Finance Stoch., 13 (2009), 591-611. doi: 10.1007/s00780-009-0093-0.
[13]	N. Bäuerle and U. Rieder, Markov Decision Processes With Applications To Finance, Springer, Heidelberg, 2011.
[14]	O. Costa and F. Dufour, Stability and ergodicity of piecewise deterministic Markov processes, SIAM J. of Control and Opt., 47 (2008), 1053-1077. doi: 10.1137/060670109.
[15]	O. Costa and F. Dufour, Singular perturbation for the discounted continuous control of Piecewise Deterministic Markov Processes, Appl. Math. and Opt., 63 (2011), 357-384. doi: 10.1007/s00245-010-9124-7.
[16]	O.L.V. Costa, C.A. B Raymundo, F. Dufour and K. Gonzalez, Optimal stopping with continuous control of piecewise deterministic Markov processes, Stoch. Stoch. Rep., 70 (2000), 41-73. doi: 10.1080/17442500008834245.
[17]	A. Crudu, A. Debussche, A. Muller and O. Radulescu, Convergence of stochastic gene networks to hybrid piecewise deterministic processe, Ann. Appl. Prob., 22 (2012), 1822-1859. doi: 10.1214/11-AAP814.
[18]	M. H. A. Davis, Piecewise-Deterministic Markov Processes: A general class of non-diffusion stochastic models, J. R. Statist. Soc., 46 (1984), 353-388.
[19]	M. H. A. Davis, Markov Models and Optimization, Chapman and Hall, 1993. doi: 10.1007/978-1-4899-4483-2.
[20]	B. de Saporta, F. Dufour and H. Zhang, Numerical Methods for Simulation and Optimization of Piecewise Deterministic Markov Processes, Wiley, 2016.
[21]	J. Diestel and J. J. Uhl, Vector Measures, American Mathematical Society, Providence, 1977.
[22]	V. Dumas, F. Guillemin and Ph. Robert, A Markovian analysis of additive-increase multiplicative-decrease algorithms, Adv. in Appl. Probab., 34 (2002), 85-111. doi: 10.1017/S000186780001140X.
[23]	N. Dunford and J. T. Schwartz, Linear Operators. Part Ⅰ: General Theory, Academic Press, New York-London, 1963.
[24]	K. -J. Engel and R. Nagel, One Parameter Semigroups for Linear Evolution Equations, Springer-Verlag New York, 2000.
[25]	M. H. A. Davis, Piecewise deterministic Markov control processes with feedback controls and unbounded costs, Acta Applicandae Mathematicae, 82 (2004), 239-267. doi: 10.1023/B:ACAP.0000031200.76583.75.
[26]	R. Gamkrelidze, Principle of Optimal Control Theory Plenum, New York, 1987.
[27]	A. Genadot and M. Thieullen, Averaging for a fully coupled piecewise deterministic Markov process in infinite dimensions, Adv. in Appl. Probab., 44 (2012), 749-773. doi: 10.1017/S0001867800005863.
[28]	D. Goreac and M. Martinez, Algebraic invariance conditions in the study of approximate (null-)controllability of Markov switch processes, Mathematics of Control, Signals, and Systems, 27 (2015), 551-578. doi: 10.1007/s00498-015-0146-1.
[29]	R. M. Gray and D. L. Neuhoff, Quantization, IEEE Trans. Inform. Theory, 44 (1998), 2325-2383. doi: 10.1109/18.720541.
[30]	A. L. Hodgkin and A. F. Huxley, A quantitative description of membrane current and its application to conduction and excitation in nerve, J. Physiol., 117 (1952), 500-544.
[31]	Q. Hu and W. Yue, Markov Decision Processes with Their Applications, Springer US, 2008.
[32]	J. Jacod, Multivariate point processes: Predictable projections, Radon-Nikodym derivatives, representation of martingales, Z. Wahrsag. Verw. Gebiete, 34 (1975), 235-253. doi: 10.1007/BF00536010.
[33]	K. Nikolic, N. Grossman, M.S. Grubb, J. Burrone, C. Toumazou and P. Degenaar, Photocycles of Channelrhodopsin-2, Photochemistry and Photobiology, 85 (2009), 400-411. doi: 10.1111/j.1751-1097.2008.00460.x.
[34]	K. Nikolic, S. Jarvis, N. Grossman and S. Schultz, Computational models of Optogenetic tools for controlling neural circuits with light, Conf. Proc. IEEE Eng. Med. Biol. Soc., (2013), 5934-5937. doi: 10.1109/EMBC.2013.6610903.
[35]	G. Pagès, H. Pham and J. Printemps, Handbook of computational and numerical methods in finance, Birkhäuser Boston, (2004), 253-297.
[36]	K. Pakdaman, M. Thieullen and G. Wainrib, Reduction of stochastic conductance-based neuron models with time-sacles separation, J. Comput. Neurosci., 32 (2012), 327-346. doi: 10.1007/s10827-011-0355-7.
[37]	N. S. Papageorgiou, Properties of the relaxed trajectories of evolution equations and optimal control, SIAM J. Control Optim., 27 (1989), 267-288. doi: 10.1137/0327014.
[38]	V. Renault, M. Thieullen and E. Trélat, Minimal time spiking in various ChR2-controlled neuron models, J. Math. Biol., (2017), 1-42. doi: 10.1007/s00285-017-1101-1.
[39]	M. Riedler, M. Thieullen and G. Wainrib, Limit theorems for infinite-dimensional Piecewise Deterministic Markov Processes. Applications to stochastic excitable membrane models, Electron. J. Probab., 17 (2012), 1-48.
[40]	D. Vermes, Optimal control of piecewise deterministic Markov processes, Stochastics. An International Journal of Probability and Stochastic Processes, 14 (1985), 165-207. doi: 10.1080/17442508508833338.
[41]	J. Warga, Relaxed variational problem, J. Math. Anal. Appl., 4 (1962), 111-128. doi: 10.1016/0022-247X(62)90033-1.
[42]	J. Warga, Necessary conditions for minimum in relaxed variational problems, J. Math. Anal. Appl., 4 (1962), 129-145. doi: 10.1016/0022-247X(62)90034-3.
[43]	J. Warga, Optimal Control of Differential and Functional Equations, Wiley-Interscience, New York, 1972.
[44]	J. C. Williams and J. Xu et al, Computational optogenetics: Empirically-derived voltage-and light-sensitive Channelrhodopsin-2 model, JPLoS Comput Biol, 9 (2013), e1003220. doi: 10.1371/journal.pcbi.1003220.
[45]	L. C. Young, Lectures on the Calculus of Variations and Optimal Control Theory, W. B. Saunders, Philadelphia, PA, 1969.
[46]	A. A. Yushkevich, On reducing a jump controllable Markov model to a model with discrete time, Theory Probab. Appl., 25 (1980), 58-69. doi: 10.1137/1125005.