Figure 1.
The function $F^*_{(n_0+1)k}(1, t, \lambda)$
-
Previous Article
Stable manifold market sequences
- JDG Home
- This Issue
-
Next Article
Constrained stochastic differential games with additive structure: Average and discount payoffs
April
2018, 5(2): 143-163.
doi: 10.3934/jdg.2018009
A risk minimization problem for finite horizon semi-Markov decision processes with loss rates
| 1. | School of Mathematical Sciences, South China Normal University, Guangzhou 510631, China |
| 2. | School of Economics and Statistics, Guangzhou University, Guangzhou 510006, China |
This paper deals with the risk probability for finite horizon semi-Markov decision processes with loss rates. The criterion to be minimized is the risk probability that the total loss incurred during a finite horizon exceed a loss level. For such an optimality problem, we first establish the optimality equation, and prove that the optimal value function is a unique solution to the optimality equation. We then show the existence of an optimal policy, and develop a value iteration algorithm for computing the value function and optimal policies. We also derive the approximation of the value function and the rules of iteration. Finally, a numerical example is given to illustrate our results.
Mathematics Subject Classification:
Primary: 90C40; Secondary: 93E20.
Citation:
Qiuli Liu, Xiaolong Zou. A risk minimization problem for finite horizon semi-Markov decision processes with loss rates. Journal of Dynamics and Games,
2018, 5
(2)
: 143-163.
doi: 10.3934/jdg.2018009
![]()
References:
| [1] |
N. Bauerle and U. Rieder, Markov Decision Processes with Application to Finance, Universitext, Springer, Heidelberg, 2011. |
| [2] |
M. Bouakiz and Y. Kebir,
Target-level criterion in Markov decision processes, Journal of Optimization Theory and Applications, 86 (1995), 1-15.
doi: 10.1007/BF02193458. |
| [3] |
M. K. Ghosh and S. Subhamay,
Non-stationary semi-Markov secision processes on a finite horizon, Stochastic Analysis and Applications, 31 (2013), 183-190.
doi: 10.1080/07362994.2013.741405. |
| [4] |
X. P. Guo and O. Hernández-Lerma, Continuous-Time Markov Decision Processes: Theory and Applications, Springer-Verlag, Berlin, 2009. |
| [5] |
X. P. Guo and J. Yang,
A new condition and approach for zero-sum stochastic games with average payoffs, Stochastic Analysis and Applications, 26 (2008), 537-561.
doi: 10.1080/07362990802007095. |
| [6] |
X. P. Guo, P. Shi and W. P. Zhu,
Strong average optimality for controlled nonhomogeneous Markov chains, Stochastic Analysis and Applications, 19 (2001), 115-134.
doi: 10.1081/SAP-100001186. |
| [7] |
O. Hernández-Lerma and J. B. Lasserre, Discrete-time Markov Control Processes, Basic optimality criteria, Springer-Verlag, New York, 1996. |
| [8] |
O. Hernández-Lerma and J. B. Lasserre, Further Topics on Discrete-Time Markov Control Processes, Springer-Verlag, New York, 1999. |
| [9] |
Y. H. Huang and X. P. Guo,
Optimal risk probability for first passage models in semi-Markov decision processes, Journal Mathematical Analysis Applications, 359 (2009), 404-420.
doi: 10.1016/j.jmaa.2009.05.058. |
| [10] |
Y. H. Huang, X. P. Guo and X. Y. Song,
Performance analysis for controlled semi-Markov systems with application to maintenance, Journal of Optimization Theory and Applications, 150 (2011), 395-415.
doi: 10.1007/s10957-011-9813-7. |
| [11] |
Y. H. Huang and X. P. Guo,
Finite horizon semi-Markov decision processes with application to maintenance systems, European Journal Operations Research, 212 (2011), 131-140.
doi: 10.1016/j.ejor.2011.01.027. |
| [12] |
Y. H. Huang, X. P. Guo and Z. F. Li,
Minimum risk probability for finite horizon semi-Markov decision processes, Journal Mathematical Analysis Applications, 402 (2013), 378-391.
doi: 10.1016/j.jmaa.2013.01.021. |
| [13] |
Y. H. Huang and X. P. Guo,
Mean-variance problems for finite horizon semi-Markov decision processes, Applications Mathematical Optimization, 72 (2015), 233-259.
doi: 10.1007/s00245-014-9278-9. |
| [14] |
N. Limnios and G. Oprisan, Semi-Markov Processes and Reliability, Birkhäuser Boston, Inc., Boston, MA, 2001. |
| [15] |
J. Y. Liu and S. M. Huang,
Markov decision processes with distribution function criterion of first-passage time, Applications Mathematical Optimization, 43 (2001), 187-201.
doi: 10.1007/s00245-001-0007-9. |
| [16] |
P. M. Madhani,
Rebalancing fixed and variable pay in a sales organization: A business cycle perspective, Compensation Benefits Review, 42 (2010), 179-189.
doi: 10.1177/0886368709359668. |
| [17] |
J. W. Mamer,
Successive approximations for finite horizon semi-Markov decision processes with application to asset liquidation, Oper. Res., 34 (1986), 638-644.
doi: 10.1287/opre.34.4.638. |
| [18] |
Y. Ohtsubo,
Minimizing risk models in stochastic shortest path problems, Mathematical Methods of Operations Research, 57 (2003), 79-88.
doi: 10.1007/s001860200246. |
| [19] |
Y. Ohtsubo,
Optimal threshold probability in undiscounted Markov decision processes with a target set, Appl. Math. Comput., 149 (2004), 519-532.
doi: 10.1016/S0096-3003(03)00158-9. |
| [20] |
M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc., New York, 1994. |
| [21] |
C. Ruhm,
Are recessions good for your health?, Quarterly Journal of Economics, 115 (2000), 617-650.
doi: 10.3386/w5570. |
| [22] |
M. Sakaguchi and Y. Ohtsubo,
Markov decision processes associated with two threshold probability criteria, Journal Control Theory Applications, 11 (2013), 548-557.
doi: 10.1007/s11768-013-2194-8. |
| [23] |
Q. D. Wei and X. P. Guo,
New average optimality conditions for semi-Markov decision processes in Borel spaces, Journal of Optimization Theory and Applications, 153 (2012), 709-732.
doi: 10.1007/s10957-012-9986-8. |
| [24] |
D. J. White,
Minimising a threshold probability in discounted Markov decision processes, J. Math. Anal. Appl., 173 (1993), 634-646.
doi: 10.1006/jmaa.1993.1093. |
| [25] |
Y. H. Wu,
Bounds for the ruin probability under a Markovian modulated risk model, Communications in statistics Stochastic Models, 15 (1999), 125-136.
doi: 10.1080/15326349908807529. |
| [26] |
S. X. Yu, Y. L. Lin and P. F. Yan,
Optimization models for the first arrival target distribution function in discrete time, J. Math. Anal. Appl., 225 (1998), 193-223.
doi: 10.1006/jmaa.1998.6015. |
show all references
References:
| [1] |
N. Bauerle and U. Rieder, Markov Decision Processes with Application to Finance, Universitext, Springer, Heidelberg, 2011. |
| [2] |
M. Bouakiz and Y. Kebir,
Target-level criterion in Markov decision processes, Journal of Optimization Theory and Applications, 86 (1995), 1-15.
doi: 10.1007/BF02193458. |
| [3] |
M. K. Ghosh and S. Subhamay,
Non-stationary semi-Markov secision processes on a finite horizon, Stochastic Analysis and Applications, 31 (2013), 183-190.
doi: 10.1080/07362994.2013.741405. |
| [4] |
X. P. Guo and O. Hernández-Lerma, Continuous-Time Markov Decision Processes: Theory and Applications, Springer-Verlag, Berlin, 2009. |
| [5] |
X. P. Guo and J. Yang,
A new condition and approach for zero-sum stochastic games with average payoffs, Stochastic Analysis and Applications, 26 (2008), 537-561.
doi: 10.1080/07362990802007095. |
| [6] |
X. P. Guo, P. Shi and W. P. Zhu,
Strong average optimality for controlled nonhomogeneous Markov chains, Stochastic Analysis and Applications, 19 (2001), 115-134.
doi: 10.1081/SAP-100001186. |
| [7] |
O. Hernández-Lerma and J. B. Lasserre, Discrete-time Markov Control Processes, Basic optimality criteria, Springer-Verlag, New York, 1996. |
| [8] |
O. Hernández-Lerma and J. B. Lasserre, Further Topics on Discrete-Time Markov Control Processes, Springer-Verlag, New York, 1999. |
| [9] |
Y. H. Huang and X. P. Guo,
Optimal risk probability for first passage models in semi-Markov decision processes, Journal Mathematical Analysis Applications, 359 (2009), 404-420.
doi: 10.1016/j.jmaa.2009.05.058. |
| [10] |
Y. H. Huang, X. P. Guo and X. Y. Song,
Performance analysis for controlled semi-Markov systems with application to maintenance, Journal of Optimization Theory and Applications, 150 (2011), 395-415.
doi: 10.1007/s10957-011-9813-7. |
| [11] |
Y. H. Huang and X. P. Guo,
Finite horizon semi-Markov decision processes with application to maintenance systems, European Journal Operations Research, 212 (2011), 131-140.
doi: 10.1016/j.ejor.2011.01.027. |
| [12] |
Y. H. Huang, X. P. Guo and Z. F. Li,
Minimum risk probability for finite horizon semi-Markov decision processes, Journal Mathematical Analysis Applications, 402 (2013), 378-391.
doi: 10.1016/j.jmaa.2013.01.021. |
| [13] |
Y. H. Huang and X. P. Guo,
Mean-variance problems for finite horizon semi-Markov decision processes, Applications Mathematical Optimization, 72 (2015), 233-259.
doi: 10.1007/s00245-014-9278-9. |
| [14] |
N. Limnios and G. Oprisan, Semi-Markov Processes and Reliability, Birkhäuser Boston, Inc., Boston, MA, 2001. |
| [15] |
J. Y. Liu and S. M. Huang,
Markov decision processes with distribution function criterion of first-passage time, Applications Mathematical Optimization, 43 (2001), 187-201.
doi: 10.1007/s00245-001-0007-9. |
| [16] |
P. M. Madhani,
Rebalancing fixed and variable pay in a sales organization: A business cycle perspective, Compensation Benefits Review, 42 (2010), 179-189.
doi: 10.1177/0886368709359668. |
| [17] |
J. W. Mamer,
Successive approximations for finite horizon semi-Markov decision processes with application to asset liquidation, Oper. Res., 34 (1986), 638-644.
doi: 10.1287/opre.34.4.638. |
| [18] |
Y. Ohtsubo,
Minimizing risk models in stochastic shortest path problems, Mathematical Methods of Operations Research, 57 (2003), 79-88.
doi: 10.1007/s001860200246. |
| [19] |
Y. Ohtsubo,
Optimal threshold probability in undiscounted Markov decision processes with a target set, Appl. Math. Comput., 149 (2004), 519-532.
doi: 10.1016/S0096-3003(03)00158-9. |
| [20] |
M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc., New York, 1994. |
| [21] |
C. Ruhm,
Are recessions good for your health?, Quarterly Journal of Economics, 115 (2000), 617-650.
doi: 10.3386/w5570. |
| [22] |
M. Sakaguchi and Y. Ohtsubo,
Markov decision processes associated with two threshold probability criteria, Journal Control Theory Applications, 11 (2013), 548-557.
doi: 10.1007/s11768-013-2194-8. |
| [23] |
Q. D. Wei and X. P. Guo,
New average optimality conditions for semi-Markov decision processes in Borel spaces, Journal of Optimization Theory and Applications, 153 (2012), 709-732.
doi: 10.1007/s10957-012-9986-8. |
| [24] |
D. J. White,
Minimising a threshold probability in discounted Markov decision processes, J. Math. Anal. Appl., 173 (1993), 634-646.
doi: 10.1006/jmaa.1993.1093. |
| [25] |
Y. H. Wu,
Bounds for the ruin probability under a Markovian modulated risk model, Communications in statistics Stochastic Models, 15 (1999), 125-136.
doi: 10.1080/15326349908807529. |
| [26] |
S. X. Yu, Y. L. Lin and P. F. Yan,
Optimization models for the first arrival target distribution function in discrete time, J. Math. Anal. Appl., 225 (1998), 193-223.
doi: 10.1006/jmaa.1998.6015. |
Figure 2.
The function $F^*_{(n_0+1)k}(2, t, \lambda)$
Figure 3.
The function $H^aF^*_{(n_0+1)k-1}(i, 10, \lambda)$
Figure 4.
The function $H^aF^*_{(n_0+1)k-1}(i, 15, \lambda)$
Figure 5.
The function $\lambda^*(i, t)$
| [1] |
Angelica Pachon, Federico Polito, Costantino Ricciuti. On discrete-time semi-Markov processes. Discrete and Continuous Dynamical Systems - B, 2021, 26 (3) : 1499-1529. doi: 10.3934/dcdsb.2020170 |
| [2] |
Mathias Staudigl. A limit theorem for Markov decision processes. Journal of Dynamics and Games, 2014, 1 (4) : 639-659. doi: 10.3934/jdg.2014.1.639 |
| [3] |
Vladimir Turetsky, Valery Y. Glizer. Optimal decision in a Statistical Process Control with cubic loss. Journal of Industrial and Management Optimization, 2021 doi: 10.3934/jimo.2021096 |
| [4] |
A. Mittal, N. Hemachandra. Learning algorithms for finite horizon constrained Markov decision processes. Journal of Industrial and Management Optimization, 2007, 3 (3) : 429-444. doi: 10.3934/jimo.2007.3.429 |
| [5] |
Yueyuan Zhang, Yanyan Yin, Fei Liu. Robust observer-based control for discrete-time semi-Markov jump systems with actuator saturation. Journal of Industrial and Management Optimization, 2021, 17 (6) : 3013-3026. doi: 10.3934/jimo.2020105 |
| [6] |
Xin Zhao, Tao Feng, Liang Wang, Zhipeng Qiu. Threshold dynamics and sensitivity analysis of a stochastic semi-Markov switched SIRS epidemic model with nonlinear incidence and vaccination. Discrete and Continuous Dynamical Systems - B, 2021, 26 (12) : 6131-6154. doi: 10.3934/dcdsb.2021010 |
| [7] |
Xia Han, Zhibin Liang, Yu Yuan, Caibin Zhang. Optimal per-loss reinsurance and investment to minimize the probability of drawdown. Journal of Industrial and Management Optimization, 2021 doi: 10.3934/jimo.2021145 |
| [8] |
Gábor Horváth, Zsolt Saffer, Miklós Telek. Queue length analysis of a Markov-modulated vacation queue with dependent arrival and service processes and exhaustive service policy. Journal of Industrial and Management Optimization, 2017, 13 (3) : 1365-1381. doi: 10.3934/jimo.2016077 |
| [9] |
Linyi Qian, Wei Wang, Rongming Wang. Risk-minimizing portfolio selection for insurance payment processes under a Markov-modulated model. Journal of Industrial and Management Optimization, 2013, 9 (2) : 411-429. doi: 10.3934/jimo.2013.9.411 |
| [10] |
Lin Xu, Rongming Wang. Upper bounds for ruin probabilities in an autoregressive risk model with a Markov chain interest rate. Journal of Industrial and Management Optimization, 2006, 2 (2) : 165-175. doi: 10.3934/jimo.2006.2.165 |
| [11] |
Jiaqin Wei, Zhuo Jin, Hailiang Yang. Optimal dividend policy with liability constraint under a hidden Markov regime-switching model. Journal of Industrial and Management Optimization, 2019, 15 (4) : 1965-1993. doi: 10.3934/jimo.2018132 |
| [12] |
Ming Yan, Hongtao Yang, Lei Zhang, Shuhua Zhang. Optimal investment-reinsurance policy with regime switching and value-at-risk constraint. Journal of Industrial and Management Optimization, 2020, 16 (5) : 2195-2211. doi: 10.3934/jimo.2019050 |
| [13] |
Vincent Renault, Michèle Thieullen, Emmanuel Trélat. Optimal control of infinite-dimensional piecewise deterministic Markov processes and application to the control of neuronal dynamics via Optogenetics. Networks and Heterogeneous Media, 2017, 12 (3) : 417-459. doi: 10.3934/nhm.2017019 |
| [14] |
Wael Bahsoun, Paweł Góra. SRB measures for certain Markov processes. Discrete and Continuous Dynamical Systems, 2011, 30 (1) : 17-37. doi: 10.3934/dcds.2011.30.17 |
| [15] |
Artur Stephan, Holger Stephan. Memory equations as reduced Markov processes. Discrete and Continuous Dynamical Systems, 2019, 39 (4) : 2133-2155. doi: 10.3934/dcds.2019089 |
| [16] |
Zhimin Zhang. On a risk model with randomized dividend-decision times. Journal of Industrial and Management Optimization, 2014, 10 (4) : 1041-1058. doi: 10.3934/jimo.2014.10.1041 |
| [17] |
Yinghui Dong, Guojing Wang. Ruin probability for renewal risk model with negative risk sums. Journal of Industrial and Management Optimization, 2006, 2 (2) : 229-236. doi: 10.3934/jimo.2006.2.229 |
| [18] |
Kebing Chen, Tiaojun Xiao. Reordering policy and coordination of a supply chain with a loss-averse retailer. Journal of Industrial and Management Optimization, 2013, 9 (4) : 827-853. doi: 10.3934/jimo.2013.9.827 |
| [19] |
H.Thomas Banks, Shuhua Hu. Nonlinear stochastic Markov processes and modeling uncertainty in populations. Mathematical Biosciences & Engineering, 2012, 9 (1) : 1-25. doi: 10.3934/mbe.2012.9.1 |
| [20] |
Thomas Kruse, Mikhail Urusov. Approximating exit times of continuous Markov processes. Discrete and Continuous Dynamical Systems - B, 2020, 25 (9) : 3631-3650. doi: 10.3934/dcdsb.2020076 |
Impact Factor:
Tools
Metrics
Other articles
by authors
[Back to Top]