American Institute of Mathematical Sciences

Advanced
March  2019, 14(1): 43-52. doi: 10.3934/nhm.2019003

Optimal stopping for response-guided dosing

Jakob Kotas

Department of Mathematics, University of Portland, 5000 N Willamette Blvd, Portland, OR 97203, USA

Received  April 2018 Published  January 2019

Fund Project: The first author was funded in part by the National Science Foundation through grant #CMMI 1536717 when the author was a doctoral student at the University of Washington.

In response-guided dosing (RGD), the goal is to make optimal dosing decisions based on the stochastic evolution of a patient's disease condition. Typically, RGD is formulated as a finite-horizon problem with decision-making occurring over a predetermined time frame. In this paper we relax the latter assumption to allow for the possibility of ending treatment early. This could occur due to remission of the disease or a finding of futility in treatment of the disease. Our framework is formulated as a stochastic dynamic program (DP) where a stop/do-not-stop decision is made in discrete sessions, and if stopping is not chosen, an optimal dose is determined for that session. Numerical simulations for rheumatoid arthritis are presented, and monotonicity of the stop/do-not-stop threshold with respect to time is proven.

Keywords: Optimal stopping, data-driven decision-making, Markov decision process, medical dosing, rheumatoid arthritis.
Mathematics Subject Classification: Primary: 60G40, 62L15; Secondary: 92C50.
Citation: Jakob Kotas. Optimal stopping for response-guided dosing. Networks & Heterogeneous Media, 2019, 14 (1) : 43-52. doi: 10.3934/nhm.2019003
References:
[1]

D. A. Berry, P. Müller, A. P. Grieve, M. Smith, T. Parke, R. Blazek, N. Mitchard and M. Krams, Adaptive Bayesian designs for dose-ranging drug trials, Case Studies in Bayesian Statistics, 5 (2002), 99-181.  doi: 10.1007/978-1-4613-0035-9_2.  Google Scholar

[2]

D. P. Bertsekas, Dynamic Programming and Optimal Control, 4th edition, Athena Scientific, Cambridge, MA, 2012.  Google Scholar

[3]

G. L. Davis, J. B. Wong, J. G. McHutchison, M. P. Manns, J. Harvey and J. Albrecht, Early virologic response to treatment with peginterferon alfa-2b plus ribavirin in patients with chronic hepatitis C, Hepatology, 38 (2003), 645-652.  doi: 10.1053/jhep.2003.50364.  Google Scholar

[4]

M. Flendrie, M. C. W. Creemers and P. L. C. M. van Riel, Titration of infliximab treatment in rheumatoid arthritis patients based on response patterns, Rheumatology, 46 (2007), 146-149.  doi: 10.1093/rheumatology/kel173.  Google Scholar

[5]

B. I. Grigelionis and A. N. Shiryaev, On Stefan's problem and optimal stopping rules for Markov processes, Theory of Probability and its Applications, 4 (1966), 541-558.   Google Scholar

[6]

I. M. Jacobson, P. Marcellin, S. Zeuzem, M. S. Sulkowski, R. Esteban, F. Poordad, S. Bruno, M. H. Burroughs, L. D. Pedicone, N. Boparai, W. Deng, M. J. DiNubile, K. M. Gottesdiener, C. A. Brass, J. K. Albrecht and J.-P. Bronowicki, Refinement of stopping rules during treatment of hepatitis C genotype 1 infection with boceprevir and peginterferon/ribavirin, Hepatology, 56 (2012), 567-575.  doi: 10.1002/hep.25865.  Google Scholar

[7]

J. Kotas and A. Ghate, Response-guided dosing for rheumatoid arthritis, IIE Transactions on Healthcare Systems Engineering, 6 (2016), 1-21.   Google Scholar

[8]

J. Kotas and A. Ghate, Bayesian learning of dose-response parameters from a cohort under response-guided dosing, European Journal of Operational Research, 265 (2018), 328-343.  doi: 10.1016/j.ejor.2017.07.034.  Google Scholar

[9]

T. M. Kuijper, F. B. G. Lamers-Karnebeek, J. W. G. Jacobs, J. M. W. Hazes and J.J. Luime, Flare rate in patients with rheumatoid arthritis in low disease activity or remission when tapering or stopping synthetic or biologic DMARD: a systematic review, Journal of Rheumatology, 42 (2015), 2012-2022.  doi: 10.3899/jrheum.141520.  Google Scholar

[10]

L. Lojo, G. Bonilla, D. Peiteado, A. Villalba, C. Plasencia, L. Nuno, A. Balsa and E. Martin-Mola, Down-titration and discontinuation of infliximab, adalimumab and etanercept in established rheumatoid arthritis, Annals of the Rheumatic Diseases, 72 (2013), 237.  doi: 10.1136/annrheumdis-2013-eular.744.  Google Scholar

[11]

E. Louis, G. Vernier-Massouille, J.-C. Grimaud, Y. Bouhnik, D. Laharie, J.-L. Dupas, H. Pillant, L. Picon, M. Veyrac, M. Flamant, G. Savoye, R. Jian, M. De Vos, G. Paintaud, E. Piver, J.-F. Colombel, J.-Y. Mary and M. Lemann, Infliximab discontinuation in Crohn's disease patients in stable remission on combined therapy with immunosuppressors: a prospective ongoing cohort study, Gastroenterology, 136 (2009), A146.  doi: 10.1016/S0016-5085(09)60659-4.  Google Scholar

[12]

M. Marino, E. Zucchi, M. Fabbro, I. Lodolo, R. Maieron, S. Vadalà and M. Zilli, Outcome of infliximab discontinuation in IBD patients and therapy rechallenging in relapsers: Single centre preliminary data, Journal of Crohn's and Colitis, 8 (2014), S232-S233.  doi: 10.1016/S1873-9946(14)60521-3.  Google Scholar

[13]

P. Müller, D. A. Berry, A. P. Grieve and M. Krams, A Bayesian decision-theoretic dose-finding trial, Decision Analysis, 3 (2006), 197-207.   Google Scholar

[14] M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc., New York, 1994.   Google Scholar
[15]

K. E. Sherman, S. L. Flamm, N. H. Afdhal, D. R. Nelson, M. S. Sulkowski, G. T. Everson, M. W. Fried, M. Adler, H. W. Reesink, M. Martin, A. J. Sankoh, N. Adda, R. S. Kauffman, S. George, C. I. Wright and F. Poordad for the ILLUMINATE study team, Response-guided telaprevir combination treatment for hepatitis C virus infection, The New England Journal of Medicine, 365 (2011), 1014-1024.  doi: 10.1056/NEJMoa1014463.  Google Scholar

[16] A. N. Shiryaev, Optimal Stopping Rules, $1^{st}$ edition, Springer-Verlag, Berlin Heidelberg, 2008.   Google Scholar
[17]

W. Slob, Dose-response modeling of continuous endpoints, Toxicological Sciences, 66 (2002), 298-312.  doi: 10.1093/toxsci/66.2.298.  Google Scholar

[18]

J. Smolen, A. Beaulieu, A. Rubbert-Roth, C. Ramos-Remus, J. Rovensky, E. Alecock, T. Woodworth, R. Alten and OP TION trial investigators, Effect of interleukin-6 receptor inhibition with tocilizumab in patients with rheumatoid arthritis (OPTION study): a double-blind, placebo-controlled, randomised trial, The Lancet, 371 (2008), 987-997.  doi: 10.1016/S0140-6736(08)60453-5.  Google Scholar

[19]

A. Sofi, A. Ali, S. A. Khuder and A. Nawras, Meta-analysis- maintenance of remission following discontinuation of infliximab in patients with Crohn's disease, Gastroenterology, 144 (2013), S637.  doi: 10.1016/S0016-5085(13)62356-2.  Google Scholar

[20]

M. J. Sonneveld, B. E. Hansen, T. Piratvisuth, J.-D. Jia, S. Zeuzem, E. Gane, Y.-F. Law, Q. Xie, E. J. Heathcote, H.L.-Y. Chan and H.L.A. Janssen, Response-guided peginterferon therapy in hepatitis B e antigen-positive chronic hepatitis B using serum hepatitis B surface antigen levels, Hepatology, 58 (2013), 872-880.  doi: 10.1002/hep.26436.  Google Scholar

[21]

M. J. Sonneveld, V. Rijckborst, C. A. Boucher, B. E. Hansen and H.L. Janssen, Prediction of sustained response to peginterferon alfa-2b for hepatitis B e antigen-positive chronic hepatitis B using on-treatment hepatitis B surface antigen decline, Hepatology, 52 (2012), 1251-1257.  doi: 10.1002/hep.23844.  Google Scholar

[22]

S. ten Wolde, F. C. Breedveld, B. A. C. Dijkmans, J. Hermans, J. P. Vandenbroucke, M. A. F. J. van de Laar, H. M. Markusse, M. Janssen and H. R. van den Brink, Randomised placebo-controlled study of stopping second-line drugs in rheumatoid arthritis, The Lancet, 347 (1996), 347-352.   Google Scholar

[23]

A. van der Maas, W. Kievit, B. J. F. van den Bemt, F. H. J. van den Hoogen, P. L. C. M. van Riel and A. A. den Broeder, Down-titration and discontinuation of infliximab in rheumatoid arthritis patients with stable low disease activity and stable treatment: an observational cohort study, Annals of Rheumatoid Disease, 71 (2012), 1849-1854.   Google Scholar

[24]

N. van Herwaarden, A. A. den Broeder, W. Jacobs, A. van der Maas, J. W. Bijlsma, R. F. van Vollenhoven and B. J. van den Bemt, Down-titration and discontinuation strategies of tumor necrosis factor-blocking agents for rheumatoid arthritis patients with low disease activity, Cochrane Database of Systematic Reviews, 9 (2013).  doi: 10.1002/14651858.CD010455.  Google Scholar

show all references

References:
[1]

D. A. Berry, P. Müller, A. P. Grieve, M. Smith, T. Parke, R. Blazek, N. Mitchard and M. Krams, Adaptive Bayesian designs for dose-ranging drug trials, Case Studies in Bayesian Statistics, 5 (2002), 99-181.  doi: 10.1007/978-1-4613-0035-9_2.  Google Scholar

[2]

D. P. Bertsekas, Dynamic Programming and Optimal Control, 4th edition, Athena Scientific, Cambridge, MA, 2012.  Google Scholar

[3]

G. L. Davis, J. B. Wong, J. G. McHutchison, M. P. Manns, J. Harvey and J. Albrecht, Early virologic response to treatment with peginterferon alfa-2b plus ribavirin in patients with chronic hepatitis C, Hepatology, 38 (2003), 645-652.  doi: 10.1053/jhep.2003.50364.  Google Scholar

[4]

M. Flendrie, M. C. W. Creemers and P. L. C. M. van Riel, Titration of infliximab treatment in rheumatoid arthritis patients based on response patterns, Rheumatology, 46 (2007), 146-149.  doi: 10.1093/rheumatology/kel173.  Google Scholar

[5]

B. I. Grigelionis and A. N. Shiryaev, On Stefan's problem and optimal stopping rules for Markov processes, Theory of Probability and its Applications, 4 (1966), 541-558.   Google Scholar

[6]

I. M. Jacobson, P. Marcellin, S. Zeuzem, M. S. Sulkowski, R. Esteban, F. Poordad, S. Bruno, M. H. Burroughs, L. D. Pedicone, N. Boparai, W. Deng, M. J. DiNubile, K. M. Gottesdiener, C. A. Brass, J. K. Albrecht and J.-P. Bronowicki, Refinement of stopping rules during treatment of hepatitis C genotype 1 infection with boceprevir and peginterferon/ribavirin, Hepatology, 56 (2012), 567-575.  doi: 10.1002/hep.25865.  Google Scholar

[7]

J. Kotas and A. Ghate, Response-guided dosing for rheumatoid arthritis, IIE Transactions on Healthcare Systems Engineering, 6 (2016), 1-21.   Google Scholar

[8]

J. Kotas and A. Ghate, Bayesian learning of dose-response parameters from a cohort under response-guided dosing, European Journal of Operational Research, 265 (2018), 328-343.  doi: 10.1016/j.ejor.2017.07.034.  Google Scholar

[9]

T. M. Kuijper, F. B. G. Lamers-Karnebeek, J. W. G. Jacobs, J. M. W. Hazes and J.J. Luime, Flare rate in patients with rheumatoid arthritis in low disease activity or remission when tapering or stopping synthetic or biologic DMARD: a systematic review, Journal of Rheumatology, 42 (2015), 2012-2022.  doi: 10.3899/jrheum.141520.  Google Scholar

[10]

L. Lojo, G. Bonilla, D. Peiteado, A. Villalba, C. Plasencia, L. Nuno, A. Balsa and E. Martin-Mola, Down-titration and discontinuation of infliximab, adalimumab and etanercept in established rheumatoid arthritis, Annals of the Rheumatic Diseases, 72 (2013), 237.  doi: 10.1136/annrheumdis-2013-eular.744.  Google Scholar

[11]

E. Louis, G. Vernier-Massouille, J.-C. Grimaud, Y. Bouhnik, D. Laharie, J.-L. Dupas, H. Pillant, L. Picon, M. Veyrac, M. Flamant, G. Savoye, R. Jian, M. De Vos, G. Paintaud, E. Piver, J.-F. Colombel, J.-Y. Mary and M. Lemann, Infliximab discontinuation in Crohn's disease patients in stable remission on combined therapy with immunosuppressors: a prospective ongoing cohort study, Gastroenterology, 136 (2009), A146.  doi: 10.1016/S0016-5085(09)60659-4.  Google Scholar

[12]

M. Marino, E. Zucchi, M. Fabbro, I. Lodolo, R. Maieron, S. Vadalà and M. Zilli, Outcome of infliximab discontinuation in IBD patients and therapy rechallenging in relapsers: Single centre preliminary data, Journal of Crohn's and Colitis, 8 (2014), S232-S233.  doi: 10.1016/S1873-9946(14)60521-3.  Google Scholar

[13]

P. Müller, D. A. Berry, A. P. Grieve and M. Krams, A Bayesian decision-theoretic dose-finding trial, Decision Analysis, 3 (2006), 197-207.   Google Scholar

[14] M. L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming, John Wiley & Sons, Inc., New York, 1994.   Google Scholar
[15]

K. E. Sherman, S. L. Flamm, N. H. Afdhal, D. R. Nelson, M. S. Sulkowski, G. T. Everson, M. W. Fried, M. Adler, H. W. Reesink, M. Martin, A. J. Sankoh, N. Adda, R. S. Kauffman, S. George, C. I. Wright and F. Poordad for the ILLUMINATE study team, Response-guided telaprevir combination treatment for hepatitis C virus infection, The New England Journal of Medicine, 365 (2011), 1014-1024.  doi: 10.1056/NEJMoa1014463.  Google Scholar

[16] A. N. Shiryaev, Optimal Stopping Rules, $1^{st}$ edition, Springer-Verlag, Berlin Heidelberg, 2008.   Google Scholar
[17]

W. Slob, Dose-response modeling of continuous endpoints, Toxicological Sciences, 66 (2002), 298-312.  doi: 10.1093/toxsci/66.2.298.  Google Scholar

[18]

J. Smolen, A. Beaulieu, A. Rubbert-Roth, C. Ramos-Remus, J. Rovensky, E. Alecock, T. Woodworth, R. Alten and OP TION trial investigators, Effect of interleukin-6 receptor inhibition with tocilizumab in patients with rheumatoid arthritis (OPTION study): a double-blind, placebo-controlled, randomised trial, The Lancet, 371 (2008), 987-997.  doi: 10.1016/S0140-6736(08)60453-5.  Google Scholar

[19]

A. Sofi, A. Ali, S. A. Khuder and A. Nawras, Meta-analysis- maintenance of remission following discontinuation of infliximab in patients with Crohn's disease, Gastroenterology, 144 (2013), S637.  doi: 10.1016/S0016-5085(13)62356-2.  Google Scholar

[20]

M. J. Sonneveld, B. E. Hansen, T. Piratvisuth, J.-D. Jia, S. Zeuzem, E. Gane, Y.-F. Law, Q. Xie, E. J. Heathcote, H.L.-Y. Chan and H.L.A. Janssen, Response-guided peginterferon therapy in hepatitis B e antigen-positive chronic hepatitis B using serum hepatitis B surface antigen levels, Hepatology, 58 (2013), 872-880.  doi: 10.1002/hep.26436.  Google Scholar

[21]

M. J. Sonneveld, V. Rijckborst, C. A. Boucher, B. E. Hansen and H.L. Janssen, Prediction of sustained response to peginterferon alfa-2b for hepatitis B e antigen-positive chronic hepatitis B using on-treatment hepatitis B surface antigen decline, Hepatology, 52 (2012), 1251-1257.  doi: 10.1002/hep.23844.  Google Scholar

[22]

S. ten Wolde, F. C. Breedveld, B. A. C. Dijkmans, J. Hermans, J. P. Vandenbroucke, M. A. F. J. van de Laar, H. M. Markusse, M. Janssen and H. R. van den Brink, Randomised placebo-controlled study of stopping second-line drugs in rheumatoid arthritis, The Lancet, 347 (1996), 347-352.   Google Scholar

[23]

A. van der Maas, W. Kievit, B. J. F. van den Bemt, F. H. J. van den Hoogen, P. L. C. M. van Riel and A. A. den Broeder, Down-titration and discontinuation of infliximab in rheumatoid arthritis patients with stable low disease activity and stable treatment: an observational cohort study, Annals of Rheumatoid Disease, 71 (2012), 1849-1854.   Google Scholar

[24]

N. van Herwaarden, A. A. den Broeder, W. Jacobs, A. van der Maas, J. W. Bijlsma, R. F. van Vollenhoven and B. J. van den Bemt, Down-titration and discontinuation strategies of tumor necrosis factor-blocking agents for rheumatoid arthritis patients with low disease activity, Cochrane Database of Systematic Reviews, 9 (2013).  doi: 10.1002/14651858.CD010455.  Google Scholar

Figure 1.  Optimal policy of tocilizumab dosing for rheumatoid arthritis. The cost functions are $ c(d) = 0.028557d + b $, with fixed cost parameter $ b $ varying among subfigures. A dose of $ -1 $ indicates a decision to stop treatment
Figure Options

Download full-size image

Download as PowerPoint slide

Figure 2.  Optimal policy with the cost function $ c(d) = 0.028557d + 0.1 $. The plot is a zoomed-in version of Figure Figure 1c around the threshold area between stopping and not stopping, with refined discretization. A dose of -1 indicates a decision to stop
Figure Options

Download full-size image

Download as PowerPoint slide

[1]

Harish Garg. Novel correlation coefficients under the intuitionistic multiplicative environment and their applications to decision-making process. Journal of Industrial & Management Optimization, 2018, 14 (4) : 1501-1519. doi: 10.3934/jimo.2018018
[2]

Harish Garg. Some robust improved geometric aggregation operators under interval-valued intuitionistic fuzzy environment for multi-criteria decision-making process. Journal of Industrial & Management Optimization, 2018, 14 (1) : 283-308. doi: 10.3934/jimo.2017047
[3]

Ana F. Carazo, Ignacio Contreras, Trinidad Gómez, Fátima Pérez. A project portfolio selection problem in a group decision-making context. Journal of Industrial & Management Optimization, 2012, 8 (1) : 243-261. doi: 10.3934/jimo.2012.8.243
[4]

Feyza Gürbüz, Panos M. Pardalos. A decision making process application for the slurry production in ceramics via fuzzy cluster and data mining. Journal of Industrial & Management Optimization, 2012, 8 (2) : 285-297. doi: 10.3934/jimo.2012.8.285
[5]

Jian Jin, Weijian Mi. An AIMMS-based decision-making model for optimizing the intelligent stowage of export containers in a single bay. Discrete & Continuous Dynamical Systems - S, 2019, 12 (4&5) : 1101-1115. doi: 10.3934/dcdss.2019076
[6]

Xue Yan, Heap-Yih Chong, Jing Zhou, Zhaohan Sheng, Feng Xu. Fairness preference based decision-making model for concession period in PPP projects. Journal of Industrial & Management Optimization, 2020, 16 (1) : 11-23. doi: 10.3934/jimo.2018137
[7]

Mathias Staudigl. A limit theorem for Markov decision processes. Journal of Dynamics & Games, 2014, 1 (4) : 639-659. doi: 10.3934/jdg.2014.1.639
[8]

Saber Saati, Adel Hatami-Marbini, Per J. Agrell, Madjid Tavana. A common set of weight approach using an ideal decision making unit in data envelopment analysis. Journal of Industrial & Management Optimization, 2012, 8 (3) : 623-637. doi: 10.3934/jimo.2012.8.623
[9]

Gleb Beliakov. Construction of aggregation operators for automated decision making via optimal interpolation and global optimization. Journal of Industrial & Management Optimization, 2007, 3 (2) : 193-208. doi: 10.3934/jimo.2007.3.193
[10]

Haiying Liu, Wenjie Bi, Kok Lay Teo, Naxing Liu. Dynamic optimal decision making for manufacturers with limited attention based on sparse dynamic programming. Journal of Industrial & Management Optimization, 2019, 15 (2) : 445-464. doi: 10.3934/jimo.2018050
[11]

G.S. Liu, J.Z. Zhang. Decision making of transportation plan, a bilevel transportation problem approach. Journal of Industrial & Management Optimization, 2005, 1 (3) : 305-314. doi: 10.3934/jimo.2005.1.305
[12]

Ruiyue Lin, Zhiping Chen, Zongxin Li. A new approach for allocating fixed costs among decision making units. Journal of Industrial & Management Optimization, 2016, 12 (1) : 211-228. doi: 10.3934/jimo.2016.12.211
[13]

Hamed Fazlollahtabar, Mohammad Saidi-Mehrabad. Optimizing multi-objective decision making having qualitative evaluation. Journal of Industrial & Management Optimization, 2015, 11 (3) : 747-762. doi: 10.3934/jimo.2015.11.747
[14]

Michael Herty, Adrian Fazekas, Giuseppe Visconti. A two-dimensional data-driven model for traffic flow on highways. Networks & Heterogeneous Media, 2018, 13 (2) : 217-240. doi: 10.3934/nhm.2018010
[15]

Matthew O. Williams, Clarence W. Rowley, Ioannis G. Kevrekidis. A kernel-based method for data-driven koopman spectral analysis. Journal of Computational Dynamics, 2015, 2 (2) : 247-265. doi: 10.3934/jcd.2015005
[16]

A. Mittal, N. Hemachandra. Learning algorithms for finite horizon constrained Markov decision processes. Journal of Industrial & Management Optimization, 2007, 3 (3) : 429-444. doi: 10.3934/jimo.2007.3.429
[17]

Bin Dan, Huali Gao, Yang Zhang, Ru Liu, Songxuan Ma. Integrated order acceptance and scheduling decision making in product service supply chain with hard time windows constraints. Journal of Industrial & Management Optimization, 2018, 14 (1) : 165-182. doi: 10.3934/jimo.2017041
[18]

Weichao Yue, Weihua Gui, Xiaofang Chen, Zhaohui Zeng, Yongfang Xie. Evaluation strategy and mass balance for making decision about the amount of aluminum fluoride addition based on superheat degree. Journal of Industrial & Management Optimization, 2017, 13 (5) : 1-22. doi: 10.3934/jimo.2018169
[19]

Gholam Hassan Shirdel, Somayeh Ramezani-Tarkhorani. A new method for ranking decision making units using common set of weights: A developed criterion. Journal of Industrial & Management Optimization, 2017, 13 (5) : 1-19. doi: 10.3934/jimo.2018171
[20]

Muhammad Qiyas, Saleem Abdullah, Shahzaib Ashraf, Saifullah Khan, Aziz Khan. Triangular picture fuzzy linguistic induced ordered weighted aggregation operators and its application on decision making problems. Mathematical Foundations of Computing, 2019, 2 (3) : 183-201. doi: 10.3934/mfc.2019013

2018 Impact Factor: 0.871

Tools

Metrics

  • PDF downloads (29)
  • HTML views (249)
  • Cited by (0)

Other articles
by authors

[Back to Top]

Copyright © 2019 American Institute of Mathematical Sciences