Kybernetika 52 no. 3, 403-426, 2016

Discrete-time Markov control processes with recursive discount rates

Yofre H. García and Juan González-HernándezDOI: 10.14736/kyb-2016-3-0403

Abstract:

This work analyzes a discrete-time Markov Control Model (MCM) on Borel spaces when the performance index is the expected total discounted cost. This criterion admits unbounded costs. It is assumed that the discount rate in any period is obtained by using recursive functions and a known initial discount rate. The classic dynamic programming method for finite-horizon case is verified. Under slight conditions, the existence of deterministic non-stationary optimal policies for infinite-horizon case is proven. Also, to find deterministic non-stationary $\epsilon-$optimal policies, the value-iteration method is used. To illustrate an example of recursive functions that generate discount rates, we consider the expected values of stochastic processes, which are solutions of certain class of Stochastic Differential Equations (SDE) between consecutive periods, when the initial condition is the previous discount rate. Finally, the consumption-investment problem and the discount linear-quadratic problem are presented as examples; in both cases, the discount rates are obtained using a SDE, similar to the Vasicek short-rate model.

Keywords:

dynamic programming method, optimal stochastic control

Classification:

49L20, 93E20

paper.pdf

References:

  1. L. Arnold: Stochastic Differential Equations: Theory and Applications. John Wiley and Sons, New York 1973.   DOI:10.1002/zamm.19770570413
  2. R. Ash and C. Doléans-Dade: Probability and Measure Theory. Academic Press, San Diego 2000.   CrossRef
  3. R. Bellman: Dynamic Programming. Princeton Univ. Press, New Jersey 1957.   CrossRef
  4. D. Bertsekas and S. Shreve: Stochastic Optimal Control: The Discrete Time Case. Athena Scientific, Massachusetts 1996.   CrossRef
  5. D. Brigo and F. Mercurio: Interest Rate Models Theory and Practice. Springer-Verlag, New York 2001.   DOI:10.1007/978-3-662-04553-4
  6. F. Black and P. Karasinski: Bond and option pricing when short rates are lognormal. Financ. Anal. J. 47 (1991), 4, 52-59.   DOI:10.2469/faj.v47.n4.52
  7. Y. Carmon and A. Shwartz: Markov decision processes with exponentially representable discounting. Oper. Res. Lett. 37 (2009), 51-55.   DOI:10.1016/j.orl.2008.10.005
  8. E. Della Vecchia, S. Di Marco and F. Vidal: Dynamic programming for variable discounted Markov decision problems. In: Jornadas Argentinas de Informática e Investigación Operativa (43JAIIO) - XII Simposio Argentino de Investigación Operativa (SIO), Buenos Aires, 2014, pp. 50-62.   CrossRef
  9. J. Cox, J. Ingersoll and S. Ross: A theory of the term structure of interest rates. Econometrica 53 (1985), 385-407.   DOI:10.2307/1911242
  10. U. Dothan: On the term structure interest rates. J. Financ. Econ. 6 (1978), 59-69.   DOI:10.1016/0304-405x(78)90020-x
  11. E. Feinberg and A. Shwartz: Markov decision models with weighted discounted criteria. J. Finan. Econ. 19 (1994), 152-168.   DOI:10.1287/moor.19.1.152
  12. J. González-Hernández, R. López-Martínez and J. Pérez-Hernández: Markov control processes with randomized discounted cost. Math. Method Oper. Res. 65 (2006), 1, 27-44.   DOI:10.1007/s00186-006-0092-2
  13. J. González-Hernández, R. López-Martínez and A. Minjarez-Sosa: Adaptive policies for stochastic systems under a randomized discounted cost criterion. Bol. Soc. Mat. Mex. 14 (2008), 3, 149-163.   CrossRef
  14. J. González-Hernández, R. López-Martínez and A. Minjarez-Sosa: Approximation, estimation and control of stochastic systems under a randomized discounted cost criterion. Kybernetika 45 (2008), 5, 737-754.   CrossRef
  15. X. Guo, A. Hernández-Del-Valle and O. Hernández-Lerma: First passage problems for a non-stationary discrette-time stochastic control systems. Eur. J. Control 15 (2012), 7, 528-538.   DOI:10.3166/ejc.18.528-538
  16. O. Hernández-Lerma and J. B. Lasserre: Discrete-Time Markov Control Processes. Basic Optimality Criteria. Springer-Verlag, New York 1996.   DOI:10.1007/978-1-4612-0729-0
  17. J. Minjarez-Sosa: Markov control models with unknown random sate-action-dependent discount factors. TOP 23 (2015), 3, 743-772.   DOI:10.1007/s11750-015-0360-5
  18. K. Hinderer: Foundations of non-stationary dynamical programming with discrete time parameter. In: Lecture Notes Operations Research (M. Bechmann and H. Künzi, eds.), Springer-Verlag 33, Zürich 1970.   DOI:10.1007/978-3-642-46229-0
  19. T. Ho and S. Lee: Term structure movements and pricing interest rate claims. J. Financ. 41 (1986), 1011-1029.   DOI:10.1111/j.1540-6261.1986.tb02528.x
  20. J. Hull: Options, Futures and other Derivatives. Sixth edition. Prentice Hall, New Jersey 2006.   CrossRef
  21. J. Hull and A. White: Pricing interest rate derivative securities. Rev. Financ. Stud. 3 (1990), 573-592.   DOI:10.1093/rfs/3.4.573
  22. F. Mercurio and J. Moraleda: A family of humped volatility models. Eur. J. Finance 7 (2001), 93-116.   DOI:10.1080/13518470151141440
  23. R. Rendleman and B. Bartter: The pricing of options on debt securities. J. Financ. Quant. Anal. 15 (1980), 11-24.   DOI:10.2307/2979016
  24. U. Rieder: Measurable selection theorems for optimization problems. Manuscripta Math. 24 (1978), 115-131.   DOI:10.1007/bf01168566
  25. M. Schäl: Conditions for optimality in dynamic programming and for the limit of n-stage optimal policies to be optimal. Z. Wahrscheinlichkeit 32 (1975), 179-196.   DOI:10.1007/bf00532612
  26. O. Vasicek: An equilibrium characterization of the term structure. J. Financ. Econ. 5 (1977), 177-188.   DOI:10.1016/0304-405x(77)90016-2
  27. Q. Wei and X. Guo X: Markov decision processes with state-dependent discount factors and unbounded rewards costs. Oper. Res. Lett. 39 (2011), 369-374.   DOI:10.1016/j.orl.2011.06.014
  28. L. Ye and X. Guo: Continuous-time Markov decision processes with state-dependent discount factors. Acta Appl. Math. 121 (2012), 1, 5-27.   DOI:10.1007/s10440-012-9669-3
  29. Y. Zhang: Convex analytic approach to constrained discounted Markov decision processes with non-constant discount factors. TOP 21 (2013), 2, 378-408.   DOI:10.1007/s11750-011-0186-8