Kybernetika 54 no. 6, 1218-1230, 2018

Risk-sensitive average optimality in Markov decision processes

Karel SladkýDOI: 10.14736/kyb-2018-6-1218

Abstract:

In this note attention is focused on finding policies optimizing risk-sensitive optimality criteria in Markov decision chains. To this end we assume that the total reward generated by the Markov process is evaluated by an exponential utility function with a given risk-sensitive coefficient. The ratio of the first two moments depends on the value of the risk-sensitive coefficient; if the risk-sensitive coefficient is equal to zero we speak on risk-neutral models. Observe that the first moment of the generated reward corresponds to the expectation of the total reward and the second central moment of the reward variance. For communicating Markov processes and for some specific classes of unichain processes long run risk-sensitive average reward is independent of the starting state. In this note we present necessary and sufficient condition for existence of optimal policies independent of the starting state in unichain models and characterize the class of average risk-sensitive optimal policies.

Keywords:

controlled Markov processes, finite state space, asymptotic behavior, risk-sensitive average optimality

Classification:

90C40, 93E20

paper.pdf

References:

  1. A. Arapostathis, V. S. Borkar, F. Fernandez-Gaucherand, M. K. Ghosh and S. I. Marcus: Discrete-time controlled Markov processes with average cost criterion: A survey. SIAM J. Control Optim. 31 (1993), 282-344.   DOI:10.1137/0331018
  2. J. Bather: Optimal decisions procedures for finite Markov chains, Part II. Adv. Appl. Probab. 5 (1973), 328-339.   DOI:10.2307/1426039
  3. T. D. Bielecki, D. Hernández-Hernández and S. R. Pliska: Risk-sensitive control of finite state Markov chains in discrete time, with application to portfolio management. Math. Methods Oper. Res. 50 (1999), 167-188.   DOI:10.1007/s001860050094
  4. R. Cavazos-Cadena: Value iteration and approximately optimal stationary policies in finite-state average Markov chains. Math. Methods Oper. Res. 56 (2002), 181-196.   DOI:10.1007/s001860200205
  5. R. Cavazos-Cadena: Solution to the risk-sensitive average cost optimality equation in a class of Markov decision processes with finite state space. Math. Methods Oper. Res. 57 (2003), 2, 263-285.   DOI:10.1007/s001860200256
  6. R. Cavazos-Cadena: Solution of the average cost optimality equation for finite Markov decision chains: risk-sensitive and risk-neutral criteria. Math. Methods Oper. Res. 70 (2009), 541-566.   DOI:10.1007/s00186-008-0277-y
  7. R. Cavazos-Cadena and F. Fernandez-Gaucherand: Controlled Markov chains with risk-sensitive criteria: average cost, optimality equations and optimal solutions. Math. Methods Oper. Res. 43 (1999), 121-139.   CrossRef
  8. R. Cavazos-Cadena and D. Hernández-Hernández: A characterization exponential functionals in finite Markov chains. Math. Methods Oper. Res. 60 (2004), 399-414.   DOI:10.1007/s001860400373
  9. R. Cavazos-Cadena and D. Hernández-Hernández: A characterization of the optimal risk-sensitive average cost in finite controlled Markov chains. Ann. Appl. Probab. 15 (2005), 175-212.   DOI:10.1214/105051604000000585
  10. R. Cavazos-Cadena and D. Hernández-Hernández: Necessary and sufficient conditions for a solution to the risk-sensitive Poisson equation on a finite state space. System Control Lett. 58 (2009), 254-258.   DOI:10.1016/j.sysconle.2008.11.001
  11. R. Cavazos-Cadena and R. Montes-de-Oca: The value iteration algorithm in risk-sensitive average Markov decision chains with finite state space. Math. Oper. Res. {\mi 28} (2003), 752-756.   DOI:10.1287/moor.28.4.752.20515
  12. R. Cavazos-Cadena and R. Montes-de-Oca: Nonstationary value iteration in controlled Markov chains with risk-sensitive average criterion. J. Appl. Probab. 42 (2005), 905-918.   DOI:10.1017/s0021900200000991
  13. R. Cavazos-Cadena, A. Feinberg and R. Montes-de-Oca: A note on the existence of optimal policies in total reward dynamic programs with compact action sets. Math. Oper. Res. {\mi 25} (2000), 657-666.   DOI:10.1287/moor.25.4.657.12112
  14. F. R. Gantmakher: The Theory of Matrices. Chelsea, London 1959.   CrossRef
  15. R. A. Howard: Dynamic Programming and Markov Processes. MIT Press, Cambridge, Mass. 1960.   CrossRef
  16. R. A. Howard and J. Matheson: Risk-sensitive Markov decision processes. Manag. Sci. 23 (1972), 356-369.   DOI:10.1287/mnsc.18.7.356
  17. P. Mandl: On the variance in controlled Markov chains. Kybernetika 7 (1971), 1-12.   CrossRef
  18. P. Mandl: Estimation and control in Markov chains. Adv. Appl. Probab. 6 (1974), 40-60.   DOI:10.2307/1426206
  19. H. Markowitz: Portfolio selection. J. Finance 7 (1952), 77-92.   DOI:10.1111/j.1540-6261.1952.tb01525.x
  20. H. Markowitz: Portfolio Selection - Efficient Diversification of Investments. Wiley, New York 1959.   CrossRef
  21. M. L. Puterman: Markov Decision Processes - Discrete Stochastic Dynamic Programming. Wiley, New York 1994.   DOI:10.1002/9780470316887
  22. S. M. Ross: Introduction to Stochastic Dynamic Programming. Academic Press, New York 1983.   CrossRef
  23. K. Sladký: Necessary and sufficient optimality conditions for average reward of controlled Markov chains. Kybernetika 9 (1973), 124-137.   CrossRef
  24. K. Sladký: On the set of optimal controls for Markov chains with rewards. Kybernetika 10 (1974), 526-547.   CrossRef
  25. K. Sladký: Growth rates and average optimality in risk-sensitive Markov decision chains. Kybernetika 44 (2008), 205-226.   CrossRef
  26. K. Sladký: Risk-sensitive and average optimality in Markov decision processes. In: Proc. 30th Int. Conf. Math. Meth. Economics 2012, Part II (J.Ramík and D.Stavárek, eds.), Silesian University, School of Business Administration, Karviná 2012, pp. 799-804.   DOI:10.1007/3-540-32539-5\_125
  27. K. Sladký: Risk-sensitive and mean variance optimality in Markov decision processes. Acta Oeconomica Pragensia 7 (2013), 146-161.   CrossRef
  28. N. M. van Dijk and K. Sladký: On the total reward variance for continuous-time Markov reward chains. J. Appl. Probab. 43 (2006), 1044-1052.   DOI:10.1017/s0021900200002412