Kybernetika 56 no. 5, 875-885, 2020

Maximizing the Bregman divergence from a Bregman family

Johannes RauhDOI: 10.14736/kyb-2020-5-0875

Abstract:

The problem to maximize the information divergence from an exponential family is generalized to the setting of Bregman divergences and suitably defined Bregman families.

Keywords:

exponential family, relative entropy, optimization, Bregman divergence

Classification:

94A17, 62B05, 62E15, 62E17, 52A41

paper.pdf

References:

  1. N. Ay: An information-geometric approach to a theory of pragmatic structuring. Ann. Probab. 30 (2002), 416-436.   DOI:10.1214/aop/1020107773
  2. N. Ay and A. Knauf: Maximizing multi-information. Kybernetika 42 (2006), 517-538.   CrossRef
  3. O. Barndorff-Nielsen: Information and Exponential Families in Statistical Theory. Wiley, 1978.   CrossRef
  4. I. Csiszár and F. Matúš: Closures of exponential families. Ann. Probab. 33 (2005), 582-600.   DOI:10.1214/009117904000000766
  5. I. Csiszár and F. Matúš: Generalized maximum likelihood extimates for exponential families. Probab. Theory Related Fields 141 (2008), 213-246.   DOI:10.1007/s00440-007-0084-z
  6. D. Geiger, C. Meek and B. Sturmfels: On the toric algebra of graphical models. Ann. Statist. 34 (2006), 1463-1492.   DOI:10.1214/009053606000000263
  7. F. Matúš: Maximization of information divergences from binary i.i.d. sequences. In: Proc. IPMU 2 (2004), 1303-1306.   CrossRef
  8. F. Matúš: Optimality conditions for maximizers of the information divergence from an exponential family. Kybernetika 43 (2007), 731-746.   CrossRef
  9. F. Matúš: Divergence from factorizable distributions and matroid representations by partitions. IEEE Trans. Inform. Theory 55 (2009), 5375-5381.   DOI:10.1109/tit.2009.2032806
  10. F. Matúš and N. Ay: On maximization of the information divergence from an exponential family. In: Proc. WUPES 2003, University of Economics, Prague 2003, pp. 199-204.   CrossRef
  11. F. Matúš and I. Csiszár: Generalized minimizers of convex integral functionals, Bregman distance, Pythagorean identities. Kybernetika 48 (2012), 637-689.   CrossRef
  12. F. Matúš and J. Rauh: Maximization of the information divergence from an exponential family and criticality. In: Proc. IEEE International Symposium on Information Theory (ISIT2011), 2011.   DOI:10.1109/isit.2011.6034269
  13. G. Montúfar, J. Rauh and N. Ay: Expressive power and approximation errors of Restricted Boltzmann Machines. In: Proc. {NIPS} 2011.   CrossRef
  14. G. Montúfar, J. Rauh and N. Ay: Maximal information divergence from statistical models defined by neural networks. In: Proc. GSI, 2013, pp. 759-766.   DOI:10.1007/978-3-642-40020-9\_85
  15. J. Rauh: Finding the maximizers of the information divergence from an exponential family. IEEE Trans. Inform. Theory 57 (2011), 3236-3247.   DOI:10.1109/tit.2011.2136230
  16. J. Rauh: Finding the Maximizers of the Information Divergence from an Exponential Family Ph.D. Dissertation, Universität Leipzig, 2011.   CrossRef
  17. R. T. Rockafellar: Convex Analysis. Princeton University Press, 1970.   DOI:10.1017/s0013091500010142
  18. N. Wang, J. Rauh and H. Massam: Approximating faces of marginal polytopes in discrete hierarchical models. Ann. Statist. 47 (2019), 1203-1233.   DOI:10.1214/18-aos1710}. Extended preprint version at \texttt{arXiv:1603.04843