Kybernetika 56 no. 5, 875-885, 2020

Maximizing the Bregman divergence from a Bregman family

Johannes RauhDOI: 10.14736/kyb-2020-5-0875


The problem to maximize the information divergence from an exponential family is generalized to the setting of Bregman divergences and suitably defined Bregman families.


exponential family, relative entropy, optimization, Bregman divergence


94A17, 62B05, 62E15, 62E17, 52A41


  1. N. Ay: An information-geometric approach to a theory of pragmatic structuring. Ann. Probab. 30 (2002), 416-436.   DOI:10.1214/aop/1020107773
  2. N. Ay and A. Knauf: Maximizing multi-information. Kybernetika 42 (2006), 517-538.   CrossRef
  3. O. Barndorff-Nielsen: Information and Exponential Families in Statistical Theory. Wiley, 1978.   CrossRef
  4. I. Csiszár and F. Matúš: Closures of exponential families. Ann. Probab. 33 (2005), 582-600.   DOI:10.1214/009117904000000766
  5. I. Csiszár and F. Matúš: Generalized maximum likelihood extimates for exponential families. Probab. Theory Related Fields 141 (2008), 213-246.   DOI:10.1007/s00440-007-0084-z
  6. D. Geiger, C. Meek and B. Sturmfels: On the toric algebra of graphical models. Ann. Statist. 34 (2006), 1463-1492.   DOI:10.1214/009053606000000263
  7. F. Matúš: Maximization of information divergences from binary i.i.d. sequences. In: Proc. IPMU 2 (2004), 1303-1306.   CrossRef
  8. F. Matúš: Optimality conditions for maximizers of the information divergence from an exponential family. Kybernetika 43 (2007), 731-746.   CrossRef
  9. F. Matúš: Divergence from factorizable distributions and matroid representations by partitions. IEEE Trans. Inform. Theory 55 (2009), 5375-5381.   DOI:10.1109/tit.2009.2032806
  10. F. Matúš and N. Ay: On maximization of the information divergence from an exponential family. In: Proc. WUPES 2003, University of Economics, Prague 2003, pp. 199-204.   CrossRef
  11. F. Matúš and I. Csiszár: Generalized minimizers of convex integral functionals, Bregman distance, Pythagorean identities. Kybernetika 48 (2012), 637-689.   CrossRef
  12. F. Matúš and J. Rauh: Maximization of the information divergence from an exponential family and criticality. In: Proc. IEEE International Symposium on Information Theory (ISIT2011), 2011.   DOI:10.1109/isit.2011.6034269
  13. G. Montúfar, J. Rauh and N. Ay: Expressive power and approximation errors of Restricted Boltzmann Machines. In: Proc. {NIPS} 2011.   CrossRef
  14. G. Montúfar, J. Rauh and N. Ay: Maximal information divergence from statistical models defined by neural networks. In: Proc. GSI, 2013, pp. 759-766.   DOI:10.1007/978-3-642-40020-9\_85
  15. J. Rauh: Finding the maximizers of the information divergence from an exponential family. IEEE Trans. Inform. Theory 57 (2011), 3236-3247.   DOI:10.1109/tit.2011.2136230
  16. J. Rauh: Finding the Maximizers of the Information Divergence from an Exponential Family Ph.D. Dissertation, Universität Leipzig, 2011.   CrossRef
  17. R. T. Rockafellar: Convex Analysis. Princeton University Press, 1970.   DOI:10.1017/s0013091500010142
  18. N. Wang, J. Rauh and H. Massam: Approximating faces of marginal polytopes in discrete hierarchical models. Ann. Statist. 47 (2019), 1203-1233.   DOI:10.1214/18-aos1710}. Extended preprint version at \texttt{arXiv:1603.04843