Kybernetika 57 no. 3, 383-403, 2021

Properties of unique information

Johannes Rauh, Maik Schünemann and Jürgen JostDOI: 10.14736/kyb-2021-3-0383

Abstract:

We study the unique information function $UI(T:X\setminus Y)$ defined by Bertschinger et al. [4] within the framework of information decompositions. In particular, we study uniqueness and support of the solutions to the convex optimization problem underlying the definition of $UI$. We identify sufficient conditions for non-uniqueness of solutions with full support in terms of conditional independence constraints and in terms of the cardinalities of $T$, $X$ and $Y$. Our results are based on a reformulation of the first order conditions on the objective function as rank constraints on a matrix of conditional probabilities. These results help to speed up the computation of $UI(T:X\setminus Y)$, most notably when $T$ is binary. Optima in the relative interior of the optimization domain are solutions of linear equations if $T$ is binary. In the all binary case, we obtain a complete picture of where the optimizing probability distributions lie.

Keywords:

unique information, information decomposition

Classification:

94A15, 94A17

paper.pdf

References:

  1. Shun-ichi Amari and Hiroshi Nagaoka: Methods of Information Geometry. American Mathematical Society 2000.   DOI:10.1090/mmono/191
  2. P. Kr. Banerjee, E. Olbrich, J. Jost and J. Rauh: Unique informations and deficiencies. In: Proc. Allerton, 2018.   DOI:10.1109/allerton.2018.8635984
  3. P. Kr. Banerjee, J. Rauh and G. Montúfar: Computing the unique information. In: Proc. IEEE ISIT 2018, pp. 141-145   DOI:10.1109/isit.2018.8437757
  4. N. Bertschinger, J. Rauh, E. Olbrich, J. Jost and N. Ay: Quantifying unique information. Entropy 16 (2014), 4, 2161-2183.   DOI:10.3390/e16042161
  5. Persi Diaconis and Bernd Sturmfels: Algebraic algorithms for sampling from conditional distributions. Ann. Statist. 26 (1998), 363-397.   CrossRef
  6. A. Fink: The binomial ideal of the intersection axiom for conditional probabilities. J. Algebr. Combin. 33 (2011), 3, 455-463.   DOI:10.1007/s10801-010-0253-5
  7. C. Finn and J. T. Lizier: Pointwise partial information decomposition using the specificity and ambiguity lattices. Entropy 20 (2018), 4, 297.   DOI:10.3390/e20040297
  8. M. Harder, Ch. Salge and D. Polani: A bivariate measure of redundant information. Phys. Rev. E 87 (2013), 012130.   DOI:10.1103/physreve.87.012130
  9. R. Ince: Measuring multivariate redundant information with pointwise common change in surprisal. Entropy 19 (2017), 7, 318.   DOI:10.3390/e19070318
  10. R. James, J. Emenheiser and J. Crutchfield: Unique information via dependency constraints. J. Physics A 52 (2018), 1, 014002.   DOI:10.1088/1361-6463/aae6f6
  11. A. Makkeh, D. O. Theis and R. Vicente: Bivariate partial information decomposition: The optimization perspective. Entropy 19 (2017), 10, 530.   DOI:10.3390/e19100530
  12. X. Niu and Ch. Quinn: A measure of synergy, redundancy, and unique information using information geometry. In: Proc. IEEE ISIT 2019.   DOI:10.1109/isit.2019.8849724
  13. J. Rauh, P. Kr. Banerjee, E. Olbrich and J. Jost: Unique information and secret key decompositions. In: 2019 IEEE International Symposium on Information Theory (ISIT), pp. 3042-3046.   DOI:10.1109/isit.2019.8849550
  14. J. Rauh and N. Ay: Robustness, canalyzing functions and systems design. Theory Biosciences 133 (2014), 2, 63-78.   DOI:10.1007/s12064-013-0186-3
  15. J. Rauh, N. Bertschinger, E. Olbrich and J. Jost: Reconsidering unique information: Towards a multivariate information decomposition. In: Proc. IEEE ISIT 2014, pp. 2232-2236.   DOI:10.1109/isit.2014.6875230
  16. N. A. Smith, R and W. Tromble: Sampling Uniformly from the Unit Simplex. Technical Report 29, Johns Hopkins University 29, 2004.   CrossRef
  17. P. Williams and R. Beer: Nonnegative decomposition of multivariate information. arXiv:1004.2515v1.   CrossRef