Kybernetika 47 no. 6, 840-854, 2011

Why L1 view and what is next?

László Györfi and Adam Krzyżak

Abstract:

N. N. Cencov wrote a commentary chapter included in the Appendix of the Russian translation of the Devroye and Györfi book [15] collecting some arguments supporting the <span class="tex">L<sub>1</sub></span> view of density estimation. The Cencov's work is available in Russian only and it hasn't been translated, so late Igor Vajda decided to translate the Cencov's paper and to add some remarks on the occasion of organizing the session "25 Years of the <span class="tex">L<sub>1</sub></span> Density Estimation" at the Prague Stochastics 2010 Symposium. In this paper we complete his task, i.e., we translate the Cencov's chapter and insert some remarks on the related literature focusing primarily on Igor's results. We would also like to acknowledge the excellent work of Alexandre Tsybakov who translated the Devroye and Györfi book in Russian, annotated it with valuable comments and included some related references published in Russian only.

Keywords:

Cencov's comments, inverse problems in distribution estimation, $L_1$ density estimation, variational distance, $\phi $-divergence

Classification:

62G08, 62G20

References:

  1. S. Abou-Jaoude: Conditions nécessaires et suffisantes de convergence $L_1$ en probabilité de l'histogramme pour une densité. Ann. Inst. H. Poincaré XII (1976), 213-231.   CrossRef
  2. O. Barndorff-Nielsen: Information and Exponential Families in Statistical Theory. Wiley, 1978.   CrossRef
  3. A. R. Barron, L. Györfi and E. C. van der Meulen: Distribution estimation consistent in total variation and two types of information divergence. IEEE Trans. Inform. Theory 38 (1992), 1437-1454.   CrossRef
  4. N. N. Cencov: Estimation of unknown density function from observations. (in Russian) Trans. SSSR Acad. Sci. 147 (1962), 45-48.   CrossRef
  5. N. N. Cencov: Categories of mathematical statistics. (in Russian) Trans. SSSR Acad. Sci. 164 (1965), 511-514.   CrossRef
  6. N. N. Cencov: General theory of exponential families of distribution functions. Theory Probab. Appl. 11 (1966), 483-494.   CrossRef
  7. N. N. Cencov: Asymmetric distance between distribution functions, entropy and Pithagoras theorem. (in Russian) Math. Notes 4 (1968), 323-332.   CrossRef
  8. N. N. Cencov: Statistical Decision Rules and Optimal Inference. (in Russian) Nauka, Moscow 1972.   CrossRef
  9. N. N. Cencov: Algebraic foundation of mathematical statistics. Math. Operationsforsch. Statist., Ser. Statistics 9 (1978), 267-276.   CrossRef
  10. N. N. Cencov: On basic concepts of mathematical statistics. Banach Center Publ. 6 (1980), 85-94.   CrossRef
  11. N. N. Cencov: On correctness of the pointwise estimation problem. (in Russian) Theory Probab. Appl. 26 (1981) 15-31.   CrossRef
  12. I. Csiszár and J. Fischer: Informationsentfernungen im Raum der Wahscheinlichkeitsverteilungen. Publ. Math. Inst. Hungar. Acad. Sci. 7 (1962), 159-180.   CrossRef
  13. I. Csiszár: Information-type measures of divergence of probability distributions and indirect observations. Studia Sci. Math. Hungar. 2 (1967), 299-318.   CrossRef
  14. I. Csiszár: On topological properties of $f$-divergence. Studia Sci. Math. Hungar. 2 (1967), 329-339.   CrossRef
  15. L. Devroye and L. Györfi: Nonparametric Density Estimation: The $L_1$ View. Wiley, 1985. Russian translation: Mir, Moscow, 1988 (Translated from English to Russian by A. Tsybakov).   CrossRef
  16. L. Devroye and L. Györfi: No empirical measure can converge in the total variation sense for all distribution. Ann. Statist. 18 (1990), 1496-1499.   CrossRef
  17. A. S. Frolov and N. N. Cencov: Application of dependent observations in the Monte Carlo method for recovering smooth curves. (in Russian) In: Proc. 6th Russian Conference on Probability Theory and Mathematical Statistics, Vilnus 1962, pp. 425-437.   CrossRef
  18. L. Györfi, I. Páli and E. C. van der Meulen: There is no universal source code for infinite alphabet. IEEE Trans. Inform. Theory 40 (1994), 267-271.   CrossRef
  19. L. Györfi, I. Páli and E. C. van der Meulen: On universal noiseless source coding for infinite source alphabets. Europ. Trans. Telecomm. 4 (1993), 9-16.   CrossRef
  20. J. A. Hartigan: The likelihood and invariance principles. Annals Math. Statist. 38 (1967), 533-539.   CrossRef
  21. I. A. Ibragimov and R. Z. Hasminski: On estimation of density. (in Russian) Scientific Notes of LOMI Seminars 98 (1980), 61-86.   CrossRef
  22. P. Kafka, F. Österreicher and I. Vincze: On powers of $f$-divergences defining a distance. Studia Sci. Math. Hungar. 26 (1991), 415-422.   CrossRef
  23. J. H. B. Kemperman: An optimum rate of transmitting information. Ann. Math. Statist. 40 (1969), 2156-2177.   CrossRef
  24. M. Khosravifard, D. Fooladivanda and T. A. Gulliver: Confliction of the convexity and metric properties in f-divergences. IEICE Trans. Fundamentals E90-A (2007), 1848-1853.   CrossRef
  25. A. L. Kolmogorov: Sulla determinazione empirica di una legge di distribuzione. Giornale dell'Istituto Italiano degli Attuari 4 (1933), 83-91.   CrossRef
  26. T. A. Kriz and J. V. Talacko: Equivalence of the maximum likelihood estimator to a minimum entropy estimator. Trab. Estadist. Invest. Oper. 19 (1968), 55-65.   CrossRef
  27. S. Kullback: A lower bound for discrimination in terms of variation. IEEE Trans. Inform, Theory 13 (1967), 126-127.   CrossRef
  28. S. Kullback: Correction to ``A lower bound for discrimination in terms of variation". IEEE Trans. Inform. Theory 16 (1970), 652.   CrossRef
  29. N. Morse and R. Sacksteder: Statistical isomorphism. Ann. Math. Statist. 37 (1966), 203-214.   CrossRef
  30. L. LeCam: On some asymptotic properties of maximum likelihood estimates and related Bayes estimates. Univ. Calif. Publ. Statist. 1 (1953), 267-329.   CrossRef
  31. F. Liese and I. Vajda: Convex Statistical Distances. Teubner, Leipzig 1987.   CrossRef
  32. E. A. Morozova and N. N. Cencov: Markov maps in noncommutative probability theory and mathematical statistics. (in Russian) In: Proc. 4th Internat. Vilnius Conf. Probability Theory and Mathematical Statistics, VNU Science Press 2 (1987), pp. 287-310.   CrossRef
  33. E. A. Nadaraya: On nonparametric estimation of Bayes risk in classification problems. (in Russian) Trans. Georgian Acad. Sci. 82 (1976), 277-280.   CrossRef
  34. E. A. Nadaraya: Nonparametric Estimation of Probability Density and Regression Curve. (in Russian) Tbilisi State University, Georgia 1983.   CrossRef
  35. F. Österreicher and I. Vajda: A new class of metric divergences on probability spaces and its statistical applications. Ann. Inst. Statist. Math. 55 (2003), 639-653.   CrossRef
  36. I. M. Sobol: Multidimensional Quadratic Formulas and Haar Functions. (in Russian) Nauka, Moscow 1969.   CrossRef
  37. W. W. Statulavicius: On Some Asymptotic Properties of Minimax Density Estimates. (in Russian) PhD. Thesis, Vilnus State University 1986.   CrossRef
  38. R. L. Stratonovich: Rate of convergence of probability density estimates. (in Russian) Trans. SSSR Acad. Sci., Ser. Technical Cybernetics 6 (1969), 3-15.   CrossRef
  39. G. T. Toussaint: Sharper lower bounds for information in term of variation. IEEE Trans. Inform. Theory {\mi 21} (1975), 99-103.   CrossRef
  40. I. Vajda: Note on discrimination information and variation. IEEE Trans. Inform. Theory IT-16 (1970), 771-773.   CrossRef
  41. I. Vajda: On the f-divergence and singularity of probability measures. Period. Math. Hungar. 2 (1972), 223-234.   CrossRef
  42. I. Vajda: On metric divergences of probability measures. Kybernetika 45 (2009), 885-900.   CrossRef
  43. A. Wald: Contributions to the theory of statistical estimation and testing hypotheses. Ann. Math. Statist. 10 (1939), 299-326.   CrossRef