February  2008, 2(1): 1-13. doi: 10.3934/amc.2008.2.1

Entropy estimators with almost sure convergence and an O(n-1) variance


Department of Physics and Computer Science, Wilfrid Laurier University, Waterloo, Ontario N2L3C5, Canada


Department of Computer Science, Yaroslavl State University, Yaroslavl, 150000, Russian Federation

Received  April 2007 Revised  October 2007 Published  January 2008

The problem of the estimation of the entropy rate of a stationary ergodic process $\mu$ is considered. A new nonparametric entropy rate estimator is constructed for a sample of n sequences $(X_1$(1)$,\ldots, (X_m$(1)$),\ldots, (X_1$(n) $,\ldots, (X_m$(n)$)$ independently generated by $\mu$. It is shown that, for $m=O(\log n)$, the estimator converges almost surely and its variance is upper-bounded by $O(n$−1$)$ for a large class of stationary ergodic processes with a finite state space. As the order $O(n$−1$)$ of the variance growth on $n$ is the same as that of the optimal Cramer-Rao lower bound, presented is the first near-optimal estimator in the sense of the variance convergence.
Citation: Alexei Kaltchenko, Nina Timofeeva. Entropy estimators with almost sure convergence and an O(n-1) variance. Advances in Mathematics of Communications, 2008, 2 (1) : 1-13. doi: 10.3934/amc.2008.2.1

Shinsuke Koyama, Lubomir Kostal. The effect of interspike interval statistics on the information gain under the rate coding hypothesis. Mathematical Biosciences & Engineering, 2014, 11 (1) : 63-80. doi: 10.3934/mbe.2014.11.63


John Kieffer and En-hui Yang. Ergodic behavior of graph entropy. Electronic Research Announcements, 1997, 3: 11-16.


Marcello Delitala, Tommaso Lorenzi. A mathematical model for value estimation with public information and herding. Kinetic and Related Models, 2014, 7 (1) : 29-44. doi: 10.3934/krm.2014.7.29


Guy Cohen, Jean-Pierre Conze. The CLT for rotated ergodic sums and related processes. Discrete and Continuous Dynamical Systems, 2013, 33 (9) : 3981-4002. doi: 10.3934/dcds.2013.33.3981


Wen Huang, Leiye Xu, Shengnan Xu. Ergodic measures of intermediate entropy for affine transformations of nilmanifolds. Electronic Research Archive, 2021, 29 (4) : 2819-2827. doi: 10.3934/era.2021015


Jean-Yves Le Boudec. The stationary behaviour of fluid limits of reversible processes is concentrated on stationary points. Networks and Heterogeneous Media, 2013, 8 (2) : 529-540. doi: 10.3934/nhm.2013.8.529


Wenjun Xia, Jinzhi Lei. Formulation of the protein synthesis rate with sequence information. Mathematical Biosciences & Engineering, 2018, 15 (2) : 507-522. doi: 10.3934/mbe.2018023


T. J. Sullivan, M. Koslowski, F. Theil, Michael Ortiz. Thermalization of rate-independent processes by entropic regularization. Discrete and Continuous Dynamical Systems - S, 2013, 6 (1) : 215-233. doi: 10.3934/dcdss.2013.6.215


M. A. Efendiev. On the compactness of the stable set for rate independent processes. Communications on Pure and Applied Analysis, 2003, 2 (4) : 495-509. doi: 10.3934/cpaa.2003.2.495


Jean René Chazottes, E. Ugalde. Entropy estimation and fluctuations of hitting and recurrence times for Gibbsian sources. Discrete and Continuous Dynamical Systems - B, 2005, 5 (3) : 565-586. doi: 10.3934/dcdsb.2005.5.565


Robert Azencott, Yutheeka Gadhyan. Accurate parameter estimation for coupled stochastic dynamics. Conference Publications, 2009, 2009 (Special) : 44-53. doi: 10.3934/proc.2009.2009.44


José Gómez-Torrecillas, F. J. Lobillo, Gabriel Navarro. Information--bit error rate and false positives in an MDS code. Advances in Mathematics of Communications, 2015, 9 (2) : 149-168. doi: 10.3934/amc.2015.9.149


Artur O. Lopes, Jairo K. Mengue. On information gain, Kullback-Leibler divergence, entropy production and the involution kernel. Discrete and Continuous Dynamical Systems, 2022, 42 (7) : 3593-3627. doi: 10.3934/dcds.2022026


Martin Heida, Alexander Mielke. Averaging of time-periodic dissipation potentials in rate-independent processes. Discrete and Continuous Dynamical Systems - S, 2017, 10 (6) : 1303-1327. doi: 10.3934/dcdss.2017070


Jong-Shenq Guo, Bei Hu. Blowup rate estimates for the heat equation with a nonlinear gradient source term. Discrete and Continuous Dynamical Systems, 2008, 20 (4) : 927-937. doi: 10.3934/dcds.2008.20.927


Aihua Fan, Jörg Schmeling, Weixiao Shen. $ L^\infty $-estimation of generalized Thue-Morse trigonometric polynomials and ergodic maximization. Discrete and Continuous Dynamical Systems, 2021, 41 (1) : 297-327. doi: 10.3934/dcds.2020363


Young-Sam Kwon. On the well-posedness of entropy solutions for conservation laws with source terms. Discrete and Continuous Dynamical Systems, 2009, 25 (3) : 933-949. doi: 10.3934/dcds.2009.25.933


Nikolai Dokuchaev. On strong causal binomial approximation for stochastic processes. Discrete and Continuous Dynamical Systems - B, 2014, 19 (6) : 1549-1562. doi: 10.3934/dcdsb.2014.19.1549


H.Thomas Banks, Shuhua Hu. Nonlinear stochastic Markov processes and modeling uncertainty in populations. Mathematical Biosciences & Engineering, 2012, 9 (1) : 1-25. doi: 10.3934/mbe.2012.9.1


Fritz Colonius. Invariance entropy, quasi-stationary measures and control sets. Discrete and Continuous Dynamical Systems, 2018, 38 (4) : 2093-2123. doi: 10.3934/dcds.2018086

2020 Impact Factor: 0.935


  • PDF downloads (141)
  • HTML views (0)
  • Cited by (12)

Other articles
by authors

[Back to Top]