# American Institute of Mathematical Sciences

February  2008, 2(1): 1-13. doi: 10.3934/amc.2008.2.1

## Entropy estimators with almost sure convergence and an O(n-1) variance

 1 Department of Physics and Computer Science, Wilfrid Laurier University, Waterloo, Ontario N2L3C5, Canada 2 Department of Computer Science, Yaroslavl State University, Yaroslavl, 150000, Russian Federation

Received  April 2007 Revised  October 2007 Published  January 2008

The problem of the estimation of the entropy rate of a stationary ergodic process $\mu$ is considered. A new nonparametric entropy rate estimator is constructed for a sample of n sequences $(X_1$(1)$,\ldots, (X_m$(1)$),\ldots, (X_1$(n) $,\ldots, (X_m$(n)$)$ independently generated by $\mu$. It is shown that, for $m=O(\log n)$, the estimator converges almost surely and its variance is upper-bounded by $O(n$−1$)$ for a large class of stationary ergodic processes with a finite state space. As the order $O(n$−1$)$ of the variance growth on $n$ is the same as that of the optimal Cramer-Rao lower bound, presented is the first near-optimal estimator in the sense of the variance convergence.
Citation: Alexei Kaltchenko, Nina Timofeeva. Entropy estimators with almost sure convergence and an O(n-1) variance. Advances in Mathematics of Communications, 2008, 2 (1) : 1-13. doi: 10.3934/amc.2008.2.1
 [1] Shinsuke Koyama, Lubomir Kostal. The effect of interspike interval statistics on the information gain under the rate coding hypothesis. Mathematical Biosciences & Engineering, 2014, 11 (1) : 63-80. doi: 10.3934/mbe.2014.11.63 [2] John Kieffer and En-hui Yang. Ergodic behavior of graph entropy. Electronic Research Announcements, 1997, 3: 11-16. [3] Guy Cohen, Jean-Pierre Conze. The CLT for rotated ergodic sums and related processes. Discrete & Continuous Dynamical Systems - A, 2013, 33 (9) : 3981-4002. doi: 10.3934/dcds.2013.33.3981 [4] Marcello Delitala, Tommaso Lorenzi. A mathematical model for value estimation with public information and herding. Kinetic & Related Models, 2014, 7 (1) : 29-44. doi: 10.3934/krm.2014.7.29 [5] Wenjun Xia, Jinzhi Lei. Formulation of the protein synthesis rate with sequence information. Mathematical Biosciences & Engineering, 2018, 15 (2) : 507-522. doi: 10.3934/mbe.2018023 [6] Jean-Yves Le Boudec. The stationary behaviour of fluid limits of reversible processes is concentrated on stationary points. Networks & Heterogeneous Media, 2013, 8 (2) : 529-540. doi: 10.3934/nhm.2013.8.529 [7] M. A. Efendiev. On the compactness of the stable set for rate independent processes. Communications on Pure & Applied Analysis, 2003, 2 (4) : 495-509. doi: 10.3934/cpaa.2003.2.495 [8] T. J. Sullivan, M. Koslowski, F. Theil, Michael Ortiz. Thermalization of rate-independent processes by entropic regularization. Discrete & Continuous Dynamical Systems - S, 2013, 6 (1) : 215-233. doi: 10.3934/dcdss.2013.6.215 [9] Robert Azencott, Yutheeka Gadhyan. Accurate parameter estimation for coupled stochastic dynamics. Conference Publications, 2009, 2009 (Special) : 44-53. doi: 10.3934/proc.2009.2009.44 [10] Jean René Chazottes, E. Ugalde. Entropy estimation and fluctuations of hitting and recurrence times for Gibbsian sources. Discrete & Continuous Dynamical Systems - B, 2005, 5 (3) : 565-586. doi: 10.3934/dcdsb.2005.5.565 [11] José Gómez-Torrecillas, F. J. Lobillo, Gabriel Navarro. Information--bit error rate and false positives in an MDS code. Advances in Mathematics of Communications, 2015, 9 (2) : 149-168. doi: 10.3934/amc.2015.9.149 [12] Jong-Shenq Guo, Bei Hu. Blowup rate estimates for the heat equation with a nonlinear gradient source term. Discrete & Continuous Dynamical Systems - A, 2008, 20 (4) : 927-937. doi: 10.3934/dcds.2008.20.927 [13] Martin Heida, Alexander Mielke. Averaging of time-periodic dissipation potentials in rate-independent processes. Discrete & Continuous Dynamical Systems - S, 2017, 10 (6) : 1303-1327. doi: 10.3934/dcdss.2017070 [14] Nikolai Dokuchaev. On strong causal binomial approximation for stochastic processes. Discrete & Continuous Dynamical Systems - B, 2014, 19 (6) : 1549-1562. doi: 10.3934/dcdsb.2014.19.1549 [15] H.Thomas Banks, Shuhua Hu. Nonlinear stochastic Markov processes and modeling uncertainty in populations. Mathematical Biosciences & Engineering, 2012, 9 (1) : 1-25. doi: 10.3934/mbe.2012.9.1 [16] Young-Sam Kwon. On the well-posedness of entropy solutions for conservation laws with source terms. Discrete & Continuous Dynamical Systems - A, 2009, 25 (3) : 933-949. doi: 10.3934/dcds.2009.25.933 [17] Antoine Hochart. An accretive operator approach to ergodic zero-sum stochastic games. Journal of Dynamics & Games, 2019, 6 (1) : 27-51. doi: 10.3934/jdg.2019003 [18] Fritz Colonius. Invariance entropy, quasi-stationary measures and control sets. Discrete & Continuous Dynamical Systems - A, 2018, 38 (4) : 2093-2123. doi: 10.3934/dcds.2018086 [19] Yoora Kim, Gang Uk Hwang, Hea Sook Park. Feedback limited opportunistic scheduling and admission control for ergodic rate guarantees over Nakagami-$m$ fading channels. Journal of Industrial & Management Optimization, 2009, 5 (3) : 553-567. doi: 10.3934/jimo.2009.5.553 [20] Gianni Gilioli, Sara Pasquali, Fabrizio Ruggeri. Nonlinear functional response parameter estimation in a stochastic predator-prey model. Mathematical Biosciences & Engineering, 2012, 9 (1) : 75-96. doi: 10.3934/mbe.2012.9.75

2019 Impact Factor: 0.734