February  2008, 2(1): 1-13. doi: 10.3934/amc.2008.2.1

Entropy estimators with almost sure convergence and an O(n-1) variance

1. 

Department of Physics and Computer Science, Wilfrid Laurier University, Waterloo, Ontario N2L3C5, Canada

2. 

Department of Computer Science, Yaroslavl State University, Yaroslavl, 150000, Russian Federation

Received  April 2007 Revised  October 2007 Published  January 2008

The problem of the estimation of the entropy rate of a stationary ergodic process $\mu$ is considered. A new nonparametric entropy rate estimator is constructed for a sample of n sequences $(X_1$(1)$,\ldots, (X_m$(1)$),\ldots, (X_1$(n) $,\ldots, (X_m$(n)$)$ independently generated by $\mu$. It is shown that, for $m=O(\log n)$, the estimator converges almost surely and its variance is upper-bounded by $O(n$−1$)$ for a large class of stationary ergodic processes with a finite state space. As the order $O(n$−1$)$ of the variance growth on $n$ is the same as that of the optimal Cramer-Rao lower bound, presented is the first near-optimal estimator in the sense of the variance convergence.
Citation: Alexei Kaltchenko, Nina Timofeeva. Entropy estimators with almost sure convergence and an O(n-1) variance. Advances in Mathematics of Communications, 2008, 2 (1) : 1-13. doi: 10.3934/amc.2008.2.1
[1]

Shinsuke Koyama, Lubomir Kostal. The effect of interspike interval statistics on the information gain under the rate coding hypothesis. Mathematical Biosciences & Engineering, 2014, 11 (1) : 63-80. doi: 10.3934/mbe.2014.11.63

[2]

John Kieffer and En-hui Yang. Ergodic behavior of graph entropy. Electronic Research Announcements, 1997, 3: 11-16.

[3]

Marcello Delitala, Tommaso Lorenzi. A mathematical model for value estimation with public information and herding. Kinetic and Related Models, 2014, 7 (1) : 29-44. doi: 10.3934/krm.2014.7.29

[4]

Guy Cohen, Jean-Pierre Conze. The CLT for rotated ergodic sums and related processes. Discrete and Continuous Dynamical Systems, 2013, 33 (9) : 3981-4002. doi: 10.3934/dcds.2013.33.3981

[5]

Wen Huang, Leiye Xu, Shengnan Xu. Ergodic measures of intermediate entropy for affine transformations of nilmanifolds. Electronic Research Archive, 2021, 29 (4) : 2819-2827. doi: 10.3934/era.2021015

[6]

Jean-Yves Le Boudec. The stationary behaviour of fluid limits of reversible processes is concentrated on stationary points. Networks and Heterogeneous Media, 2013, 8 (2) : 529-540. doi: 10.3934/nhm.2013.8.529

[7]

Wenjun Xia, Jinzhi Lei. Formulation of the protein synthesis rate with sequence information. Mathematical Biosciences & Engineering, 2018, 15 (2) : 507-522. doi: 10.3934/mbe.2018023

[8]

T. J. Sullivan, M. Koslowski, F. Theil, Michael Ortiz. Thermalization of rate-independent processes by entropic regularization. Discrete and Continuous Dynamical Systems - S, 2013, 6 (1) : 215-233. doi: 10.3934/dcdss.2013.6.215

[9]

M. A. Efendiev. On the compactness of the stable set for rate independent processes. Communications on Pure and Applied Analysis, 2003, 2 (4) : 495-509. doi: 10.3934/cpaa.2003.2.495

[10]

Jean René Chazottes, E. Ugalde. Entropy estimation and fluctuations of hitting and recurrence times for Gibbsian sources. Discrete and Continuous Dynamical Systems - B, 2005, 5 (3) : 565-586. doi: 10.3934/dcdsb.2005.5.565

[11]

Robert Azencott, Yutheeka Gadhyan. Accurate parameter estimation for coupled stochastic dynamics. Conference Publications, 2009, 2009 (Special) : 44-53. doi: 10.3934/proc.2009.2009.44

[12]

José Gómez-Torrecillas, F. J. Lobillo, Gabriel Navarro. Information--bit error rate and false positives in an MDS code. Advances in Mathematics of Communications, 2015, 9 (2) : 149-168. doi: 10.3934/amc.2015.9.149

[13]

Artur O. Lopes, Jairo K. Mengue. On information gain, Kullback-Leibler divergence, entropy production and the involution kernel. Discrete and Continuous Dynamical Systems, 2022, 42 (7) : 3593-3627. doi: 10.3934/dcds.2022026

[14]

Martin Heida, Alexander Mielke. Averaging of time-periodic dissipation potentials in rate-independent processes. Discrete and Continuous Dynamical Systems - S, 2017, 10 (6) : 1303-1327. doi: 10.3934/dcdss.2017070

[15]

Jong-Shenq Guo, Bei Hu. Blowup rate estimates for the heat equation with a nonlinear gradient source term. Discrete and Continuous Dynamical Systems, 2008, 20 (4) : 927-937. doi: 10.3934/dcds.2008.20.927

[16]

Aihua Fan, Jörg Schmeling, Weixiao Shen. $ L^\infty $-estimation of generalized Thue-Morse trigonometric polynomials and ergodic maximization. Discrete and Continuous Dynamical Systems, 2021, 41 (1) : 297-327. doi: 10.3934/dcds.2020363

[17]

Young-Sam Kwon. On the well-posedness of entropy solutions for conservation laws with source terms. Discrete and Continuous Dynamical Systems, 2009, 25 (3) : 933-949. doi: 10.3934/dcds.2009.25.933

[18]

Nikolai Dokuchaev. On strong causal binomial approximation for stochastic processes. Discrete and Continuous Dynamical Systems - B, 2014, 19 (6) : 1549-1562. doi: 10.3934/dcdsb.2014.19.1549

[19]

H.Thomas Banks, Shuhua Hu. Nonlinear stochastic Markov processes and modeling uncertainty in populations. Mathematical Biosciences & Engineering, 2012, 9 (1) : 1-25. doi: 10.3934/mbe.2012.9.1

[20]

Fritz Colonius. Invariance entropy, quasi-stationary measures and control sets. Discrete and Continuous Dynamical Systems, 2018, 38 (4) : 2093-2123. doi: 10.3934/dcds.2018086

2020 Impact Factor: 0.935

Metrics

  • PDF downloads (141)
  • HTML views (0)
  • Cited by (12)

Other articles
by authors

[Back to Top]