February  2008, 2(1): 1-13. doi: 10.3934/amc.2008.2.1

Entropy estimators with almost sure convergence and an O(n-1) variance

1. 

Department of Physics and Computer Science, Wilfrid Laurier University, Waterloo, Ontario N2L3C5, Canada

2. 

Department of Computer Science, Yaroslavl State University, Yaroslavl, 150000, Russian Federation

Received  April 2007 Revised  October 2007 Published  January 2008

The problem of the estimation of the entropy rate of a stationary ergodic process $\mu$ is considered. A new nonparametric entropy rate estimator is constructed for a sample of n sequences $(X_1$(1)$,\ldots, (X_m$(1)$),\ldots, (X_1$(n) $,\ldots, (X_m$(n)$)$ independently generated by $\mu$. It is shown that, for $m=O(\log n)$, the estimator converges almost surely and its variance is upper-bounded by $O(n$−1$)$ for a large class of stationary ergodic processes with a finite state space. As the order $O(n$−1$)$ of the variance growth on $n$ is the same as that of the optimal Cramer-Rao lower bound, presented is the first near-optimal estimator in the sense of the variance convergence.
Citation: Alexei Kaltchenko, Nina Timofeeva. Entropy estimators with almost sure convergence and an O(n-1) variance. Advances in Mathematics of Communications, 2008, 2 (1) : 1-13. doi: 10.3934/amc.2008.2.1
[1]

Shinsuke Koyama, Lubomir Kostal. The effect of interspike interval statistics on the information gain under the rate coding hypothesis. Mathematical Biosciences & Engineering, 2014, 11 (1) : 63-80. doi: 10.3934/mbe.2014.11.63

[2]

John Kieffer and En-hui Yang. Ergodic behavior of graph entropy. Electronic Research Announcements, 1997, 3: 11-16.

[3]

Marcello Delitala, Tommaso Lorenzi. A mathematical model for value estimation with public information and herding. Kinetic & Related Models, 2014, 7 (1) : 29-44. doi: 10.3934/krm.2014.7.29

[4]

Guy Cohen, Jean-Pierre Conze. The CLT for rotated ergodic sums and related processes. Discrete & Continuous Dynamical Systems, 2013, 33 (9) : 3981-4002. doi: 10.3934/dcds.2013.33.3981

[5]

Wen Huang, Leiye Xu, Shengnan Xu. Ergodic measures of intermediate entropy for affine transformations of nilmanifolds. Electronic Research Archive, 2021, 29 (4) : 2819-2827. doi: 10.3934/era.2021015

[6]

Jean-Yves Le Boudec. The stationary behaviour of fluid limits of reversible processes is concentrated on stationary points. Networks & Heterogeneous Media, 2013, 8 (2) : 529-540. doi: 10.3934/nhm.2013.8.529

[7]

Wenjun Xia, Jinzhi Lei. Formulation of the protein synthesis rate with sequence information. Mathematical Biosciences & Engineering, 2018, 15 (2) : 507-522. doi: 10.3934/mbe.2018023

[8]

T. J. Sullivan, M. Koslowski, F. Theil, Michael Ortiz. Thermalization of rate-independent processes by entropic regularization. Discrete & Continuous Dynamical Systems - S, 2013, 6 (1) : 215-233. doi: 10.3934/dcdss.2013.6.215

[9]

M. A. Efendiev. On the compactness of the stable set for rate independent processes. Communications on Pure & Applied Analysis, 2003, 2 (4) : 495-509. doi: 10.3934/cpaa.2003.2.495

[10]

Jean René Chazottes, E. Ugalde. Entropy estimation and fluctuations of hitting and recurrence times for Gibbsian sources. Discrete & Continuous Dynamical Systems - B, 2005, 5 (3) : 565-586. doi: 10.3934/dcdsb.2005.5.565

[11]

Robert Azencott, Yutheeka Gadhyan. Accurate parameter estimation for coupled stochastic dynamics. Conference Publications, 2009, 2009 (Special) : 44-53. doi: 10.3934/proc.2009.2009.44

[12]

José Gómez-Torrecillas, F. J. Lobillo, Gabriel Navarro. Information--bit error rate and false positives in an MDS code. Advances in Mathematics of Communications, 2015, 9 (2) : 149-168. doi: 10.3934/amc.2015.9.149

[13]

Martin Heida, Alexander Mielke. Averaging of time-periodic dissipation potentials in rate-independent processes. Discrete & Continuous Dynamical Systems - S, 2017, 10 (6) : 1303-1327. doi: 10.3934/dcdss.2017070

[14]

Aihua Fan, Jörg Schmeling, Weixiao Shen. $ L^\infty $-estimation of generalized Thue-Morse trigonometric polynomials and ergodic maximization. Discrete & Continuous Dynamical Systems, 2021, 41 (1) : 297-327. doi: 10.3934/dcds.2020363

[15]

Jong-Shenq Guo, Bei Hu. Blowup rate estimates for the heat equation with a nonlinear gradient source term. Discrete & Continuous Dynamical Systems, 2008, 20 (4) : 927-937. doi: 10.3934/dcds.2008.20.927

[16]

Young-Sam Kwon. On the well-posedness of entropy solutions for conservation laws with source terms. Discrete & Continuous Dynamical Systems, 2009, 25 (3) : 933-949. doi: 10.3934/dcds.2009.25.933

[17]

Nikolai Dokuchaev. On strong causal binomial approximation for stochastic processes. Discrete & Continuous Dynamical Systems - B, 2014, 19 (6) : 1549-1562. doi: 10.3934/dcdsb.2014.19.1549

[18]

H.Thomas Banks, Shuhua Hu. Nonlinear stochastic Markov processes and modeling uncertainty in populations. Mathematical Biosciences & Engineering, 2012, 9 (1) : 1-25. doi: 10.3934/mbe.2012.9.1

[19]

Xiaoli Feng, Meixia Zhao, Peijun Li, Xu Wang. An inverse source problem for the stochastic wave equation. Inverse Problems & Imaging, , () : -. doi: 10.3934/ipi.2021055

[20]

Fritz Colonius. Invariance entropy, quasi-stationary measures and control sets. Discrete & Continuous Dynamical Systems, 2018, 38 (4) : 2093-2123. doi: 10.3934/dcds.2018086

2020 Impact Factor: 0.935

Metrics

  • PDF downloads (119)
  • HTML views (0)
  • Cited by (12)

Other articles
by authors

[Back to Top]