3.9 Probability density estimation

The number of length- $L$ observation sequences having a given type $(P)$ approximately equals $e^{-L\mathscr{H}((P))}$ . The probability that a given sequence has a given type approximately equals $e^{-L((P), P)}$ , which means that the probability a given sequence has a type not equal to the true distribution decays exponentially with the number of observations. Thus, whilethe coin flip sequences $\{H, H, H, H, H\}$ and $\{T, T, H, H, T\}$ are equally likely (assuming a fair coin), the second is more typical because its type is closer to the true distribution.

Histogram estimators

By far the most used technique for estimating the probability distribution of a continuous-valued random variable is the histogram ; more sophisticated techniques are discussed in Silverman . For real-valued data, subdivide the real line into $N$ intervals $\left(r_i , r_{i+1}\right]$ having widths ${\delta }_i=r_{i+1}-r_i$ , $i=\{1, \dots , N\}$ . These regions are called bins and they should encompass the range of values assumed by the data. For largevalues, the "edge bins" can extend to infinity to catch the overflows. Given $L$ observations of a stationary random sequence $r(l)$ , $l=\{0, \dots , L-1\}$ , the histogram estimate $h(i)$ is formed by simply forming a type from the number $L_i$ of these observations that fall into the $i^{\mathrm{th}}$ bin and dividing by the binwidth ${\delta }_i$ . $$(p(r, r))=\begin{cases}h(1)=\frac{L_1}{L{\delta }_1} & \text{if $r_1< r\le r_2$}\\ h(2)=\frac{L_2}{L{\delta }_2} & \text{if $r_2< r\le r_3$}\\ ⋮ & \text{if $$}\\ h(N)=\frac{L_N}{L{\delta }_N} & \text{if $r_N< r\le r_{N+1}$}\end{cases}$$ The histogram estimate resembles a rectangular approximation to the density. Unless the underlying density has the sameform (a rare event), the histogram estimate does not converge to the true density as the number $L$ of observations grows. Presumably, the value of the histogram at each bin convergesto the probability that the observations lie in that bin. $$\lim_{L\to }L\to$$∞ L i L r r i r i + 1 p r r To demonstrate this intuitive feeling, we compactly denote the histogram estimate by using indicator functions . An indicator function $I_i(r(l))$ for the $i^{\mathrm{th}}$ bin equals one if the observation $r(l)$ lies in the bin and is zero otherwise. The estimate is simply the average of the indicator functions across theobservations. $$h(i)=\frac{1}{L{\delta }_i}\sum_{l=0}^{L-1} I_i(r(l))$$ The expected value of $I_i(r(l))$ is simply the probability $P_i$ that the observation lies in the $i^{\mathrm{th}}$ bin. Thus, the expected value of each histogram value equals the integral of the actual density over the bin, showing thatthe histogram is an unbiased estimate of this integral. Convergence can be tested by computing the variance of theestimate. The variance of one bin in the histogram is given by $$\sigma(h(i))^2=\frac{P_i-P_i^2}{L{\delta }_i^2}+\frac{1}{L^2{\delta }_i^2}\sum_{k\neq l} (I_i(r(k))I_i(r(l)))-P_i^2$$ To simplify this expression, the correlation between the observations must be specified. If the values arestatistically independent (we have white noise), each term in the sum becomes zero and the variance is given by $\sigma(h(i))^2=\frac{P_i-P_i^2}{L{\delta }_i^2}$ . Thus, the variance tends to zero as $L\to$∞ and the histogram estimate is consistent, converging to $\frac{P_i}{{\delta }_i}$ . If the observations are not white, convergence becomes problematical. Assume, for example, that $I_i(r(k))$ and $I_i(r(l))$ are correlated in a first-order, geometric fashion. $$(I_i(r(k))I_i(r(l)))-P_i^2=P_i^2\rho ^{\left|k-l\right|}$$ The variance does increase with this presumed correlation until, at the extreme ( $\rho =1$ ), the variance is a constant independent of $L$ ! In summary, if the observations are mutually correlated and the histogramestimate converges, the estimate converges to the proper value but more slowly than if the observations were white. Theestimate may not converge if the observations are heavily dependent from index to index. This type of dependencestructure occurs when the power spectrum of the observations is lowpass with an extremely low cutoff frequency.