0.9 Error bounds in countably infinite spaces  (Page 2/2)

This implies that for any $\delta >0$ with probability at least $1-\delta$ we have $\forall f\in \mathcal{F}$

Application

Histogram classifiers

X=[0,1] ${}^d$ , Y={0,1}. Let ${\mathcal{F}}_k$ , k=1, 2, ... denote the collection of histogram classification rules with k equal volumebins. We can use the following codebook for the index k.

And follow this codeword with $k={log}_2$ $|{\mathcal{F}}_k|$ bits to indicate which of the 2 ${}^k$ possible histogram rules is under consideration. Thus for any $f\in {\mathcal{F}}_k$ for some k $\ge$ 1 there is a prefix code of length

It follows that for any $\delta >0$ with probability at least $1-\delta$ we have $\forall f\in {\bigcup }_{k\ge 1}{\mathcal{F}}_k$

where $k_f$ is the number of bins in histogram corresponding to $f$ . Contrast with the bound we had for the class of m bin histograms alone: with probability $\ge 1-\delta$ , $\forall f\in {\mathcal{F}}_m$

Notice the bound for all histograms rules is almost as good as the bound for only the $m$ -bin rules. That is, when $k_f=m$ the bounds are within a factor of $\sqrt{2}$ . On the other hand, the new bound is a big improvement, since it also gives us a guide for selecting thenumber of bins.

Proof of the kraft inequality

We will prove that for any binary prefix code, the codeword lengths $c_1$ , $c_2$ , ... satisfy ${\sum }_{k\ge 1}{2}^{-c_k}\le 1$ . The converse is easy to prove also, but it not central to ourpurposes here (for a proof, see Cover $\&$ Thomas '91). Consider a binary tree like the one shown below.

The sequence of bit values leading from the root to a leaf of the tree represents a codeword. The prefix condition implies that nocodeword is a descendant of any other codeword in the tree. Let c ${}_{max}$ be the length of the longest codeword (also the number of branches to the deepest leaf) in the tree.

Consider a leaf $i$ in the tree at level c ${}_i$ . This leaf would have 2 ${}^{c_{max}-c_i}$ descendants at level c ${}_{max}$ . Furthermore, for each leaf the set of possible descendants at level c ${}_{max}$ is disjoint (since no codeword can be a prefix of another). Therefore,since the total number of possible leafs at level c ${}_{max}$ is $2^{c_{max}}$ , we have

which proves the case when the number of codewords is finite.

Suppose now that we have a countably infinite number of codewords. Let b ${}_1$ b ${}_2$ ... b ${}_{c_i}$ be the ith codeword and let

be the real number corresponding to the binary expansion of the codeword. We canassociate the interval $[r_i,r_i+2^{-c_i})$ with the ith codeword. This is the set of all real numbers whose binaryexpansion begins with b ${}_1$ b ${}_2$ ... b ${}_{c_i}$ . Since this is a subinterval of $[0,1]$ , and all such subintervals corresponding to prefix codewords are disjoint, the sum of their lengths must beless than or equal to 1. This proves the case where the number of codewords is infinite.