4.5 Stein's lemma

As important as understanding the false-alarm, miss and error probabilities of the likelihood ratio test might be, no generalexpression exists for them. In analyzing the Gaussian problem, we find these in terms of $Q(·)$ , which has no closed form expression. In the general problem, the situation is much worse: No expression of any kindcan be found! The reason is that we don't have the probability distribution of the sufficient statistic in the likelihood ratiotest, which is needed in these expressions . We are faced with the curious situation that while knowing the decision rule thatoptimizes the performance probabilities, we usually don't know what the resulting performance probabilities will be.

Some general expressions are known for the asymptotic form of these error probabilities: limiting expressions as the number ofobservations $L$ becomes large. These results go by the generic name of Stein's Lemma Cover and Thomas, §12.8 .

• $(p_1, p_0)\ge 0$ and $𝒞(p_0, p_1)\ge 0$ . Furthermore, these distances equal zero only when $p_0(r)=p_1(r)$ . The Kullback-Leibler and Chernoff distances are always non-negative, with zero distance occurring only whenthe probability distributions are the same.
• $(p_1, p_0)$∞ whenever, for some $r$ , $p_0(r)=0$ and $p_1(r)\neq 0$ . If $p_1(r)=0$ , the value of $p_1(r)\lg \left(\frac{p_1(r)}{p_0(r)}\right)$ is defined to be zero.
• When the underlying stochastic quantities are random vectors having statistically independent components withrespect to both $p_0$ and $p_1$ , the Kullback-Leibler distance equals the sum of the component distances. Stated mathematically, if $p_0(r)=\prod p_0(r_l)$ and $p_1(r)=\prod p_1(r_l)$ ,
  $(p_1(r), p_0(r))=\sum (p_1(r_l), p_0(r_l))$
  The Chernoff distance does not have this property unless the components are identicallydistributed.
• $(p_1, p_0)\neq (p_0, p_1)$ ; $𝒞(p_0, p_1)=𝒞(p_1, p_0)$ . The Kullback-Leibler distance is usually not a symmetric quantity. In some special cases, it can besymmetric (like the just described Gaussian example), but symmetry cannot, and should not, be expected. The Chernoffdistance is always symmetric.
• $(p(r_1, r_2), p(r_1)p(r_2))=(r_1, r_2)$ . The Kullback-Leibler distance between a joint probability density and the product of the marginaldistributions equals what is known in information theory as the mutual information between the random variables $r_1$ , $r_2$ . From the properties of the Kullback-Leibler distance, we see that the mutual information equals zeroonly when the random variables are statistically independent.