4.3 Beyond two models

Frequently, more than two viable models for data generation can be defined for a given situation. The classification problem is to determine which of several models best "fits" a set of measurements. For example,determining the type of airplane from its radar returns forms a classification problem. The model evaluation framework has theright structure if we can allow more than two models. We happily note that in deriving the likelihood ratio test we did not needto assume that only two possible descriptions exist. Go back and examine the expression for the maximum probability correct decision rule . If $K$ models seem appropriate for a specific problem, the decision rule maximizingthe probability of making a correct choice is $$\forall i, i\in \{1, , K\}\colon \max\{{}_{i}p(r, {}_i, r)\}$$ To determine the largest of $K$ quantities, exactly $K-1$ numeric comparisons need be made. When we have two possible models( $K=2$ ), this decision rule reduces to the computation of the likelihood ratio and its comparison to a threshold. In general, $K-1$ likelihood ratios need to be computed and compared to a threshold. Thus the likelihood ratio test can be viewed as aspecific method for determining the largest of the decision statistics ${}_{i}p(r, {}_i, r)$ .

Since we need only the relative ordering of the $K$ decision statistics to make a decision, we can apply any transformation $T()$ to them that does not affect ordering. In general, possible transformations must be positively monotonic to satisfythis condition. For example, the needless common additive components in the decision statistics can be eliminated, even ifthey depend on the observations. Mathematically, "common" means that the quantity does not depend on the model index $i$ . The transformation in this case would be of the form $T(z_i)=z_i-a$ , clearly a monotonic transformation. A positive multiplicative factor can also be "canceled"; if negative, the ordering would be reversed andthat cannot be allowed. The simplest resulting expression becomes the sufficient statistic ${}_i(r)$ for the model. Expressed in terms of the sufficient statistic, the maximum probability correct or the Bayesiandecision rule becomes $$\forall i, i\in \{1, , K\}\colon \max\{C_i+{}_i(r)\}$$ where $C_i$ summarizes all additive terms that do not depend onthe observation vector $r$ . The quantity ${}_i(r)$ is termed the sufficient statistic associated with model $i$ . In many cases, the functional form of the sufficient statistic varies little from one model to another andexpresses the necessary operations that summarize the observations. The constants $C_i$ are usually lumped together to yield the threshold against which we compare the sufficient statistic. For example,in the binary model situation, the decision rule becomes $${}_1(r)+C_1\underset{{}_0}{\overset{{}_1}{}}{}_0(r)+C_0$$ or $${}_1(r)-{}_0(r)\underset{{}_0}{\overset{{}_1}{}}{C}_0-C_1$$ Thus, the sufficient statistic for the decision rule is ${}_1(r)-{}_0(r)$ and the threshold $$ is $C_0-C_1$ .

When employing the Neyman-Pearson test, we need to specify the various error probabilities $(H_j\text{true}, \text{say}{}_i)$ . These specifications amount to determining the constants $c_i$ when the sufficient statistic is used. Since $K-1$ comparisons will be used to home in on the optimal decision, only $K-1$ error probabilities need be specified. Typically, the quantities $({}_0\text{true}, \text{say}{H}_i)$ , $i\in \{1, , K-1\}$ are used, particularly when the model ${}_0$ represents the situation when no signal is present (see this problem ).