4.1 Criteria in hypothesis testing  (Page 2/3)

Neyman-pearson criterion

Situations occur frequently where assigning or measuring the a priori probabilities $P_i$ is unreasonable. For example, just what is the a priori probability of a supernova occurring in any particular region of the sky? We clearlyneed a model evaluation procedure which can function without a priori probabilities. This kind of test results when the so-called Neyman-Pearson criterion is used toderive the decision rule. The ideas behind and decision rules derived with the Neyman-Pearson criterion ( Neyman and Pearson ) will serve us well in sequel; their result is important!

Using nomenclature from radar, where model ${}_1$ represents the presence of a target and ${}_0$ its absence, the various types of correct and incorrect decisions have the following names ( Woodward, pp. 127-129 ).

• Detection

  we say it's there when it is; $P_D=\mathrm{Pr}(\text{say}{}_1|{}_1\text{true})$

• False-alarm

  we say it's there when it's not; $P_F=\mathrm{Pr}(\text{say}{}_1|{}_0\text{true})$

• Miss

  we say it's not there when it is; $P_M=\mathrm{Pr}(\text{say}{}_0|{}_1\text{true})$

These two probabilities are related to each other in an interesting way. Expressing these quantities in terms of thedecision regions and the likelihood functions, we have $$P_F=\int p(r, {}_0, r)\,d r$$ $$P_D=\int p(r, {}_1, r)\,d r$$ As the region ${}_1$ shrinks, both of these probabilities tend toward zero; as ${}_1$ expands to engulf the entire range of observation values, they both tend toward unity. This rather directrelationship between $P_D$ and $P_F$ does not mean that they equal each other; in most cases, as ${}_1$ expands, $P_D$ increases more rapidly than $P_F$ (we had better be right more often than we are wrong!). However, the "ultimate" situation where a rule isalways right and never wrong ( $P_D=1$ , $P_F=0$ ) cannot occur when the conditional distributions overlap. Thus, to increase the detection probability we mustalso allow the false-alarm probability to increase. This behavior represents the fundamental tradeoff in hypothesistesting and detection theory.

One can attempt to impose a performance criterion that depends only on these probabilities with the consequent decision rulenot depending on the a priori probabilities. The Neyman-Pearson criterion assumes that the false-alarm probability is constrained to be less than orequal to a specified value $$ while we attempt to maximize the detection probability $P_D$ . $$\forall P_F, P_F\le \colon \max\{{}_1 , P_D\}$$ A subtlety of the succeeding solution is that the underlying probability distribution functions may not becontinuous, with the result that $P_F$ can never equal the constraining value $$ . Furthermore, an (unlikely) possibility is that the optimum value for the false-alarm probability is somewhat lessthan the criterion value. Assume, therefore, that we rephrase the optimization problem by requiring that the false-alarmprobability equal a value $^\prime$ that is less than or equal to $$ .