16.1 Elements of markov sequences  (Page 3/13)

Remark . In this case, the actual transition takes place throughout the period. However, for purposes of analysis, we examine the state onlyat the end of the period (before restocking). Thus, the transitions are dispersed in time, but the observations are at discrete instants.

Remark . Each of these four examples exhibits the pattern

1. $\{X_0,Y_n:1\le n\}$ is independent
2. $X_{n+1}=g_{n+1}(X_n,Y_{n+1}),n\ge 0$

We now verify the Markov condition and obtain a method for determining the transition probabilities.

A pattern yielding Markov sequences

Suppose $\{Y_n:0\le n\}$ is independent (call these the driving random variables ). Set

Then

1. X _N is Markov
2. $P(X_{n+1}\in Q|X_n=u)=P[g_{n+1}(u,Y_n+1)\in Q]$ for all $n,u$ , and any Borel set Q .

VERIFICATION

1. It is apparent that if $Y_0,Y_1,\cdots ,Y_n$ are known, then U _n is known. Thus $U_n=h_n(Y_0,Y_1,\cdots ,Y_n)$ , which ensures each pair $\{Y_{n+1},U_n\}$ is independent. By property (CI13) , with $X=Y_{n+1},Y=X_n$ , and $Z=U_{n-1}$ , we have
   $$\{Y_{n+1},U_{n-1}\}\mathrm{ci}|X_n$$
   Since $X_{n+1}=g_{n+1}(Y_{n+1},X_n)$ and $U_n=h_n(X_n,U_{n-1})$ , property (CI9) ensures
   $$\{X_{n+1},U_n\}\mathrm{ci}|X_n\forall n\ge 0$$
   which is the Markov property.
2. $P(X_{n+1}\in Q|X_n=u)=E\left\{I_Q\left[g_{n+1}(X_n,Y_{n+1})\right]\right|X_n=u\}\mathrm{a}.\mathrm{s}.$ $=E\left\{I_Q\left[g_{n+1}(u,Y_{n+1})\right]\right\}\mathrm{a}.\mathrm{s}.\left[P_X\right]$ by (CE10b) $=P[g_{n+1}(u,Y_{n+1})\in Q]$ by (E1a)

— $\square$

The application of this proposition, below, to the previous examples shows that the transition probabilities are invariant with n . This case is important enough to warrant separate classification.

Definition . If $P(X_{n+1}\in Q|X_n=u)$ is invariant with n , for all Borel sets Q , all $u\in \mathbf{E}$ , the Markov process X _N is said to be homogeneous .

As a matter of fact, this is the only case usually treated in elementary texts. In this regard, we note the following special case of the proposition above.

Homogenous Markov sequences

If $\{Y_n:1\le n\}$ is iid and $g_{n+1}=g$ for all n , then the process is a homogeneous Markov process, and

— $\square$

Remark .

In the homogeneous case, the transition probabilities are invariant with n . In this case, we write

These are called the (one-step) transition probabilities .

The transition probabilities may be arranged in a matrix P called the transition probability matrix , usually referred to as the transition matrix ,