16.1 Elements of markov sequences  (Page 6/13)

Various properties of conditional independence, particularly (CI9) , (CI10) , and (CI12) , may be used to establish the following. The immediate future $X_{n+1}$ may be replaced by any finite future $U_{n,n+p}$ and the present X _n may be replaced by any extended present $U_{m,n}$ . Some results of abstract measure theory show that the finite future $U_{n,n+p}$ may be replaced by the entire future U ⁿ . Thus, we may assert

Extended Markov property

X _N is Markov iff

— $\square$

The Chapman-Kolmogorov equation and the transition matrix

As a special case of the extended Markov property, we have

Setting $g(U^{n+k},X_{n+k})=X_{n+k+m}$ and $h(U_n,X_{n+k})=X_n$ in (CI9) , we get

By the iterated conditioning rule (CI9) for conditional independence, it follows that

This is the Chapman-Kolmogorov equation , which plays a central role in the study of Markov sequences. For a discrete state space E , with

this equation takes the form

To see that this is so, consider

Homogeneous case

For this case, we may put $\left(C{K}^{\text{'}}\right)$ in a useful matrix form. The conditional probabilities p ^m of the form

are known as the m-step transition probabilities . The Chapman-Kolmogorov equation in this case becomes

In terms of the m-step transition matrix ${\mathbf{P}}^{\left(m\right)}=\left[p^m(i,k)\right]$ , this set of sums is equivalent to the matrix product

Now

A simple inductive argument based on $\left(C{K}^{\text{'}\text{'}}\right)$ establishes

The product rule for transition matrices

The m -step probability matrix ${\mathbf{P}}^{\left(m\right)}={\mathbf{P}}^m$ , the m th power of the transition matrix P

— $\square$

We consider next the state probabilities for the various stages. That is, we examine the distributions for the various X _n , letting $p_k\left(n\right)=P(X_n=k)$ for each $k\in \mathbf{E}$ . To simplify writing, we consider a finite state space $\mathbf{E}=\{1,\cdots ,M\}$ . We use $\pi \left(n\right)$ for the row matrix

As a consequence of the product rule, we have

Probability distributions for any period

For a homogeneous Markov sequence, the distribution for any X _n is determined by the initial distribution (i.e., for X ₀ ) and the transition probability matrix P .

VERIFICATION

Suppose the homogeneous sequence X _N has finite state-space $\mathbf{E}=\{1,2,\cdots ,M\}$ . For any $n\ge 0$ , let $p_j\left(n\right)=P(X_n=j)$ for each $j\in \mathbf{E}$ . Put

Then

• $\pi \left(0\right)=$ the initial probability distribution
• $\pi \left(1\right)=\pi \left(0\right)\mathbf{P}$
• .....
• $\pi \left(n\right)=\pi (n-1)\mathbf{P}=\pi \left(0\right){\mathbf{P}}^{\left(n\right)}=\pi \left(0\right){\mathbf{P}}^n=$ the n th-period distribution

The last expression is an immediate consequence of the product rule.

— $\square$