0.16 Markov chains

The transition matrix for [link] is given below.

Write the transition matrix from a) Monday to Thursday, b) Monday to Friday.

In writing a transition matrix from Monday to Thursday, we are moving from one state to another in three steps. That is, we need to compute $T^3$ .

$T^3=\left[\begin{array}{cc}\text{11}/\text{32}& \text{21}/\text{32}\\ \text{21}/\text{64}& \text{43}/\text{64}\end{array}\right]$

b) To find the transition matrix from Monday to Friday, we are moving from one state to another in 4 steps. Therefore, we compute $T^4$ .

$T^4=\left[\begin{array}{cc}\text{43}/\text{128}& \text{85}/\text{128}\\ \text{85}/\text{256}& \text{171}/\text{256}\end{array}\right]$

It is important that the student is able to interpret the above matrix correctly. For example, the entry $\text{85}/\text{128}$ , states that if Professor Symons walked to school on Monday, then there is $\text{85}/\text{128}$ probability that he will bicycle to school on Friday.

Got questions? Get instant answers now!

Got questions? Get instant answers now!

There are certain Markov chains that tend to stabilize in the long run, and they are the subject of [link] . It so happens that the transition matrix we have used in all the above examples is just such a Markov chain. The next example deals with the long term trend or steady-state situation for that matrix.

When this happens, we say that the system is in steady-state or state of equilibrium. In this situation, all row vectors are equal. If the original matrix is an $n$ by $n$ matrix, we get n vectors that are all the same. We call this vector a fixed probability vector or the equilibrium vector $E$ . In the above problem, the fixed probability vector $E$ is $\left[\begin{array}{cc}1/3& 2/3\end{array}\right]$ . Furthermore, if the equilibrium vector $E$ is multiplied by the original matrix $T$ , the result is the equilibrium vector $E$ . That is,

or,

Regular markov chains

At the end of [link] , we took the transition matrix $T$ and started taking higher and higher powers of it. The matrix started to stabilize, and finally it reached its steady-state or state of equilibrium . When that happened, all the row vectors became the same, and we called one such row vector a fixed probability vector or an equilibrium vector $E$ . Furthermore, we discovered that $\text{ET}=E$ .

Section overview

In this section, we wish to answer the following four questions.

1. Does every Markov chain reach a state of equilibrium?
2. Does the product of an equilibrium vector and its transition matrix always equal the equilibrium vector? That is, does $\text{ET}=E$ ?
3. Can the equilibrium vector $E$ be found without raising the matrix to higher powers?
4. Does the long term market share distribution for a Markov chain depend on the initial market share?