Class 13
Positive recurrent:
\(f_i = 1\)
\(m_i < \infty\)
\(\pi_i > 0\) if \(X_0 \leftrightarrow i\)
Null recurrent
\(f_i = 1\)
\(m_i = \infty\)
\(\pi_i = 0\)
Transient
\(f_i < 1\)
\(\pi_i = 0\)
Let \(i\) be a positive recurrent state.
Define \(\displaystyle I_n = \begin{cases} 1 & \text{ if } X_n = i\\ 0 & \text{ otherwise} \end{cases}\)
Define \(\displaystyle M_{in} = \operatorname{E}\left(\sum_{k=1}^n I_k\right)\).
Define \(\displaystyle\pi_i = \lim_{n\to\infty} \frac{M_{in}}{n}\), the long run proportion in state \(i\).
\(\pi_i = \operatorname{P}(X_n = i \text{ for a randomly selected } n) = \operatorname{P}(X_n = i \text{ for a randomly selected } n > 0)\)
Question: If I randomly pick a step \(n\), what is the probability that the transition from \(X_n\) to \(X_{n+1}\) will be from state \(i\) to state \(j\)?
Question: If I randomly pick a step \(n > 0\), what is the probability that \(X_n = j\)?
Suppose \(X\) is an irreducible Markov chain. Then
\[\begin{aligned} \pi_j &= \sum_i \pi_i P_{ij} \text{ for every state } j\\ \sum_i \pi_i &= 1 \end{aligned}\]
If this system has no solution, then \(X\) is either transient or null recurrent, and \(\pi_j = 0\) for every \(j\).
Example:
\[\mathbf{P} = \begin{bmatrix}.9&.1\\.4&.6\end{bmatrix}\]
\[ \begin{bmatrix} \mathstrut \\ \mathbf{P}^T - I \\ \mathstrut\\ 1\quad 1\quad \cdots \quad 1 \end{bmatrix} \begin{bmatrix} \pi_0 \\ \vdots \\ \pi_n \end{bmatrix} = \begin{bmatrix} 0 \\ \vdots \\ 0 \\ 1 \end{bmatrix} \]
Suppose the initial state \(X_0\) is chosen according to the long run proportions:
\(\operatorname{P}(X_0 = i) = \pi_i\) for each \(i\).
What is the probability that \(\operatorname{P}(X_n = j)\) for some fixed \(n\)?
For a state \(i\), define \(\displaystyle\alpha_i = \lim_{n\to\infty} \operatorname{P}(X_n = i)\).
This limit may not exist, but when it does, it is called the limiting probability of \(i\).
How do we calculate these?