Lecture-16: Invariant Distribution

(1)

Lecture-16: Invariant Distribution

1 Transient and recurrent states

1.1 Hitting and return times

Definition 1.1. For a homogeneous Markov chainX:Ω→_X^Z⁺, we can definefirst hitting timeto state x∈_{X, as}

τ_x⁺≜inf{n∈_N:Xn=x}. IfX0=x, thenτ_x⁺is called thefirst return timeto statex.

Lemma 1.2. For an irreducible Markov chain X:Ω→_X^Z⁺ on finite state spaceX, we haveExτ_y⁺<_∞for all states x,y∈X.

Proof. From the definition of irreducibility, for each pair of statesz,w∈X, we have a positive integer nzw∈_Nsuch thatpⁿzw^zw>ϵzw>0. Since the state spaceXis finite, We define

ϵ≜ inf

z,w∈Xϵ_zw>0, r≜ sup

z,w∈Xnzw∈N.

Hence, there exists a positive integerr∈_Nand a realϵ>0 such that p⁽ⁿ⁾zw >ϵfor somen⩽rand all statesz,w∈X. It follows thatP(∪_n∈[r]{Xn=y})>ϵor Pz

n

τ_y⁺>ro

⩽1−ϵfor any initial condition X0=z∈X. Therefore, we can write fork∈_N

Px

n

τ_y⁺>kro

=Px

n

τ_y⁺>(k−1)ro

P(ⁿτ_y⁺>kro

|ⁿτ_y⁺>(k−1)r,X0=xo

)⩽(1−ϵ)Px

n

τ_y⁺>(k−1)ro . By induction, we havePx

n

τ_y⁺>kro

⩽(1−ϵ)^k. SincePx

n

τ_y⁺>no

is decreasing inn, we can write Exτ_y⁺=

∑

k∈Z+ r−1

∑

i=0

Px{τ_y⁺>kr+i}⩽

∑

k∈Z+

rPx{τ_y⁺>kr}⩽ ^r ϵ<∞.

Corollary 1.3. For an irreducible Markov chain X:Ω→_X^Z⁺on finite state spaceX, we have Px

n

τ_y⁺<_∞^o=1 for all states x,y∈_X.

Proof. This follows from the fact thatτ_y⁺ is a positive random variable with finite mean for all states y∈Xand any initial statex∈X.

1.2 Recurrence and transience

Definition 1.4. Let fxy⁽ⁿ⁾denote the probability that starting from statex, the first transition into statey happens at timen. Then, fxy⁽ⁿ⁾=Px

n

τ_y⁺=no

. Then we denote the probability of eventually entering stateygiven that we start at statex, asfxy=_∑^∞_n=1fxy⁽ⁿ⁾=Px

n

τ_y⁺<_∞^o. The stateyis said to betransient if fyy<1 andrecurrentif fyy=1.

Definition 1.5. For a discrete time processX:Ω→X^Z⁺, the total number of visits to a statey∈Xin first nsteps is denoted by Ny(n)≜∑ⁿ_i=11{X_i=y}. Total number of visits to statey∈Xis denoted by Ny≜Ny(_∞).

Remark 1. From the linearity of expectations and monotone convergence theorem, we get EyNy =

∑n∈Np⁽ⁿ⁾yy.

1

(2)

Lemma 1.6. Consider a homogeneous Markov chain X:Ω→_X^Z⁺. For each m∈_Z₊ and state x,y∈_{X, we} have

Px

Ny=m =

(1− fxy m=_0, fxyf_yy^m−1(1−fyy) m∈_N.

Proof. For eachk∈N, the time τ_y⁺(k)of the kth visit to the statey is a stopping time. From strong Markov property, the next return to statey is independent of the past. That is, (τ_y⁺(k+1)−τ_y⁺(k): k∈N)_{is an}i.i.d. sequence, distributed identically toτ_y⁺ starting from an initial stateX₀=y. When X0=x̸=y, thenτ_y⁺is independent of sequence(τ_y⁺(k+1)−τ_y⁺(k):k∈_N)and distributed differently.

We observe that

Ny=m =ⁿτ_y⁺(m)<_∞,τ_y⁺(m+1) =_∞ô=∩^m_k=1ⁿτ_y⁺(k)−τ_y⁺(k−1)<_∞ô∩ⁿτ_y⁺(m+1)−τ_y⁺(m) =_∞ô. It follows from the strong Markov property for processX, that

Px

Ny=m =Px

n

τ_y⁺(1)<∞o ^m

k=2

∏

Py

n

τ_y⁺(k)−τ_y⁺(k−1)<∞o Py

n

τ_y⁺(m+1) =∞o .

Corollary 1.7. For a homogeneous Markov chain X:Ω→X^Z⁺, we have Py

Ny<∞ =1{fyy<1}^.

Proof. We can write the event

Ny<_∞ as the disjoint union of events

Ny=m form∈_Z₊, and the result follows from additivity of probability over disjoint events, and the expression for the conditional probability mass functionPy

Ny=m in Lemma 1.6.

Remark2. In particular, this corollary implies the following.

1. A transient state is visited a finite amount of times almost surely.

2. A recurrent state is visited infinitely often almost surely.

3. Since∑y∈XNy=∞, it follows that all states can be transient in a finite state Markov chain.

Proposition 1.8. A state y∈Xis recurrent iff∑k∈Np^(k)yy =∞.

Proof. For any statey∈X, we can write

p^(k)_yy =Px{X_k=y}=_E_x₁_{X

k=y}.

Using monotone convergence theorem to exchange expectation and summation, we obtain

k∈

∑

N

p^(k)_yy =_E_y

∑

k∈N

1{X_k=y}=_E_yNy.

Thus,∑k∈Np^(k)_yy represents the expected number of returnsEyNyto a stateystarting from statey, which we know to be finite if the state is transient and infinite if the state is recurrent.

Proposition 1.9. Transience and recurrence are class properties.

Proof. Let us start with proving recurrence is a class property. Letxbe a recurrent state and letx↔y.

Then, we will show thatyis a recurrent state. From the reachability, there exist somem,n>0, such that p^(m)xy >0 andp⁽ⁿ⁾yx >0. As a consequence of the recurrence,∑s∈Z+p^(s)xx =∞. It follows thatyis recurrent by observing

k∈

∑

Z+

p^(k)yy ⩾

∑

s∈Z+

p^(m+n+s)yy ⩾

∑

s∈Z+

p⁽ⁿ⁾yx p^(s)xxp^(m)xy =∞.

Now, ifxwere transient instead, we conclude thatyis also transient by the following observation

s∈

∑

Z+

p^(s)yy ⩽^∑^s∈^Z⁺^p

(m+n+s) xx

p⁽ⁿ⁾yx p^(m)xy

<∞.

Corollary 1.10. If y is recurrent, then for any state x such that x↔y, fxy=1.

2

(3)

2 Invariant distribution

Definition 2.1. For a time-homogeneous Markov chainX:Ω→X^Z⁺ with transition matrixP, a distri- butionπ∈ M(X)is calledinvariantif it is a left eigenvector of the probability transition matrixPwith eigenvalue unity, or

π=πP.

Remark 3. Recall that ν(n)∈ M(X) _where νx(n) =P{Xn=x}for all x∈X, denotes the probability distribution of the Markov chainXbeing in one of the states at stepn∈N. Then, if ν(0) =π, then ν(n) =ν(0)Pⁿ=πfor all time-stepsn∈_N.

Definition 2.2. For a time-homogeneous Markov chain X:Ω →_X^Z⁺ with transition matrix P, the stationary distribution is defined asν(_∞)≜limn→∞ν(n).

Remark 4. For a Markov chain with initial distribution being invariant, the stationary distribution is invariant distribution.

Example 2.3 (Simple random walk on a directed graph). LetG= (V,E)be a finite directed graph.

We define a simple random walk on this graph as a Markov chain with state spaceVand transition matrixP:V×V→[0, 1]wherepxy≜_deg¹

out(x)1_{(x,y)∈E}. We observe that vector(deg_out(x):x∈X) is a left eigenvector of the transition matrixPwith unit eigenvalue. Indeed we can very that

x∈

∑

X

deg_out(x)pxy=

∑

x∈X1{(x,y)∈E}=deg_out(y).

Since∑x∈Xdeg_out(x) =2|E|, it follows thatπ:X→[0, 1]defined byπ_x≜ ^deg_2|E|^out^(x)for eachx∈V, is the equilibrium distribution of this simple random walk.

2.1 Existence of an invariant distribution

Proposition 2.4. Consider an irreducible and aperiodic homogeneous DTMC X:Ω→_X^Z⁺ with transition matrix P and starting from initial state X0=x. Let the positive vectorπ˜_x:X→R+defined as

˜

πx(y)≜Ex τ_x⁺ n=1

∑

1{X_n=y}=_E_x

∑

n∈N1{n⩽τx⁺}¹^{Xⁿ^=y}^, ^y∈_X.

Thenπ˜_x=π˜_xP if Px{τ_x⁺<∞}=1, andπ≜_E^π^˜^x

xτ_x⁺ is a stationary distribution ifExτ_x⁺<∞.

Proof. We will first show thatπis a distribution on state spaceX. We first observe that

y∈

∑

X

˜

πx(y) =

∑

y∈X τ_x⁺ n=1

∑

1{X_n=y}=

τ_x⁺ n=1

∑

1{X_n∈X}=_E_xτ_x⁺.

Thus ˜πx(y) =_E_x_∑^τ_n=1^x⁺ ₁_{X_n_=y}⩽Exτ_x⁺for all statesy∈_{X. If}_E_xτ_x⁺<_{∞, then ˜}πx(y)<_∞for eachy∈_X.

Further, we have ˜πx(x) =1. Since ˜πx(y)⩾0, it follows that _E^π^˜^x

xτx⁺ is a distribution on the state spaceX.

We next show that ˜πxis an invariant distribution of DTMCX. Using the monotone convergence theorem, we can write

w∈

∑

X

˜

πx(w)pwz=

∑

n∈N

∑

w∈X

Px

τ_x⁺⩾n,Xn=w P({Xn+1=z} | {Xn=w}).

We first focus on the termw=x. We see that{Xn=x,τ_x⁺⩾n}={τ_x⁺=n}. Hence, from the strong Markov property, we havePx{Xn=x,Xn+1=z,τ_x⁺⩾n}=Px{τ_x⁺=n}pxz. Summing over alln∈_N, we get

˜

π_x(x)pxz=

∑

n∈N

Px

Xn=x,Xn+1=z,τ_x⁺⩾n =pxz

∑

n∈N

Px

τ_x⁺=n =pxz.

3

(4)

We next focus on the termsw̸=x, such that{Xn=w,τ_x⁺⩾n}={Xn=w,τ_x⁺⩾n+1} ∈_F_n. Hence, from the Markov property ofX, we can write

Px

τ_x⁺⩾n+1,Xn=w,Xn+1=z =Px

τ_x⁺⩾n,Xn=w P({Xn+1=z} |Xn=w,τ_x⁺⩾n,X0=x )

=Px

τ_x⁺⩾n,Xn=w pwz. Summing both sides overn∈_Nandw̸=x, we get

w̸=x

∑

˜

πx(w)pwz=

∑

n∈N

∑

w̸=x

Px

τ_x⁺⩾n+1,Xn=w,Xn+1=z =

∑

n⩾2

Px

τ_x⁺⩾n,Xn=z

=π˜x(z)−Px

τ_x⁺⩾1,X₁=z =π˜x(z)−pxz. The result follows from summing both the cases.

2.2 Uniqueness of stationary distribution

Recall that distributionsπon state spaceXsuch thatπP=πis called a stationary distribution. Similarly, a functionh:X→_Ris calledharmonic atxif

h(x) =

∑

y∈X

pxyh(y).

A function isharmonic on a subsetD⊂_Xif it is harmonic at every statex∈D. That is, Ph=hfor a function harmonic on the entire state spaceX.

Lemma 2.5. For a finite irreducible Markov chain, a function f that is harmonic on all states inXis a constant.

Proof. Supposehis not a constant, then there exists a statex₀∈X, such thath(x₀)⩾h(y)for all states y∈X. Since the Markov chain is irreducible, there exists a statez∈_Xsuch thatpx₀,z>0. Let’s assume h(z)<h(x0), then

h(x0) =px₀,zh(z) +

∑

y̸=z

px₀,yh(y)<h(x0).

This implies that h(x0) =h(z)for all states zsuch thatpx₀,z >0. By induction, this implies that any h(x0) =h(y)for any states y reachable from state x0. Since all states are reachable from state x0by irreducibility, this implieshis a constant on the state spaceX.

Corollary 2.6. For any irreducible and aperiodic finite Markov chain, there exists a unique stationary distribu- tionπ.

Proof. For an aperiodic and irreducible DTMCX:Ω→_X^Z⁺with finite state spaceX, we havePx

n

τ_y⁺<_∞^o= 1 andExτ_y⁺<∞for all statesx,y∈X. Therefore, we have seen the existence of a positive stationary distributionπfor an irreducible and aperiodic finite Markov chain. Further, from previous Lemma we have that the dimension of null-space of(P−I)is unity. Hence, the rank ofP−Iis|X| −1. Therefore, all vectors satisfyingν=_νPare scalar multiples ofπ.

2.3 Stationary distribution for irreducible and aperiodic finite DTMC

For a finite state irreducible and aperiodic DTMCX:Ω→_X^Z⁺, we haveExτ_y⁺<_∞andPx

τy<_∞ =1 for allx,y∈X. That is, the return times are finite almost surely, and hence we can apply strong Markov property at these stopping times to obtain that DTMCXis a regenerative process with delayed renewal sequenceτ_y⁺:Ω→N^N, whereτ_y⁺(0)≜0, andτ_y⁺(n)≜infn

m>τ_y⁺(n−1):Xm=yo .

Theorem 2.7. For a finite state irreducible and aperiodic Markov chain X:Ω→X^Z⁺, its invariant distribution is same as its stationary distribution.

Proof. We can create an on-off alternating renewal function on this DTMCX, which is ON when in state y. Then, from the limiting ON probability of alternating renewal function, we know that

π(y)≜ lim

k→∞Px{X_k=y}= lim

n→∞

1 n

∑

n k=1

1{X_k=y}= ¹ Eyτ_y⁺. We observe thatπ(y) =^π^˜^y^(y)

Eyτy⁺ for each statey∈X. From the uniqueness of invariant distribution, it follows thatπis the unique invariant distribution of the DTMCX. We observe thatπ(x)is the long- term average of the amount of time spent in statexand from renewal reward theoremπ(x) =_E¹

xτx⁺. 4