Lecture-20: Invariant Distribution of Markov Processes
1 Class Properties
Definition 1.1. For a CTMC X:Ω→XR+ defined on the countable state spaceX⊆R, we say a state yisreachablefrom statexifPxy(t)>0 for somet>0, and we denotex→y. If two statesx,y∈Xare reachable from each other, we say that theycommunicateand denote it byx↔y.
Lemma 1.2. Communication is an equivalence relation.
Definition 1.3. Communication equivalence relation partitions the state spaceXinto equivalence classes calledcommunicating classes. A CTMC with a single communicating class is calledirreducible.
Theorem 1.4. A regular CTMC and its embedded DTMC have the same communicating classes.
Proof. It suffices to show thatx→yfor the regular Markov process iffx→yin the embedded chain. If x→yfor embedded chain, then there exists a pathx=x0,x1, . . . ,xn=ysuch thatpx0x1px1x2. . .pxn−1xn>0 and νx0νx1. . .νxn−1 >0. It follows that Sn is a stopping time and sum of n independent exponential random variables with ratesνx0, . . . ,νxn−1, and we can write
Pxy(t)⩾
n−1
∏
k=0
pxkxk+1Ex0[P{Tn+1>t−Sn} | {Z0=x0, . . .Zn=xn}]>0.
Conversely, if the statesyis not reachable from statexin embedded chain, then it won’t be reachable in the regular CTMC.
Corollary 1.5. A regular CTMC is irreducible iff its embedded DTMC is irreducible.
Remark 1. There is no notion of periodicity in CTMCs since there is no fundamental time-step that can be used as a reference to define such a notion. In fact, for any statex∈Xof a non-instantaneous homogeneous CTMC we havePxx(t)>e−νxt>0 for allt⩾0.
1.1 Recurrence and transience
Definition 1.6. For any statey∈X, we define the first hitting time to stateyafter leaving stateyas τy+=inf{t>Y0:Xt=y}.
The stateyis said to berecurrentifPy
n
τy+<∞o=1 andtransientifPy
n
τy+<∞o<1. Furthermore, a recurrent stateyis said to bepositive recurrentifEyτy+<∞andnull recurrentifEyτy+=∞.
Theorem 1.7. An irreducible pure jump CTMC is recurrent iff its embedded DTMC is recurrent.
Proof. There is nothing to prove for |X|=1. Hence, we assume |X|⩾2 without loss of generality.
Suppose that the embedded Markov chainZ:Ω→XNis recurrent. Let the initial stateZ0=x∈X, the number of visits to stateyduring successive visit to statexbe denoted byNxy, and thekth sojourn time in statey byYk(y). Since the embedded chain is irreducible and recurrent, it has no absorbing states.
This impliesNxyandNx≜∑y∈XNxyare finite almost surely, and the random sequenceY(y):Ω→RN+ isi.i.d.exponential with rateνy∈(0,∞), and sequencesY(y)are independent for each statey∈X. Since we can writeτx+=∑y∈X∑Nk=1xyYk(y), it follows that the recurrence timeτx+is finite almost surely.
Conversely, if the embedded Markov chain is not recurrent, it has a transient statex∈Xfor which Px{Nx=∞}>0. By the same argument,Px{τx+=∞}>0 and hence the CTMC is not recurrent.
Corollary 1.8. Recurrence is a class property.
1
Remark 2. Consider an irreducible positive recurrent DTMCX:Ω→XN with transition probability matrixp∈[0, 1]X×Xand invariant distributionu∈ M(X). LetX0=x andτx+(k)thekth return time to statex, and let Nxy(k)≜∑τn=τx+(k)+
x (k−1)+11{Xn=y}denote the number of visits to statey between two successive visits to statex. We observe thatτx+:Ω→NN is a renewal process with rewardNxy(k)in thekth renewal duration. From the renewal reward theorem, we get
uy= lim
N→∞
∑
N n=11{Xn=y}= ExNxy(k)
Eτx+(k) =uxExNxy(k). That is, we obtainExNxy(k) =uuy
x.
Theorem 1.9. Consider an irreducible recurrent CTMC X :Ω→XR+ with sojourn time ratesν∈RX+ and transition matrix p∈[0, 1]X×Xfor the embedded Markov chain. Let u∈RX+be any strictly positive solution of u=up, then
Exτx+= 1 ux
∑
y∈X
uy
νy, x∈X.
Proof. LetX0=x∈X, andNxybe the number of visits to statey∈Xbetween successive visits to state xin the embedded Markov chain. From the recurrence of the embedded Markov chain, we know that for any strictly positive solution tou=uPwe haveExNxy=uuy
x. LetYk(x)denote the sojourn time of the CTMCXin statexduring thekth visit. The random sequenceY(x):Ω→RN+ isi.i.d.exponential with rateνx. Therefore, we can write
τx+=Y0x+
∑
y∈X\{x}
Nxy k=1
∑
Yk(y).
We recall that jump chain and sojourn times are independent given the initial state, and henceNxyand Y(y)sequences are independent for each statey̸=x. Result follows from taking expectations on both sides, exchanging summation and expectations for positive random variables, to get
Exτx+=ExY0x+
∑
y∈X\{x}
Ex Nxy
k=1
∑
Yk(y)=
∑
y∈XEYk(y)ExNxy.
Remark3. An irreducible regular CTMC maybe null recurrent where embedded Markov chain is posi- tive recurrent.
Corollary 1.10. Consider an irreducible recurrent CTMC X:Ω→XR+ with sojourn time ratesν∈RX+and the transition matrix p for the embedded Markov chain. Let u be any strictly positive solution to u=up. Then, CTMC X is positive recurrent iff∑x∈Xux
νx <∞. In particular, the CTMC is positive recurrent iff∑x∈Xux νx =1.
2 Invariant Distribution
Definition 2.1. A distributionπ∈ M(X)is aninvariant distributionof a homogeneous continuous time Markov chainX:Ω→XR+ with probability transition kernelP:R+→[0, 1]X×XifπP(t) =πfor allt∈R+.
Remark4. Letν(0)∈ M(X)denote the marginal distribution of initial stateX0, then by definition of probability transition kernel for Markov processX, we can write the marginal distribution ofXtas
ν(t) =ν(0)P(t), t∈R+.
In general, we can write ν(s+t) =ν(s)P(t), and hence if there exists a stationary distribution π≜ lims→∞ν(s)for this processX, then we would haveπ=πP(t)for all timest∈R+.
Remark5. Recall that an irreducible DTMC is positive recurrent iff it has a strictly positive stationary distribution.
Corollary 2.2. For a homogeneous continuous time Markov chain X:Ω→XR+ with generator matrix Q, a distributionπ:X→[0, 1]is an equilibrium distribution iffπQ=0.
2
Proof. Recall that we can write the transition probability matrix P(t)at any time t∈R+ in terms of generator matrixQasP(t) =etQ. Using the exponentiation of a matrix, we can write
πP(t) =etQ=π+
∑
n∈N
tn
n!πQn, t∈R+. Therefore,πQ=0 iffπis an equilibrium distribution of the Markov processX.
Theorem 2.3. Let X:Ω→XR+ be an irreducible recurrent homogeneous CTMC with probability transition kernel P:R+→[0, 1]X×X, the transition rate sequenceν∈RX+, and the transition matrix for embedded jump chain p∈[0, 1]X. Then for all states x,y∈Xthelimt→∞Pxy(t)exists, this limit is independent of the initial state x∈Xand denoted byπy. Let u be any strictly positive invariant measure such that u=up. If∑x∈Xux
νx =∞, thenπx=0for all x∈X. If∑x∈Xuνx
x <∞then for all y∈X, πy=
uy νy
∑x∈Xux νx
= ν
−1y
Eyτy+
.
Proof. Fix a statey∈X, and define a processW:Ω→ {0, 1}R+ such thatWt=1{Xt=y}. Then, from the regenerative property of the homogeneous CTMC and renewal reward theorem, we have
t→lim∞Px{Xt=y}= ν
−1y
Eyτy+.
3
