Lecture-21: Reversible Processes and Queues

Lecture-21: Reversible Processes and Queues

1 Equilibrium distribution of M/M/1 Queue

Recall the global balance equations for equilibrium distributionπ∈[0, 1]^Z+ are

−π0λ+π1µ=0, πk−1λ−π_k(λ+µ) +πk+1µ=0, k∈N.

Recognizing thatπ is a one-sided countably infinite sequence, we denote the discrete Fourier transform or thez- transform of the distributionπ∈[0, 1]^Z+ as

Π(z)_,

∑

k∈Z+

π_kz^k. Using this notation, we can compute

(λ+µ)

∑

k∈N

π_kz^k=λ

∑

k∈N

πk−1z^k+µ

∑

k∈N

πk+1z^k. Using the definition ofΠ(z), we can re-write this as

(λ+µ)(_Π(z)−π0) =λzΠ(z) +µz⁻¹(_Π(z)−π0−zπ₁). Sinceπ1=π0ρwhereρ= ^λ_µ, we can re-arrange the terms to get

Π(z) = ^π0 (1−ρz)^. Inverting thez-transform, we get

π_k=π₀ρ^k, k∈N.

2 Reversed Processes

Definition 2.1. LetX:Ω→X^T be a stochastic process with index setT being an additive ordered group such asR orZ. Then,Xˆ^τ:Ω→X^T defined asXˆ^τ(t)_,X(τ−t)for allt∈T is thereversed processfor someτ∈T.

Remark1. Note that a reversed process, doesn’t have to have the identical distribution to the original process. For a reversible processX, the reversed process would have identical distribution.

Lemma 2.2. If X:Ω→X^T is a Markov process, then the reversed processXˆ^τis also Markov for anyτ∈T . Proof. LetFt=σ(X(s)_:s6t)denote the history of the process until timet. From the Markov property of process X, we have for any eventB∈Ft+u, statesx,y∈Xand timesu,s>0

P(B| {X_t=y,Xt−s=x}) =P(B| {X_t=y}). Markov property of the reversed process follows from the observation, that P({Xt−s=x} | {X_t=y}∩B) =^P({Xt−s=x,X_t=y} ∩B)

P({X_t=y} ∩B) =^P{Xt−s=x,X_t=y}P(B| {Xt−s=x,X_t=y})

P{X_t=y}P(B| {X_t=y}) =P({Xt−s=x} | {X_t=y}).

1

Remark2. Even if the forward processXis time-homogeneous, the reversed process need not be time-homogeneous.

For a non-stationary time-homogeneous Markov process, the reversed process is Markov but not necessarily time- homogeneous.

Theorem 2.3. If X:Ω→X^Ris an irreducible, positive recurrent, stationary, and homogeneous Markov process with transition kernel P and equilibrium distributionπ, then the reversed Markov processXˆ^τ:Ω→X^Ris also irreducible, positive recurrent, stationary, and homogeneous with the same equilibrium distributionπand transition kernelP givenˆ

Pˆ_xy(t)_,^πy

π_xP_yx(t) for all t∈T,and states x,y∈X.

Further, for any finite sequence x∈Xⁿ, we have

P_π∩ⁿ_i=1{X_ti =x_i}=P^ˆ_π∩ⁿ_i=1Xˆ_ti =xn−i+1 .

Proof. We can check thatPˆis a probability transition kernel, sincePˆ_xy>0 for allt∈T and

y∈

∑

X

Pˆxy(t) = ¹ π_x

∑

y∈X

πyPyx(t) =1.

Further, we see thatπis an invariant distribution forP, since for all statesˆ x,y∈X

x∈

∑

X

πxPˆ_xy(t) =πy

∑

x∈X

P_yx(t) =πy.

We next wish to show thatPˆdefined in the Theorem, is the probability transition kernel for the reversed process. Since the forward process is stationary and time-homogeneous, we can write the probability transition kernel for the reversed process as

P(X^ˆ_τ−t+s^τ =x |Xˆ_τ−t^τ =y ) =^P

Xˆ_τ−t+s^τ =x,Xˆ_τ−t^τ =y

PXˆ_τ−t^τ =y =^Pπ{Xt−s=x,X_t=y}

P_π{X_t=y} =^πxPxy(0,s) π_y .

This implies that the reversed process is time-homogeneous and has the desired probability transition kernel. Further, πis the stationary distribution for the reversed process and is the marginal distribution for the reversed process at any timet, and hence the reversed process is also stationary.

For an irreducible and positive recurrent Markov process with stationary distributionπ, we haveπ_x>0 for each statex∈X. Since the forward process is irreducible, there exists a timet>0 such thatP_yx(t)>0 for statesx,y∈X, and hencePˆ_xy(t)>0 implying irreducibility of the reversed process. From the Markov property of the underlying processes and definition ofP, we can writeˆ

P_π{X_t1 =x₁,. . .,X_tn =x_n}=π_x1

n−1

∏

i=1

P_xixi+1(ti+1−t_i) =π_xn

n−1

∏

i=1

Pˆ_xi+1xi(ti+1−t_i) =P^ˆ_π{Xˆ_t1=x_n,. . .,Xˆ_tn =x₁}.

This follows from the fact that

π_x1P_x1x2(t₂−t₁) =π_x2Pˆ_x2X1(t₂−t₁), and hence we have

πx₁ n−1

∏

i=1

P_xixi+1(t_i+1−t_i) =πxn

n−1

∏

i=1

Pˆ_xi+1xi(t_i+1−t_i).

Let’s takeτ=t_n+t₁, then we haveXˆ_t^τ=X(t_n+t₁−t)and hence we have(X_t1,. . .,X_ti,. . .,X_tn) = (X^ˆ_t^τ

n,. . .,Xˆ^τ(t₁+ t_n−t_i),. . .,Xˆ_t^τ

1). From the Markovity of the reversed process, we can write Pˆπ

Xˆ_t^τ_n=x₁,. . .,Xˆ_t^τ

1=x_n =P^ˆπ

Xˆ_t^τ

1 =x_n,. . .,Xˆ_t^τ_n =x₁ =π_xn

n−1

∏

i=1

Pˆ(X^ˆ_τ−t^τ _n−i=xn−i|Xˆ_τ−t^τ _n−i+

1 =xn−i+1)

=πxn

n−1

∏

i=1

Pˆ_{xn−i+1xn−i}(tn−i+1−tn−i) =πxn

n−1

∏

i=1

Pˆ_xi+1xi(t_i+1−t_i).

For any finiten∈N, we see that the joint distributions of(X_t1,. . .,X_tn)and(Xs+t₁,. . .,Xs+t_n)are identical for alls∈T, from the stationarity of the processX. It follows thatXˆ is also stationary, since(Xˆ_tn,. . .,Xˆ_t1)and(Xˆs+t_n,. . .,Xˆs+t₁) have the identical distribution.

2

Corollary 2.4. If X :Ω→X^Z is an irreducible, stationary, homogeneous Markov chain with transition matrix P and equilibrium distributionπ, then the reversed chainXˆ^τ:Ω→X^Zis an irreducible stationary, time homogeneous Markov chain with the same equilibrium distributionπ, and transition matrixP given byˆ

Pˆ_xy= ^πy πx

P_yx.

Corollary 2.5. If X:Ω→X^Ris an irreducible, stationary, homogeneous Markov process with generator matrix Q and equilibrium distributionπ, then the reversed processXˆ^τ:Ω→X^Ris also an irreducible, stationary, homogeneous Markov process with same equilibrium distributionπand generator matrixQ such thatˆ

Qˆxy= ^πy π_xQyx. Proof.

Corollary 2.6. Consider irreducible Markov chain with transition matrix P:X×X→[0, 1]. If one can find a non- negative vectorα∈[0, 1]^Xand other transition matrix P^∗:X×X→[0, 1]such that∑x∈Xα_x=1 and satisfies the detailed balance equation

α_xP_xy=α_yP_yx^∗,

thenα is the stationary probability vector of P and P^∗is the transition matrix for the reversed chain.

Proof. SummingαiPi j=αjP_ji^∗overigives,∑iαiPi j=αj. Henceαis are the stationary probabilities of the forward and reverse process. SinceP^∗_ji= ^{αiPi j}

αj ,P_{i j}^∗are the transition probabilities of the reverse chain.

Corollary 2.7. Let Q:X×X→Rdenote the rate matrix for an irreducible Markov process. If we can find Q^∗: X×X→[0, 1]and a vectorπ∈[0, 1]^Xsuch that∑_x∈Xπ_x=1and for y6=x∈X, we have

πxQ_xy=πyQ^∗_yx, and

∑

y6=x

Q_xy=

∑

y6=x

Q^∗_xy,

then Q^∗is the rate matrix for the reversed Markov chain andπis the equilibrium distribution for both processes.

3 Applications of Reversed Processes

3.1 Truncated Markov Processes

Definition 3.1. For a Markov processX:Ω→X^R, and a subsetA⊆Xthe boundary ofAis defined as

∂A,

y∈/A:Q_xy>0, for somex∈A .

Definition 3.2. Consider a transition rate matrixQ:X×X→Ron the countable state spaceX. Given a nonempty subsetA⊆X, the truncation ofQtoAis the transition rate matrixQ^A:A×A→R, where for allx,y∈A

Q^A_xy=

(Q_xy, y6=x,

−∑z∈A\{x}Q_xz, y=x.

Proposition 3.3. Suppose X:Ω→X^Ris an irreducible, time-reversible CTMC on the countable state spaceX, with generator Q:X×X→Rand stationary probabilitiesπ∈[0, 1]^X. Suppose the trunctated Markov process to a set of states A⊆Xis irreducible. Then, any stationary CTMC with state space A and generator Q^Ais also time-reversible, with stationary probabilities

π_y^A= ^πy

∑x∈Aπ_x, y∈A.

Proof. It is clear thatπ^Ais a distribution on state spaceA. We must show the reversibility with this distributionπ^A. That is, we must show for alli,j∈A

π_x^AQ_xy=π_y^AQ_yx. However, this is true since the original chain is time reversible.

3

Example 3.4 (Limiting waiting room: M/M/1/K). Consider a variant of the M/M/1 queueing system that has a finite buffer capacity of at mostKcustomers. Thus, customers that arrive when there are alreadyKcustomers present are ‘rejected’. It follows that the CTMC for this system is simply the M/M/1 CTMC truncated to the state space{0, 1,. . .,K}, and so it must be time-reversible with stationary distribution

πi= ^ρ

i

∑^kj=0

ρ^j, 06i6k.

Example 3.5 (Two queues with joint waiting room). Consider two independent M/M/1 queues with arrival and service ratesλiandµirespectively fori∈[2]. Then, joint distribution of two queues is

π(n₁,n₂) = (1−ρ₁)ρ₁ⁿ¹(1−ρ2)ρ₂ⁿ², n₁,n₂∈Z+.

Suppose both the queues are sharing a common waiting room, where if arriving customer findsRwaiting customer then it leaves. In this case,

π(n₁,n₂) = ^ρ

n₁ 1 ρ₂ⁿ²

∑(m₁,m2)∈Aρ₁^m1ρ₂^m2, (n₁,n₂)∈A⊆Z+×Z+.

4