Lecture-23: Reversed Processes

Lecture-23: Reversed Processes

1 Reversed Processes

Definition 1.1. LetX:Ω→X^Tbe a stochastic process with index setTbeing an additive ordered group such asRorZ. Then, ˆX^τ:Ω→X^T defined as ˆX^τ_t ≜Xτ−tfor allt∈Tis thereversed processfor some τ∈T.

Remark1. Note that a reversed process, doesn’t have to have the identical distribution to the original process. For a reversible processX, the reversed process would have identical distribution.

Lemma 1.2. If X:Ω→_X^Tis a Markov process, then the reversed processXˆ^τis also Markov for anyτ∈T.

Proof. LetF•be the natural filtration of the processX. From the Markov property of processX, for any eventF∈σ(Xu:u>t),H∈σ(Xs:s<t), statesx,y∈X, and timesu,s>0, we have

P(F| {Xt=y} ∩H) =P(F| {Xt=y}). Markov property of the reversed process follows from the observation, that

P(H| {Xt=y} ∩F) =^P(H∩ {Xt=y})P(F|H∩ {Xt=y})

P{Xt=y}P(F| {Xt=y}) =P(H| {Xt=y}).

Remark2. Even if the forward processXis time-homogeneous, the reversed process need not be time- homogeneous. For a non-stationary time-homogeneous Markov process, the reversed process is Markov but not necessarily time-homogeneous.

Theorem 1.3. If X:Ω→X^Ris an irreducible, positive recurrent, stationary, and homogeneous Markov process with transition kernel P:R→[0, 1]^X×X and invariant distribution π∈ M(X), then the reversed Markov processXˆ^τ:Ω→X^Ris also irreducible, positive recurrent, stationary, and homogeneous with the same invariant distributionπand transition kernelPˆ:R→[0, 1]^X×Xdefined for all t∈T and states x,y∈X, as

Pˆxy(t)≜ ^πy πxPyx(t).

Further, for any finite sequence of states x∈Xⁿ, finite sequence of times t∈Tⁿsuch that t1<t2<· · ·<tn, and any shiftτ∈R, we have

Pπ∩ⁿ_i=1{Xt_i =xi}=P^ˆπ∩ⁿ_i=1ⁿX^ˆ_t^τ_i =xi

o . Proof. We observe that ˆX^τ_ti =X_τ−t_ifor alli∈[n].

Step 1: We can verify that ˆPis a probability transition kernel.

• ˆPxy⩾0 for allt∈T.

• ∑y∈XPˆxy(t) = ¹

πx∑y∈XπyPyx(t) =1.

Step 2: πis an invariant distribution for ˆP, since for all statesx,y∈_X

x∈

∑

X

πxPˆxy(t) =πy

∑

x∈X

Pyx(t) =πy.

1

Step 3: We next wish to show that ˆPdefined in the Theorem, is the probability transition kernel for the reversed process. Since the forward process is stationary and time-homogeneous, we can write the probability transition kernel for the reversed process as

P(X^ˆτ_t+s=y |X^ˆ_s^τ=x ) = ^Pπ{Xτ−t−s=y,Xτ−s=x}

P_π{X_τ−s=x} =^πyPyx(t) πx .

This implies that the reversed process is time-homogeneous and has the desired probability tran- sition kernel. Further,πis the invariant distribution for the reversed process and is the marginal distribution for the reversed process at any timet, and hence the reversed process is also station- ary.

Step 4: We now need to check if ˆP is irreducible. For an irreducible and positive recurrent Markov process with invariant distribution π, we haveπ_x>0 for each statex ∈X. Since the forward process is irreducible, there exists a timet⩾0 such thatPyx(t)>0 for states x,y∈X, and hence Pˆxy(t)>0 implying irreducibility of the reversed process.

Step 5: Now, we want to show thatP_π∩ⁿ_i=1{Xt_i =x_i}=P^ˆ_π∩ⁿ_i=1ⁿX^ˆ_t^τ

i =x_io

. From the Markov property of the underlying processes and definition of ˆP, we can write

Pπ

∩ⁿ_i=1{Xt_i=_xi}=πx₁ n−1

∏

i=1

Px_ix_i+1(_ti+1−ti) =πx_n n−1

∏

i=1

Pˆx_i+1x_i((τ−ti)−(τ−ti+1)) =_P^ˆ_π∩_i=1^{n n}X^ˆτ_ti =_xi^o _. This follows from the fact thatπx₁Px₁x₂(t2−t1) =πx₂Pˆx₂x₁(t2−t1), and hence we have

πx₁ n−1

∏

i=1

Px_ix_i+1(ti+1−ti) =πx_n n−1

∏

i=1

Pˆx_i+1x_i(ti+1−ti).

For any finiten∈N, we see that the joint distributions of(Xt₁, . . . ,Xtn)and(Xs+t₁, . . . ,Xs+tn)are identical for alls∈T, from the stationarity of the processX. It follows that ˆX^τis also stationary, since(X^ˆ_t^τ_n, . . . , ˆX_t^τ₁)and(X^ˆ_s+t^τ _n, . . . , ˆX_s+t^τ ₁)have the identical distribution.

1.1 Reversed Markov Chain

Corollary 1.4. If X:Ω→X^Zis an irreducible, stationary, homogeneous Markov chain with transition matrix P and invariant distributionπ, then the reversed chainXˆ^τ:Ω→X^Zis an irreducible stationary, time homogeneous Markov chain with the same invariant distributionπ, and transition matrixP defined asˆ Pˆxy≜ ^π_π^y_xPyx,for all x,y∈X.

Corollary 1.5. Consider an irreducible Markov chain X with transition matrix P:X×_X→[0, 1]. If one can find a non-negative vectorα∈ M(_X)and other transition matrix P^∗:X×_X→[0, 1]that satisfies the detailed balance equation for all x,y∈_X

αxPxy=αyP_yx^∗ , (1)

then P^∗is the transition matrix for the reversed chain andαis the invariant distribution for both chains.

Proof. Summing both sides of the detailed balance equation (1) overx, we obtain thatαis the invariant distribution of the forward chain. SinceP_yx^∗ = ^αx_α^Pxy

y , it follows thatP^∗:X×_X→[0, 1]is the transition matrix of the the reversed chain andαis the invariant distribution of the reversed chain.

1.2 Reversed Markov Process

Corollary 1.6. If X:Ω→X^Ris an irreducible, stationary, homogeneous Markov process with generator matrix Q and invariant distributionπ, then the reversed processXˆ^τ:Ω→X^Ris also an irreducible, stationary, homo- geneous Markov process with same invariant distributionπand generator matrixQ defined asˆ Qˆxy≜ ^π_π^y_xQyx, for all x,y∈X.

Corollary 1.7. Let Q:X×_X→_Rdenote the rate matrix for an irreducible Markov process. If we can find Q^∗:X×_X→_Rand a distributionπ∈ M(_X)such that for y̸=x∈_{X, we have}

πxQxy=πyQ^∗_yx, and

∑

y̸=x

Qxy=

∑

y̸=x

Q^∗_xy,

then Q^∗is the rate matrix for the reversed Markov process andπis the invariant distribution for both processes.

2

2 Applications of Reversed Processes

2.1 Truncated Markov Processes

Definition 2.1. For a Markov processX:Ω→_X^R, and a subsetA⊆_Xthe boundary ofAis defined as

∂A≜y∈/A:Qxy>0, for somex∈A . Example 2.2. Consider a birth-death process. LetA={3, 4}. Then,∂A={2, 5}

Definition 2.3. Consider a transition rate matrixQ:X×X→Ron the countable state spaceX. Given a nonempty subsetA⊆X, the truncation ofQtoAis the transition rate matrixQ^A:A×A→R, where for allx,y∈A

Q^A_xy≜

(Qxy, y̸=x,

−_∑z∈A\{x}Qxz, y=x.

Proposition 2.4. Suppose X:Ω→X^Ris an irreducible, time-reversible CTMC on the countable state spaceX, with generator Q:X×X→Rand invariant distributionπ∈ M(X). Suppose the truncated Markov process X^A to a set of states A⊆Xwith generator matrix Q^A is irreducible. Then, X^A:Ω→A^R at stationarity is time-reversible, with invariant distributionπ^A∈ M(A)defined as

π_y^A≜ ^πy

∑x∈Aπx, y∈A.

Proof. It is clear that π^A is a distribution on state space A. We must show the reversibility with this distributionπ^A. That is, we must show for all statesx,y∈A

π_x^AQxy=π_y^AQyx. However, this is true since the original chain is time reversible.

Example 2.5 (Limiting waiting room: M/M/1/K). Consider a variant of the M/M/1 queueing system with loadρ≜ ^λ_µ that has a finite buffer capacity of at mostK customers. Thus, customers that arrive when there are alreadyKcustomers present arerejected. It follows that the CTMC for this system is simply theM/M/1 CTMC truncated to the state space{0, 1, . . . ,K}, and so it must be time-reversible with invariant distribution

π_i= ^ρ

i

∑^kj=0ρ^j

, 0⩽i⩽k.

Example 2.6 (Two queues with joint waiting room). Consider two independentM/M/1 queues with arrival and service ratesλ_iandµ_irespectively fori∈[2]. Then, the joint distribution of two queues is

π(n₁,n2) = (1−ρ₁)ρⁿ₁¹(1−ρ2)ρⁿ₂², n₁,n2∈_Z+.

Suppose both the queues are sharing a common waiting room, where if arriving customer findsRwait- ing customer then it leaves. Defining A≜n∈_Z²₊:n1+n2⩽R , we observe that the joint Markov porcess is restricted to the set of statesA, and the invariant distribution for the truncated Markov pro- cess is

π(n1,n2) = ^ρ

n₁ 1 ρⁿ₂²

∑(m₁,m₂)∈Aρ^m₁¹ρ^m₂², (n1,n2)∈A.

3