22 Z transform
Theorem 2: In an irreducible and aperiodic homogeneous Markov chain, the limiting probability distribution
6.4 Birth-Death processes
This section discusses a special class of continuous-time homogenous Markov chains that have the property that state transi- tions are permitted from state j only to states j-1 and j+1 (if these states exist). This is well-suited to describe processes like the arrival and departure of customers from a queue (the subject of queueing theory, after all!) where the state index corre- sponds to the number of customers awaiting service. More precisely, if the number of customers in the system is j, then the Markov chain is considered to be in state j. Customer arrivals cause the number of customers in the system to increase by one, which moves the process to state j+1 and this happens at a rate qj,j+1. Similarly, customer departures (due to service) cause the process to move from state j to state j-1, and this happens at the rate qj,j-1. In keeping with standard terminology, we denote:
(EQ 85)
and these are also called the birthand death rates respectively. Note that these rates can be state-dependent (but cannot be time-dependent due to homogeneity). Also, by definition, at state i, the transition rates qij are 0 for all jother than i, i-1, and i+1. Given this fact, Equation 82, and Equation 85, we find that:
qjj = - (λj + μj) (EQ 86)
For a birth-death process, being in state j has the intuitive meaning that the population size is j, that is, there are j customers in the queueing system. Note that when j = 1, we have one customer in the system, the customer that is receiving service, and there are none in the queue. Generalizing, in state j, we have one customer receiving service and j-1 in the queue awaiting service.
6.4.1 Time-evolution of a birth-death process
Because a birth-death process is a continuous-time Markov chain, the time evolution of is given by Equation 83. We substitute Equation 85 and Equation 86 to find:
(EQ 87)
t→∞ πj*
qjjπj* qkjπk*
k
∑
≠j+ = 0
πj*
j
∑
= 1λj = qj j, +1 μj = qj j, –1
πj( )t
dπj( )t
---dt = –(λj+μj)πj( )t +λj–1πj–1( ) μt + j+1πj+1( ) jt ≥1 dπ0( )t
---dt = –λ0π0( )t +μ1π1( ) j = 0t
DRAFT - Version 2 - Birth-Death processes
This describes the time-evolution of a birth-death system. In practice, solving these equations is complex, and does not give too many insights into the structure of the system. These are better obtained from the stationary probability distribution, which we study next.
6.4.2 Stationary probability distribution of a birth-death process
Because a birth-death process is an ergodic continuous-time Markov chain, its stationary probability distribution is given by Equation 84. Denoting the stationary probability of being in state j by Pj and substituting Equation 85 and Equation 86 into Equation 84, we obtain the following equations:
(EQ 88)
In matrix form, we can write the first two equations as
PQ = 0 (EQ 89)
where
(EQ 90)
The two matrices are infinite-dimensional if the population size is unbounded. By defining P(t) as
we can rewrite Equation 87 as
dP(t)/dt = P(t)Q (EQ 91)
6.4.3 Finding the transition-rate matrix
The Q matrix defined by Equation 90 is also called the transition rate matrix. It is important because it allows us to derive both the time-dependent evolution of the system (i.e., Pj(t)), through Equation 91, and the long-term probability of being in state j, through Equation 89. Thus, in practice, the first step in studying a birth-death process is to write down its Q matrix.
Consider the following representation of a generic birth-death process:
FIGURE 18. State-transition-rate diagram for a birth-death process πj*
0 = –(λj+μj)πj*+λj–1πj*–1+μj+1πj*+1 j≥1 0 = –λ0π0*+μ1π1* j = 0
πj
j
∑
= 1P = π0* π1* π2* …
Q λ
– 0 λ0 0 … …
μ1 –(λ1+μ1) λ1 … … 0 μ2 –(λ2+μ2) λ2 …
… … … … …
=
P t( ) = π0( ) πt 1( ) πt 2( ) …t
0
1
2 1
2 0
1
2
3
. . . j-1
j
j+1 j
j+1 j-1
j
j+1
j+2
. . .
DRAFT - Version 2 - Finding the transition-rate matrix
183
Here, we represent each state j by a circle and we label the arc from state j to state k with the transition rate qjk. From this fig- ure, we can determine Q for a birth-death process as follows: notice that the diagonal elements of Q, i.e., qjj are the negative of the quantities leaving state j. Focussing on the jth column, the qj-1,jth elements, immediately above the diagonal (such as element q01), are the rates entering state j from state j-1, i.e., λj-1 and the qj+1,jth elements, immediately below the diagonal (such as element q32), are the rates entering state j from state j+1. All other elements are 0. In each row, the quantities sum to zero, due to Equation 82.
Thus, given the state-transition-rate diagram, it is possible to quickly construct Q and use this to obtain the time-dependent and time-independent (stationary) probabilities of being in each state. We will now use this approach to study some standard birth-death systems.
Note that this inspection approach also applies to all continuous time Markov chains, where we can determine the elements of the Q matrix by inspecting the corresponding state-transition-rate diagram, then solving for P and P(t) using Equation 89 and Equation 91.
EXAMPLE 18: TRANSITION-RATEMATRIXFORABIRTH-DEATHPROCESS
Consider the state-rate-transition diagram in Figure 19. What are the P and Q matrices for this system? What are the equa- tions for its time-evolution and the long-term probability of being in each state?
FIGURE 19. A simple birth-death process
The P matrix is and . By inspection, we can write the Q matrix as
Q =
Therefore, the time-evolution of state probabilities is given by:
which is a system of differential equations that can be solved to give the evolution of the time-varying probabilities . The long-term probability of being in each state is given by:
0
5
2 1
8 1
5
4
10 3
π0* π1* π2* π3* P t( ) = π0( ) πt 1( ) πt 2( ) πt 3( )t
1
– 1 0 0
5 –10 5 0 0 8 –12 4 0 0 10 –10
d
dt---⎝⎛ π0( ) πt 1( ) πt 2( ) πt 3( )t ⎠⎞ π0( ) πt 1( ) πt 2( ) πt 3( )t 1
– 1 0 0
5 –10 5 0 0 8 –12 4 0 0 10 –10
=
π·0 ( ) πt ·1
( ) πt ·2 ( ) πt ·3
( )t π0( ) πt 1( ) πt 2( ) πt 3( )t 1
– 1 0 0
5 –10 5 0 0 8 –12 4 0 0 10 –10
=
πi( )t
DRAFT - Version 2 - Birth-Death processes
which is a system of linear equations in four variables that can be solved to obtain the stationary probability distribution of the chain.
6.4.4 A pure-birth (Poisson) process
Consider a system where λj = λfor all j (the departure rate from all states is the same), and μj= 0 for all j (the death rate is 0).
This represents a Poisson process whose population grows without bound and whose rate of growth is independent of the population size (that is, we expect the population to grow by 1 every 1/λseconds independent of the current population size).
We study two properties of this process: the probability of being in state j at time t, which corresponds to having j arrivals in time t, and the distribution of inter-arrival times, that is, the expected time between going from any state to the adjacent state.
We can derive the probability of being in any state directly from Equation 87. Substituting the values for and μ in this equation, we get:
(EQ 92)
The second equation is a trivial differential equation whose solution is given by
(EQ 93)
We substitute this into the first equation to get
(EQ 94)
whose solution is
(EQ 95)
By repeatedly substituting this into the first equation, we obtain
(EQ 96)
This is the density function for the Poisson distribution (see Section 1.5.4 on page 20) with parameter . Thus, for a Pois- son process with parameter λ, the probability of j arrivals in time t, which is also the probability of being in state j at time t, is given by a Poisson distribution with parameter λt. Because the mean of the Poisson distribution is also its parameter, the expected number of arrivals in time t is λt. This is intuitively pleasing: the arrival rate is λ so in time t we should see, on aver- age, λt arrivals.
EXAMPLE 19: POISSONPROCESS
π0* π1*π2* π3* 1
– 1 0 0
5 –10 5 0 0 8 –12 4 0 0 10 –10
0
=
λ
λ
dπj( )t
---dt = –λπj( )t +λπj–1( ) jt ≥1 dπ0( )t
---dt = –λπ0( ) t
π0( )t = e–λt
dπ1( )t
---dt = –λπ1( )t +λe–λt
π1( )t = λe–λt
πj( )t ( )λt j ---j! e–λt
=
λt
DRAFT - Version 2 - Stationary probability distribution for a birth-death process
185
Consider students arriving to a class as a Poisson process at a mean rate of 5 students/second. What is the probability that the classroom has (a) 2 students after 2 seconds (b) 10 students after 2 seconds?
Solution
We have λ = 5 and t = 2, so the Poisson parameter is 10. The probability of having 2 students in the classroom after 2 seconds is (2) = (102/2!) e-10 = 50*e-10 = 2.26*10-3. This is a rather unlikely event.
The probability of having 10 students in the classroom after 2 seconds is (2) = (1010/10!) e-10 = 0.125. Note that the expected number of students after 2 seconds is 10, yet the probability that the expected number of students is actually achieved is only one in eight!
We now derive the interarrival time distribution for a Poisson process. Let adenote the continuous random variable that rep- resents the time between any two arrivals: we seek the distribution for a. Consider the cumulative density function of a, given by the probability P( ) = 1 - P(a > t). But P(a > t) is just P(0 customer arrivals in time (0, t)) = 1- (t) =
, . The density function is given by differentiating this expression to get:
(EQ 97)
We recognize this as an exponential distribution (see Section 1.6.3 on page 23). This gives us the following important result:
The interarrival times for a Poisson process are drawn from an exponential distribution.
We note that the exponential distribution is memoryless. Thus, for a Poisson process, not only is the rate of transitioning to the next state (the birth rate) independent of the current population size, the time at which this transition occurs does not depend on how long the process has been at the current population size.
6.4.5 Stationary probability distribution for a birth-death process
We now return to computing the stationary probability distribution for a general birth-death process using Equation 88. From the second equation, we immediately obtain
(EQ 98)
Substituting this into the first equation, we find that
(EQ 99)
Repeating this substitution, we find that Pj is given by
(EQ 100)
We therefore obtain the long-term probabilities of being in any state j as a function of the probability of being in state 0 and the system parameters. Knowing that these probabilities sum to 1, we can determine
π2
π10
a≤t π0
1–e–λt t≥0
a t( ) = λe–λt
π1* λ0 μ1 ---π0*
=
π2* λ0λ1 μ1μ2 ---π0*
=
πj* λ0λ1…λj–1 μ1μ2…μj
---π0* π0* λi μi+1 ---
i=0 j–1
∏
= =
DRAFT - Version 2 - The M/M/1 queue
(EQ 101)
This can be substituted back into Equation 100 to obtain the long-term probability of being in any state j. Of course, we need to ensure that the series in the denominator of Equation 101 actually converges! Otherwise, is undefined, and so are all the other s. It turns out that the condition for convergence (as well as for the chain to be ergodic) is the existence of a value j0 such that for all values of j > j0, λj < μj. We interpret this to mean that after the population reaches some threshold
, the rate of departures must exceed the rate of arrivals. This makes intuitive sense: otherwise, the population size will grow (in expectation) without bound and the probability of any particular population size will be 0.
EXAMPLE 20: GENERALEQUILIBRIUMSOLUTION
Find the equilibrium probabilities of being in each state for the birth-death process shown in Figure 19.
Solution
From Equation 101, we get
= 1/[1 + 1/5 + (1*5)/(5*8) + (1*5*4)/5*8*10)] = 0.73.
This can be substituted into Equation 100 obtain = 1/5 = 0.2 * 0.73 =0.146.
= 1/8 = 0.125 * 0.73 =0.09.
= 1/20 = 0.05 * 0.73 = 0.0365.
As a check, 0.73 + 0.146 + 0.09 + 0.0365 = 1.0025, which is within the rounding error.