Discrete and Continuous Markov Processes
3.4 BIRTH-DEATH PROCESSES
d
dtp t0( ) = −λ0p t0( ) +µ1 1P t( ) k=0 (3.35) In general, fi nding the time-dependent solutions of a birth-death process is diffi cult and tedious, and at times unmanageable. We will not pursue it further but rather show the solutions of some simple special cases. Under very general conditions and for most of the real-life systems, Pk(t) approaches a limit Pk as t →∝ and we say the system is in statistical equilibrium.
Example 3.10
A special case of the birth-death process is the pure-birth process where lk= l > 0 and mk= 0 for all k. Assume that the initial condition is P0(0) = 0, we have from Equations (3.31) and (3.32) the following equations:
dP t
dt P t P t k
dP t
dt P t
k
k k
( ) ( ) ( )
( ) ( )
= − + ≥
= −
λ λ −
λ
1
0
0
1
It was shown in Chapter 1 that the solution that satisfi es these set of equa- tions is a Poisson distribution:
P t t
k e
k
k
( ) ( ) t
= λ! −λ
This gives us another interpretation of the Poisson process. It can now be viewed as a pure-birth process.
Example 3.11
A typical telephone conversation usually consists of a series of alternate talk spurts and silent spurts. If we assume that the length of these talk spurts and silent spurts are exponentially distributed with mean 1/l and 1/m, respectively, then the conversation can be modelled as a two-state Markov process.
Solution
Let us defi ne two states; state 0 for talk spurts and state 1 for silent spurts. The state transition rate diagram is shown in Figure 3.7.
BIRTH-DEATH PROCESSES 97
The infi nitesimal generator is given by Q= −
−
λ λ
µ µ . Using expression (3.32) we have
d
dtP t P t P t d
dtP t P t P t
0 0 1
1 0 1
( ) ( ) ( )
( ) ( ) ( )
= − +
= −
λ µ
λ µ
Let us defi ne the Laplace transform of P0(t) and P1(t) as F s0 e stP t dt F s e stP t dt
0
0 1
0
( )=∞
∫
− ( ) and ( )=∞∫
− 1( ) and we havesF0(s) − P0(0) =−lF0(s) + mF1(s) sF1(s) − P1(0) = lF0(s) − mF1(s)
Let us further assume that the system begins in state 0 at t = 0, that is P0(0) = 1 and P1(0) = 0. Solving the two equations coupled with the initial condition, we have
F s s
s s s s
0
1 1
( )= ( + ) ( )
+ + =
+ ⋅ + + ⋅
+ + µ
λ µ µ λ µ
λ
λ µ λ µ
and F s
s s s s
1
1 1
( )= ( ) ( )
+ + =
+ ⋅ − + ⋅
+ + λ
λ µ λ λ µ
λ
λ µ λ µ
Inverting the Laplace expressions, we obtain the time domain solutions as
P t e
P t e
t
t 1
0
1 ( )
( )
)
( )
= + − +
= + + +
= + + −
+
−( +
− +
λ λ µ
λ λ µ µ
λ µ λ λ µ µ
λ µ
µ λ µ
λ µ
λ µ
e− +(λ µ)t
0 1
λ
µ
Figure 3.7 A two-state Markov process
For l = m = 1, the plot of the two curves are shown in Figure 3.8. We see that both p0= p1= 0.5 when t →∞:
Example 3.12
A Yule process is a pure-birth process where lk = kl for k = 0, 1, 2, . . . . Assuming that the process begins with only one member, that is P1(0) = 1, fi nd the time-dependent solution for this process.
Solution
Given the assumption of lk= kl, we have the following equation from Equa- tion (3.33):
dP t
dtk kP tk k Pk t k ( )= −λ ( )+λ( −1) −1( ) ≥1
Defi ne the Laplace transform of
P tk F sk e stP t dt ( ) as ( )=∞
∫
− k( )0
1
probability
0.8
0.6
0.4
0.2
0
0 0.5 1 1.5 2 2.5 3
time P0(t) P1(t)
Figure 3.8 Probability distribution of a two-state Markov chain
BIRTH-DEATH PROCESSES 99
then we have
F s P k F s
s k
k
k k
( ) ( ) ( ) ( )
= + −
+ −
0 λ 1 1
λ
Since the system begins with only one member at time t = 0, we have
P k
otherwise
k( )0 1 1
=
{
0 = Back substituting Fk(s), we haveF s s
F s s s
F sk s j k 1
2
1 1
1 1
2 1
( ) ( ) ( )
= +
= +
+
= +
=
∏
−λ λ
λ λ λ
jj s j λ
λ + +
( 1)
The expression can be solved by carrying out partial-fraction expansion of the right-hand side fi rst and then inverting each term. Instead of pursuing the general solution of Pk(t), let us fi nd the time distribution of P2(t) and P3(t) just to have a feel of the probability distribution:
F s s s s s
F s s s
2
3
1
2
1 1
2 1
2 2 ( )
( )
= +
+
= + − +
= +
+
λ
λ
λ λ λ
λ λ
λ λλ
λ λ λ λ
s+ s s s
= + −
+ +
+ 3
1 2
2 1
3 Inverting these two expressions, we have the time domain solutions as
P2(t) = e−lt− e−2lt P3(t) = e−lt− 2e−2lt+ e−3lt
The plots of these two distributions are shown in Figure 3.9.
Problems
1. Consider a sequence of Bernoulli trials with probability of ‘success’
p, if we defi ne Xn to be the number of uninterrupted successes that
have been completed at n trial; that is if the fi rst 6 outcomes are ‘S’,
‘F’, ‘S’,’S’, ‘S’, ‘F’; then X1 = 1, X2 = 0, X3 = 1, X4 = 2, X5 = 3, X6= 0.
(i) argue that Xn is a Markov chain
(ii) draw the state transition diagram and write down the one-step transition matrix.
2. By considering a distributed system that consists of three processors (A, B and C), a job which has been processed in processor A has a probability p of being redirected to processor B and probability (1 − p) to processor C for further processing. However, a job at processor B is always redirected back to process A after completion of its pro- cessing, whereas at processor C a job is redirected back to proces- sor B q fraction of the time and (1 − q) fraction of the time it stays in processor C.
(i) Draw the state transition diagram for this multi-processor system.
(ii) Find the probability that a job X is in processor B after 2 routings assuming that the job X was initially with processor A.
(iii) Find the steady-state probabilities.
(iv) Find value of p and q for which the steady state probabilities are all equal.
0.25
probability
0.2
0.15
0.1
0.05
0
0 1 2 3 4 5 6
time axis P2(t) P3(t)
Figure 3.9 Probability distribution of a Yule process
BIRTH-DEATH PROCESSES 101
3. Consider the discrete-time Markov chain whose state transition diagram is given in Figure 3.10.
(i) Find the probability transition matrix P˜. (ii) Find P˜5.
(iii) Find the equilibrium state probability vector p˜.
4. The transition probability matrix of the discrete-time counterpart of Example 3.11 is given by
P p p
q q
= −
−
1 1
Draw the Markov chain and fi nd the steady-state vector for this Markov chain.
5. Consider a discrete-time Markov chain whose transition probability matrix is given by
P =
1 0 0 1 (i) Draw the state transition diagram.
(ii) Find the k-step transition probability matrix P˜k. (iii) Does the steady-state probability vector exist?
6. By considering the lift example 3.2, for what initial probability vector will the stationary probabilities of fi nding the life at each fl oor be proportional to their initial probabilities? What is this proportional constant?
7. A pure-death process is one in which members of the population only die but none are born. By assuming that the population begins with N members, fi nd the transient solution of this pure death process.
8. Consider a population with external immigration. Each individual member in the population gives birth at an exponential rate l and dies at an exponential rate m. The external immigration is assumed to contribute to an exponential rate of increase q of the population.
Assume that the births are independent of the deaths as well as the external immigration. How do you model the population growth as a birth-death process?
1 2
1/4 3/4
1/4
3/4
Figure 3.10 A two-state discrete Markov chain