Optimality

THHHHT

2.4 Efficient Generation of k Random Bits

2.4.3 Optimality

Corollary 2.10. The algorithm Φ_k generates m random bits for some m with 0 ≤ m ≤ k, and m=k with probability at least1/2.

Proof. The sequence generated by Φk is independent and unbiased. This conclusion is immediate from lemma 2.9. Assume that the input sequence x ∈ Si for some i with 1 ≤ i ≤ w, then the probability ofm=k is

⌊^|S₂k^i|⌋2^k

|Si| ,

which is at least 1/2 based on the fact that |S_i| ≥2^k. Since this conclusion is true for allS_i with 1≤i≤w, we can claim thatm=k with probability at least 1/2.

Since the algorithm Φk generatesmrandom bits for some mwith 0≤m≤kfrom an arbitrary biased coin, we are able to generate k bits iteratively: After generating m random bits, we apply the algorithm Φk−m for generating k−m bits. Repeating this procedure, the total number of random bits generated will converge to kvery quickly. We call this scheme as an iterative scheme for generatingkrandom bits.

To generate krandom bits, we do not want to iterate Φ_k too many times. Fortunately, in the following theorem, we show that in our scheme the expected number of iterations is upper bounded by a constant 2.

Theorem 2.11. The expected number of iterations in the iterative scheme for generatingkrandom bits is at most2.

Proof. According to corollary 2.10, Φ_k generatesm=k random bits with probability at least 1/2.

Hence, the scheme stops at each iteration with probability more than 1/2. Following this fact, the result in the theorem is immediate.

Lemma 2.12. Given a biased coin with probability p being H, let n be the number of coin tosses used by the algorithm Φ_k, then

lim

k→∞

E[n]

k ≤ 1

H(p).

Proof. We consider the probability of having an input sequence of length at least n, denote as Pn. In this case, we can writen=k1+k2, wherek1 is the number of H’s andk2 is the number of T’s.

According to the construction of the stopping set, ( n−1

min(k1, k2)−1 )

<2^k n−1 min(k1, k2)−1.

Or we can write it as

( n−2 min(k₁, k₂)−2

)

<2^k.

Hence, we get an upper bound for min(k1, k2), which is

t_n= max{i∈ {0,1, ..., n}|

(n−2 i−2 )

<2^k}. (2.1)

Note that if

(n−2

n 2 −2

)

≥2^k, thentn is a nondecreasing function ofn.

According to the symmetry of our criteria, we can get

P_n ≤

tn

∑

i=0

(pⁱ(1−p)ⁿ⁻ⁱ+ (1−p)ⁱpⁿ⁻ⁱ) (n

i )

.

For convenience, we write

Qn=

t_n

∑

i=0

(pⁱ(1−p)ⁿ⁻ⁱ+ (1−p)ⁱpⁿ⁻ⁱ) (n

i )

,

thenPn≤Qn andQn is also a nondecreasing function ofn.

Now, we are ready to calculate the expected number of coin tosses required, which equals

E[n] =

∑∞ n=1

(Pn−Pn+1)n=

∑∞ n=1

Pn (2.2)

≤

k H(p)(1+ϵ)

∑

n=1

Pn+

∑∞ n=_H(p)^k (1+ϵ)

Qn+

∑∞ n=2_H(p)^k (1+ϵ)

Qn,

whereϵ >0 is a small constant. In the rest, we study the upper bounds for all the three terms when nis large enough.

For the first term, we have

k H(p)(1+ϵ)

∑

n=1

P_n≤ k

H(p)(1 +ϵ). (2.3)

Now let us consider the second term

2_H(p)^k (1+ϵ)

∑

n=_H(p)^k (1+ϵ)

Qn≤ k

H(p)(1 +ϵ)Q k H(p)(1+ϵ).

Using the Stirling bounds on factorials yields

nlim→∞

1 nlog₂

(n ρn

)

=H(ρ),

whereH is the binary entropy function. Hence, following (2.1), we can get

nlim→∞H(tn

n) = lim

n→∞

k n.

Whenn= _H(p)^k (1 +ϵ), we can write

nlim→∞H(t_n

n) = H(p) 1 +ϵ,

which implies that

nlim→∞

tn

n =p−ϵ1

for someϵ₁>0. So there exists anN₁ such that forn > N₁, ⁿ_n^t ≤p−ϵ₁/2.

By the weak law for the binomial distribution, given any ϵ₂>0 andδ >0, there is anN₂ such that forn > N2, with probability at least 1−δthere arei H’s among the ncoin tosses such that

|_nⁱ −p| ≤ϵ2. Lettingϵ2=ϵ1/2 andn=_H(p)^k (1 +ϵ) gives

Qn≤δ,

for anyδ >0 when n >max(N₁, N₂).

So for anyδ >0, whenkis large enough, we have

2_H(p)^k (1+ϵ)

∑

n=_H(p)^k (1+ϵ)

Q_n≤ k

H(p)(1 +ϵ)δ. (2.4)

To calculate the third term, we notice that Qn decays very quickly as n increase when n ≥ 2_H(p)^k (1 +ϵ). In this case,

Qn+1

Qn

=

∑tn+1

i=0 (pⁱ(1−p)ⁿ⁺¹⁻ⁱ+ (1−p)ⁱpⁿ⁺¹⁻ⁱ) (n+ 1

i )

∑t_n

i=0(pⁱ(1−p)ⁿ⁻ⁱ+ (1−p)ⁱpⁿ⁻ⁱ) (n

i )

≤

∑tn

i=0(pⁱ(1−p)ⁿ⁺¹⁻ⁱ+ (1−p)ⁱpⁿ⁺¹⁻ⁱ) (n+ 1

i )

∑t_n

i=0(pⁱ(1−p)ⁿ⁻ⁱ+ (1−p)ⁱpⁿ⁻ⁱ) (n

i )

≤ max^tn

i=0

(pⁱ(1−p)ⁿ⁺¹⁻ⁱ+ (1−p)ⁱpⁿ⁺¹⁻ⁱ) (n+ 1

i )

(pⁱ(1−p)ⁿ⁻ⁱ+ (1−p)ⁱpⁿ⁻ⁱ) (n

i )

≤ (1−p)max^tn

i=1

n+ 1 n+ 1−t_n

≤ (1−p)n n−tn

.

Whenn≥2_H(p)^k (1 +ϵ), we have

nlim→∞H(tn

n) = lim

n→∞

k

n ≤ H(p) 2(1 +ϵ).

This implies that whennis large enough,H(^t_nⁿ)≤ ^H(p)₂ . Let us define a constantαsuch thatα≤¹₂ andH(α) = ^H(p)₂ . Then for alln≥2_H(p)^k (1 +ϵ), whenkis large enough,

Q_n+1

Qn ≤ 1−p 1−α<1.

Therefore, given any δ >0, whenk is large enough, the value of the third term

∑∞ n=2_H(p)^k (1+ϵ)

Qn ≤ Q₂ k H(p)(1+ϵ)

∑∞ i=0

(1−p 1−α)ⁱ

≤ Q k H(p)(1+ϵ)

1 1−¹₁⁻_−α^p

≤ 1−α

p−αδ. (2.5)

Substituting (2.3), (2.4), and (2.5) into (2.2) yields that for any ϵ > 0 and δ >0, if k is large enough, we have

E[n]≤ k

H(p)(1 +ϵ)(1 +δ) +1−α p−αδ, withα < p.

Then it is easy to get that

klim→∞

E[n]

k ≤ 1

H(p). This completes the proof.

Theorem 2.13. Given a biased coin with probability pbeing H, let nbe the number of coin tosses required to generatek random bits in the iterative scheme, then

klim→∞

E[n]

k = 1

H(p).

Proof. First, we prove that lim_k→∞^E[n]_k ≥_H(p)¹ . LetX∈ {0,1}^∗ be the input sequence, then

lim

k→∞

E[n]H(p) H(X) = 1.

Shannon’s theory tells us that it is impossible to extract more than H(X) random bits fromX, i.e.,H(X)≥k. So

lim

k→∞

E[n]

k ≥ 1

H(p).

To get the conclusion in the theorem, we only need to show that

lim

k→∞

E[n]

k ≤ 1

H(p).

To distinguish thenin this theorem and the one in the previous theorem, we usen_(k)denote the number of coin tosses required to generatekrandom bits in the iterative scheme and letn^Φ_(k)denote the number of coin tosses required by Φk. Letpmbe the probability for Φk generating mrandom bits with 0≤m≤k. Then we have that

E[n_(k)] =E[n^Φ_(k)] +

∑k m=0

p_mE[n_(k−m)]. (2.6)

According to the algorithm, pk ≥ ¹₂ and E[n_(k−m)] ≤ E[n_(k)]. Substituting them into the equation above gives

E[n_(k)]≤E[n^Φ_(k)] +1

2E[n_(k)], i.e.,E[n(k)]≤2E[n^Φ_(k)].

Now, we divide the second term in (2.6) into two parts such that

E[n_(k)]≤E[n^Φ_(k)] +

k∑−ϵk m=0

p_mE[n_(k−m)] +

∑k m=k−ϵk

p_mE[n_(k−m)],

for a constantϵ >0. In which,

k∑−ϵk m=0

pmE[n_(k−m)]≤(

k∑−ϵk m=0

pm)2E[n^Φ_(k)],

∑k m=k−ϵk

pmE[n_(k−m)]≤2E[n^Φ_(ϵk)].

Hence

E[n_(k)]≤E[n^Φ_(k)] + (

k∑−ϵk m=0

p_m)2E[n^Φ_(k)] + 2E[n^Φ_(ϵk)]. (2.7)

Given k, all the possible input sequences are divided into w prefix sets S₁, S₂, ..., S_w, where w can be an infinite number. Given an input sequenceX ∈S_i for 1≤i≤w, we are considering the probability for Φk generating a sequence of length m.

In our algorithm,|Si| ≥2^k. Assume

|Si|=αk2^k+αk−12^k−1+...+α02⁰,

whereα_k≥1 and 0≤α₀, α₁, ..., α_k−1≤1. Given the conditionX ∈S_i, we have

k∑−ϵk m=0

pm=

∑k−ϵk i=0 αi2ⁱ

∑k

i=0α_i2ⁱ ≤ 2^k−ϵk+1

2^k+ 2^k−ϵk+1 ≤2^k−ϵk+1 2^k .

So given anyδ >0, whenkis large enough, we have

k∑−ϵk m=0

pm≤δ. (2.8)

Although we reach this conclusion for X ∈ S_i, this conclusion holds for any S_i with 0 ≤ i ≤ w.

Hence, we are able to remove this constrain thatX ∈S_i.

According to the previous lemma, for anyδ >0, whenkis large enough, we have

E[n^Φ_(ϵk)]

ϵk ≤ 1

H(p)+δ, (2.9)

E[n^Φ_(k)]

k ≤ 1

H(p)+δ. (2.10)

Substituting (2.8), (2.9), and (2.10) into (2.7) gives us

E[n_(k)]≤k( 1

H(p)+δ)(1 + 2δ) + 2kϵ( 1 H(p)+δ).

From which, we obtain

lim

k→∞

E[n]

k = lim

k→∞

E[n_(k)]

k ≤ 1

H(p). This completes the proof.

The theorem above shows that the iterative scheme is asymptotically optimal, i.e., the expected number of coin tosses for generatingkrandom bits approaches the information theoretic bound by below whenk becomes large.