E2 201 (Aug–Dec 2014)
Homework Assignment 2
Discussion: (to be decided) Quiz: Friday, Sept. 5
This assignment consists of two pages.
1. LetXandY be discrete random variables taking values inX andY respectively.
(a) Show thatH(Y|X) = 0if and only ifY is a function ofX, i.e., for eachx ∈ X such thatp(x) > 0, there exists a uniquey∈ Y such thatp(x, y)>0.
(b) For any functionfwith domainX, show thatH(f(X))≤H(X).
[Hint: There are many ways of showing this. One way is to consider the two different ways of expanding H X, f(X)
using the chain rule.]
2. Problem 2.8, Cover & Thomas, 2nd ed.
3. Problem 2.10, Cover & Thomas, 2nd ed.
4. LetXbe a random variable taking values in{1,2,3, . . .}, with meanE[X] =µ. LetZbe a geometric random variable also with meanµ, i.e.,E[Z] =E[X].
(a) Show thatH(X)≤H(Z), with equality iffXhas the same distribution asZ.
[Hint: Letp= (pn)be the pmf ofX, and letγ= (γn)be the pmf ofZ. ConsiderD(pkγ).]
Thus, the maximum entropy among all positive integer-valued random variables with a fixed meanµis attained by a geometric random variable.
(b) Prove thatH(X)≤E[X]. When does equality hold?
[Hint: EvaluateH(Z).]
5. Problem 2.25, Cover & Thomas, 2nd ed.
6. Find an example of pmfsp(x),q(x)andr(x)defined on the same alphabetX, such thatD(pk q),D(q kr) andD(pkr)are all finite and satisfy
D(pkq) +D(qkr)< D(pkr).
7. A non-negative matrixW = wi,j
is calleddoubly stochasticif all its row sums and all its column sums are equal to1:P
jwi,j = 1for alli, andP
jwi,j = 1for allj.
Letp= (p1, . . . , pm)be a probability mass function, and letW be anm×mdoubly stochastic matrix.
(a) Show thatpˆ =pW is also a probability mass function.
(b) Letq= (q1, . . . , qm)be another pmf. Show that
D(pW kqW)≤D(pkq)
(c) From (b), deduce thatH(pW) ≥ H(p). Here, of course,H(p) = −P
ipilogpi, andH(pW)is the entropy corresponding topW.
[Remark: the result of part (b) is sometimes called the data processing inequality for relative entropy. In fact, your proof of (a) and (b) should only use the fact that the row sums ofW are all equal to1. Doubly stochastic will be needed for part (c).]
8. LetX1, . . . , Xn, Y1, . . . , Ynbe discrete random variables.
(a) Show that ifX1, . . . , Xnare independent rvs, then
I(X1, . . . , Xn;Y1, . . . , Yn)≥
n
X
i=1
I(Xi;Yi).
Give a necessary-and-sufficient condition for equality to hold above.
(b) Show that ifp(y1, . . . , yn|x1, . . . , xn) =Qn
i=1p(yi|xi), then
I(X1, . . . , Xn;Y1, . . . , Yn)≤
n
X
i=1
I(Xi;Yi).
Give a necessary-and-sufficient condition for equality to hold above.