PDF Lecture-23: Exchangeability - Indian Institute of Science

Lecture-23: Exchangeability

1 Exchangeability

LetX_ibe a random variable on the probability space(Ω_i,Si,µ_i). Consider the probability space(Ω,F,P)for the processX= (X_i:i∈N)where

Ω=Ω1×Ω1×. . . , F=S1×S2×. . . , P=µ1×µ2×. . . .

Let p_i:Ω→Ω_i be a projection operator, such thatp_i(ω) =ω_i, then for eachi∈N, we haveµ_i=P◦p⁻¹_i . A finite permutationofNis a bijective mapπ:N→Nsuch thatπ(i)6=ifor only finitely manyi. That is, for a finite subsetF⊂N, we haveπ(F) =Fandπ(i) =ifori∈/F. It is clear that a finite permutationπ can always be defined on an interval of form[n], wheren=max{i:i∈F}.

Letω∈Ω,p_ibe projection operators, andπbe a finite permutation, then we can define a finitely permuted outcomeπ(ω)in terms of its projections, asp_i◦π(ω),p_π(i)◦ωfor eachi∈N. An eventA⊂Ωisn-permutable if for all permutationsπ over[n], we have

A=π⁻¹A={ω∈Ω:π(ω)∈A}.

An eventA⊂Ωispermutableif it isn-permutable for alln∈N. The collection ofn-permutable events is aσ-field calledn-exchangeableand is denoted byEn. The collection of permutable events is aσ-field called theexchange- ableσ-field and denoted byE. A sequenceX= (X_n:n∈N)of random variables is calledexchangeableif for each finite permutationπ on a finite set[n], the joint distribution of(X₁,X₂, . . . ,X_n)and(X_π(1),X_π(2), . . . ,X_π(n)) are identical.

Example 1.1 (One-dimensional random walk). Let Ωi =R, and Sn=∑ⁿ_i=1Xi. Then the events{S_n6 xinfinitely often}and{lim sup_n∈N^S_cⁿ

n >1}are permutable, becauseS_n(ω) =S_n(π(ω))for largen. Further, all events in tailσ-fieldTare permutable. This follows from the fact that if the eventA∈σ(X_n+1,Xn+2, . . .), then the eventAremains unaffected by the permutation of(X₁, . . . ,Xn).

Example 1.2 (Draw without replacement). Suppose balls are selected randomly, without replacement, from an urn consisting ofnballs of whichkare white. For drawi∈[n], letξibe the indicator of the event that theith selection is white. Then the finite collection(ξ₁, . . . ,ξn)is exchangeable but not independent. In particular, letA={i∈[n]:ξi=1}. Then, we know that|A|=k, and we can write

P{ξ_i=1,i∈A,ξ_j=0,j∈/A}=P{A= (i₁,i2, . . . ,i_k)}=(n−k)!k!

n! = 1

n k

.

This joint distribution is independent of set of exact locationsA, and hence exchangeable. However, one can see the dependence from

P(ξ₂=1|ξ₁=1) =k−1 n−1 6= k

n−1=P(ξ₂=1|ξ₁=0).

Example 1.3 (Conditionally independent sequence). LetΛdenote a random variable having distributionG.

LetX be a sequence of dependent random variables, where each of these random variables are conditionally iidwith distributionF_λgivenΛ=λ. We can write the joint finite dimensional distribution of the sequenceX,

P{X₁6x1. . . ,X_n6x_n}= Z

Λ n

∏

i=1

F_λ(x_i)dG(λ).

Since any finite dimensional distribution of the sequenceX is symmetric in(x₁, . . .x_n), it follows thatX is exchangeable.

1

Theorem 1.4 (De Finetti’s Theorem). If X is an exchangeable sequence of random variables then conditioned onE, the sequence X is iid.

Proof. To show the independence of exchangeable sequenceXof random variables, conditioned on exchangeable σ-fieldE, we need to show that for bounded functions f_i:R→R

E[

k

∏

i=1

f_i(X_i)|E] =

k

∏

i=1

E[f_i(X_i)|E].

Using induction, it suffices to show that any two bounded functions f :R^k−1→Randg:R→Rare independent conditioned on the exchangeableσ-field. That is,

E[f(X₁, . . . ,Xk−1)g(X_k)|E] =E[f(X₁, . . . ,Xk−1)|E]E[g(X_k)|E].

LetIn,k,{i⊆[n]^k:ijdistinct}, then the cardinality of this set is denoted by(n)_k,|I_n,k|=n(n−1). . .(n−k+1).

For a bounded functionφ:R^k→R, we can define

An(φ), 1

|I_n,k|

∑

i∈I_n,k

φ(X_i1,Xi₂, . . . ,Xi_k).

It is clear thatA_n(φ)∈Enmeasurable and henceE[A_n(φ)|En] =A_n(φ). For eachi∈I_n,k, we can find a finite permutation on[n], such thatπ(i_j) =jfor j∈[k]. SinceXis exchangeable, the distribution of(X_i1, . . . ,X_ik)and

(X₁, . . . ,X_k)are identical for eachi∈I_n,k. Therefore, we have

A_n(φ) = 1

|I_n,k|

∑

i∈I_n,k

E[φ(X_i1,X_i2, . . . ,X_ik)|En] =E[φ(X₁,X₂, . . . ,X_k)|En].

SinceEn→E, using bounded convergence theorem for conditional expectations, we have

n∈limN

A_n(φ) =lim

n∈NE[φ(X₁,X₂, . . . ,X_k)|En] =E[φ(X₁,X₂, . . . ,X_k)|E].

Let f andgbe bounded functions onR^k−1andRrespectively, such thatφ(x₁, . . . ,x_k) = f(x₁, . . . ,xk−1)g(x_k). We also defineφ_j(x₁, . . . ,xk−1) =f(x₁, . . . ,xk−1)g(x_j), to write

(n)_k−1A_n(f)nA_n(g) =

∑

i∈I_n,k−1

f(X_i1, . . . ,X_ik−1)

∑

m

g(X_m)

=

∑

i∈I_n,k

f(X_i1, . . . ,Xik−1)g(X_ik) +

∑

i∈I_n,k−1 k−1

∑

j=1

f(X_i1, . . . ,Xik−1)g(X_ij)

= (n)_kA_n(φ) +

k−1

∑

j=1

(n)_k−1A_n(φ_j).

Dividing both sides by(n)_kand rearranging terms, we get

A_n(φ) = n

n−k+1A_n(f)A_n(g)− 1 n−k+1

k−1

∑

j=1

A_n(φ_j),

Taking limits on both sides, we obtain the result

E[f(X₁, . . . ,Xk−1)g(X_k)|E] =E[f(X₁, . . . ,Xk−1)|E]E[g(X_k)|E].

Corollary 1.5 (De Finetti 1931). A binary sequence X= (X_n:n∈N)is exchangeable iff there exists a distribution function F(p)on[0,1]such that for any n∈N, and s_n=∑ix_i

P{X₁=x₁, . . . ,X_n=x_n}= Z 1

0

p^sn(1−p)^n−sndF(p).

Proof. The distribution of ann-exchangeable binary sequence is given by∑ⁿ_i=0p_i1_{Sn=i}. LetY_n=^S_nⁿ andY = lim_n∈N^S_nⁿ. Hence, the collection of n-permutable events is En=σ(Y_n) for binary sequences. Therefore, the exchangeableσ-fieldE=σ(Y), andFis the distribution function for the random variableY.

2

Example 1.6 (Polya’s Urn Scheme). We now discuss a non-trivial example of exchangeable random vari- ables. Consider a discrete time stochastic process{(B_n,W_n):n∈N}, whereBn,W_nrespectively denote the number of black and white balls in an urn aftern∈Ndraws. At each drawn, balls are uniformly sampled from this urn. After each draw, one additional ball of the same color to the drawn ball, is returned to the urn.

We are interested in characterizing evolution of this urn, given initial urn content(B₀,W₀).

Letξibe a random variable indicating the outcome of theith draw being a black ball. For example, if the first drawn ball is a black, thenξ1=1 and(B₁,W₁) = (B₀+1,W₀). In general,

B_n=B0+

n i=1

∑

ξ_i=Bn−1+ξ_n, W_n=W0+

n i=1

∑

(1−ξ_i) =Wn−1+1−ξ_n.

It is clear that B_n+W_n=B₀+W₀+n. Letξ be any sequence of indicatorsξ ∈ {0,1}^N. We can find the indices of black balls being drawn in firstndraws, as

I_n(ξ) ={i∈[n]:ξ_i=1}.

With this, we can write the probability ofx∈ {0,1}ⁿ

Pr{ξ₁=x₁, . . . ,ξn=x_n}=∏^|I_i=1^n(x)|(B₀+i−1)∏^n−|I_j=1^n(x)|(W₀+i−1)

∏ⁿ_i=1(B₀+W₀+i−1)

Since this probability depends only on|I_n(x)|and notx, it shows that any finite number of draws is finitely permutable event. That is,(ξ₁, . . . ,ξn)∈Enfor eachn∈N. Hence, any sequence of drawsξ ={ξ_i:i∈N} for Polya’s Urn scheme is exchangeable.

3