Lecture-03: Independence

Lecture-03: Independence

1 Law of Total Probability

Exercise 1.1 (Countably infinite coin tosses). Consider a sequence of coin tosses, such that the sample space is Ω={H,T}^N. For set of outcomes En≜{ω∈Ω:ωn=H}, we consider an event space generated by F≜σ({En:n∈_N}). LetFn be the event space generate by the first n coin tosses, i.e. Fn≜σ({E_i:i∈[n]}). LetAn be the set of outcomes corresponding to at least one head in firstnoutcomesAn≜{ω∈_Ω:ω_i=Hfor somei∈[n]}=∪ⁿ_i=1E_i ∈_{F, and}Bn be the set of out- comes corresponding to first head at thenth outcomeBn≜{ω∈_Ω:ω₁=· · ·=ωn−1=T,ω=H}=

∩ⁿ⁻¹_i=1E_i^c∩En∈_F.

1. Show thatF=σ({_Fn:n∈_N}).

2. Show thatσ({An:n∈_N})⊆_Fandσ({Bn:n∈_N})⊆_F.

Theorem 1.2 (Law of total probability). For a probability space(Ω,F,P), consider a sequence of events B∈F^N that partitions the sample spaceΩ, i.e. Bm∩Bn=_∅for all m̸=n, and∪n∈NBn=Ω. Then, for any event A∈F, we have

P(A) =

∑

n∈N

P(A∩Bn).

Proof. We can expand any eventA∈Fin terms of any partitionBof the sample spaceΩas A=A∩Ω=A∩(∪n∈NBn) =∪n∈N(A∩Bn).

From the mutual disjointness of the eventsB∈F^N, it follows that the sequence(A∩Bn ∈F:n∈N)_is mutually disjoint. The result follows from the countable additivity of probability of disjoint events.

Example 1.3 (Countably infinite coin tosses). Consider the sample spaceΩ={H,T}^Nand event spaceF generated by sequenceE∈F^Ndefined in Exercise 1.1. We observe that any eventA∈Fncan be written as

A=∪_ω∈A{ω}=∪_ω∈A∩ⁿ_i=1({ω∈Ei} ∪ {ω∈/Ei}).

2 Independence

Definition 2.1 (Independence of events). For a probability space(_Ω,F,P), a family of eventsA∈_F^Iis said to be independent, if for any finite setF⊆I, we have

P(∩_i∈FA_i) =

∏

i∈F

P(A_i).

Remark1. The certain eventΩand the impossible event∅are always independent to every eventA∈F.

1

Example 2.2 (Two coin tosses). Consider two coin tosses, such that the sample space is Ω = {HH,HT,TH,TT}, and the event space isF=P(Ω). It suffices to define a probability functionP:F→[_{0, 1}] on the sample space. We define one such probability functionP, such that

P({HH}) =P({HT}) =P({TH}) =P({TT}) = ¹ 4.

Let eventE1≜{HH,HT}andE2≜{HH,TH}correspond to getting a head on the first or the second toss respectively.

From the defined probability function, we obtain the probability of getting a tail on the first or the second toss is ¹₂, and identical to the probability of getting a head on the first or the second toss. That is, P(E1) =P(E2) =¹₂and the intersecting eventE1∩E2={HH}with the probabilityP(E1∩E2) =¹₄. That is, for eventsE₁,E₂∈F, we have

P(E₁∩E₂) =P(E₁)P(E₂). That is, eventsE₁andE₂are independent.

Example 2.3 (Countably infinite coin tosses). Consider the outcome spaceΩ={H,T}^Nand event space Fgenerated by the sequenceEdefined in Exercise 1.1. We define a probability functionP:F→[0, 1]by P(∩_i∈FE_i) =p^|F|for any finite subset F⊆N. By definition, E∈_F^N is a sequence of independent events.

Consider A,B∈_F^N, whereAn≜∪_i=1ⁿ Ei andBn≜∩ⁿ⁻¹_i=1E^c_i ∩En∈_Ffor alln∈N. It follows thatP(An) = 1−(1−p)ⁿandP(Bn) =p(1−p)ⁿ⁻¹forn∈_N.

For any ω ∈ Ω, we can define the number of heads in first n trials by kn(ω)≜∑ⁿi=11{H}(ω_i) =

∑ⁿi=11{ω∈E_i}. For any general eventA∈_Fn=σ({E_i:i∈[n]}), we can write P(A) =

∑

ω∈A

∏

n i=1

hP{ω∈E_i}+P{ω∈E_i^c}ⁱ=

∑

ω∈A

p^kn(ω)(1−p)^n−kn(ω).

Example 2.4 (Counter example). Consider a probability space(Ω,F,P)and the eventsA1,A2,A3∈F. The condition P(A1∩A2∩A3) =P(A1)P(A2)P(A3)is not sufficient to guarantee independence of the three events. In particular, we see that if

P(A₁∩A₂∩A₃) =P(A₁)P(A₂)P(A₃)_, P(A₁∩A₂∩A^c₃)̸=P(A₁)P(A₂)P(A^c₃)_, thenP(A₁∩A₂) =P(A₁∩A₂∩A₃) +P(A₁∩A₂∩A^c₃)̸=P(A₁)P(A₂).

Definition 2.5. A family of collections of events(Ai⊆F:i∈ I)is called independent, if for any finite set F⊆IandAi∈Aifor alli∈F, we have

P(∩i∈FA_i) =

∏

i∈F

P(A_i)_.

3 Conditional Probability

Consider Ntrials of a random experiment over an outcome space Ωand an event spaceF. Letωn ∈_Ω denote the outcome of the experiment of thenth trial. Consider two eventsA,B∈_Fand denote the number

2

of times eventAand eventBoccurs byN(A)andN(B)respectively. We denote the number of times both eventsAandBoccurred byN(A∩B). Then, we can write these numbers in terms of indicator functions as

N(A) =

∑

N n=1

1{ω_n∈A}, N(B) =

∑

N n=1

1{ω_n∈B}, N(A∩B) =

∑

N n=1

1{ω_n∈A∩B}.

We denote the relative frequency of eventsA,B,A∩BinNtrials by ^N(A)_N ,^N(B)_N ,^N(A∩B)_N respectively. We can find the relative frequency of eventsA, on the trials whereBoccurred as

N(A∩B) N N(B)

N

= ^N(A∩B) N(B) ^.

Inspired by the relative frequency, we define the conditional probability function conditioned on events.

Definition 3.1. Fix an event B∈_Fsuch that P(B)>0, we can define the conditional probability P(·|B): F→[0, 1]of any eventA∈_Fconditioned on the eventBas

P(A|B) =^P(A∩B) P(B) ^.

Lemma 3.2 (Conditional probability). For any event B∈F such that P(B)>0, the conditional probability P(·|B):F→[0, 1]is a probability measure on space(Ω,F).

Proof. We will show that the conditional probability satisfies all three axioms of a probability measure.

Non-negativity: For all eventsA∈_{F, we have}P(A|B)⩾0 sinceP(A∩B)⩾0.

σ-additivity: For an infinite sequence of mutually disjoint events (A_i∈_F:i∈_N)such that A_i∩A_j=_∅ for alli̸=j, we have P(∪_i∈NA_i|B) =_∑i∈NP(A_i|B). This follows from disjointness of the sequence (Ai∩B∈_F:i∈_N).

Certainty: SinceΩ∩B=B, we haveP(Ω|B) =1.

Remark 2. For two independent events A,B∈F such that P(A∩B)>0, we have P(A|B) =P(A) and P(B|A) =P(B). If eitherP(A) =_{0 or}P(B) =_{0, then}P(A∩B) =_0.

Remark3. For any partitionBof the sample spaceΩ, ifP(Bn)>0 for alln∈N, then from the law of total probability and the definition of conditional probability, we have

P(A) =

∑

n∈N

P(A|Bn)P(Bn).

4 Conditional Independence

Definition 4.1 (Conditional independence of events). For a probability space(Ω,F,P), a family of events A∈F^Iis said to be conditionally independent given an eventC∈Fsuch thatP(C)>0, if for any finite set F⊆I, we have

P(∩i∈FA_i|C) =

∏

i∈F

P(A_i|C)_.

Remark 4. Let C∈Fbe an event such that P(C)>0. Two events A,B∈Fare said to be conditionally independent given eventC, if

P(A∩B|C) =P(A|C)P(B|C). If the eventC=Ω, it implies thatA,Bare independent events.

Remark5. Two events may be independent, but not conditionally independent and vice versa.

3

Example 4.2. Consider two independent events A,B∈_Fsuch thatP(A∩B)>0 andP(A∪B)<1. Then the eventsAandBare not conditionally independent givenA∪B. To see this, we observe that

P(A∩B|A∪B) =^P((A∩B)∩(A∪B))

P(A∪B) = ^P(A∩B)

P(A∪B)= ^P(A)P(B)

P(A∪B) =P(A|A∪B)P(B).

We further observe thatP(B|A∪B) =_P(A∪B)^P(B) ̸=P(B)and henceP(A∩B|A∪B)̸=P(A|A∪B)P(B|A∪B). Example 4.3. Consider two non-independent eventsA,B∈Fsuch thatP(A)>0. Then the eventsAandB are conditionally independent givenA. To see this, we observe that

P(A∩B|A) = ^P(A∩B)

P(A) =P(B|A)P(A|A).

4