Lecture-03: Independence

(1)

Lecture-03: Independence

1 Independence

Definition 1.1 (Independence of events). For a probability space(Ω,F,P), a family of events(Ai∈F:i∈I) is said to be independent, if for any finite setF⊆I, we have

P(∩i∈FA_i) =

∏

i∈F

P(A_i)_.

Remark1. The certain eventΩand the impossible event∅are always independent to every eventA∈_F.

Example 1.2 (Two coin tosses). Consider two coin tosses, such that the sample space is Ω = {HH,HT,TH,TT}, and the event space isF=2^Ω. It suffices to define a probability functionP:F→ [0, 1]on the sample space. We define one such probability functionP, such that

P({HH}) =P({HT}) =P({TH}) =P({TT}) = ¹ 4.

Let event A₁,{HH,HT}andB₂,{HH,TH}correspond to getting a head on the first or the second toss respectively.

From the defined probability function, we obtain the probability of getting a tail on the first or the second toss is ¹₂, and identical to the probability of getting a head on the first or the second toss. That is, P(A₁) =P(A₂) =¹₂ and the intersecting eventA₁∩A₂={HH}with the probabilityP(A₁∩A₂) =¹₄. That is, for eventsA₁,A₂∈_{F, we have}

P(A₁∩A2) =P(A₁)P(A2). That is, eventsA1andA2are independent.

Example 1.3 (Countably infinite coin tosses). Consider a sequence of coin tosses, such that the sample space isΩ={H,T}^N. For set of outcomesEn,{ω∈Ω:ωn=H}, we consider an event space gener- ated byF,^σ({En:n∈N}). We define a probability functionP:F→[0, 1]byP(∩i∈FEi) =p^|F|for any finite subsetF⊆N. By definition,(En:n∈N)is a sequence of independent events.

We observe that the set of outcomes corresponding to at least one head in firstnoutcomes An,{ω∈_Ω:ω_i=Hfor somei∈[n]}=∪ⁿ_i=1E_i∈_F,

and set of outcomes corresponding to first head at thenth outcome

Bn,{ω∈_Ω:ω₁=· · ·=ωn−1=T,ω=H}=∩ⁿ⁻¹_i=1E^c_i ∩En∈_F.

In particular this implies thatσ({An:n∈_N})⊆_Fandσ({Bn:n∈_N})⊆F. We can show thatP(An) = 1−(1−p)ⁿandP(Bn) =p(1−p)ⁿ⁻¹forn∈_N.

1

(2)

LetFn be the event space generate by the firstn coin tosses, i.e. Fn,^σ({Ei:i∈[n]}). Then, we can show that F=σ({Fn:n∈N}). For any ω∈ Ω, we can define the number of heads in first n trials bykn,∑i=1ⁿ 1{ωi=H}. Then, we observe that any eventA∈Fncan be written as union of∩ⁿ_i=1Ci

whereCi=EiorE^c_i. That is, we can specify the firstnoutcomes for eachω∈A. Since the probability P(∩_i=1ⁿ Ci) =_∏ⁿ_i=1P(Ci), we have

P(A) =

∑

ω∈A

p^kⁿ^(ω)(₁−p)^n−kⁿ^(ω)_.

Example 1.4 (Counter example). Consider a probability space(_Ω,F,P)and the eventsA₁,A₂,A₃∈_F.

The conditionP(A₁∩A₂∩A₃) =P(A₁)P(A₂)P(A₃)is not sufficient to guarantee independence of the three events. In particular, we see that if

P(A₁∩A2∩A3) =P(A₁)P(A2)P(A3), P(A₁∩A2∩A^c₃)6=P(A₁)P(A2)P(A^c₃), thenP(A1∩A2) =P(A1∩A2∩A3) +P(A1∩A2∩A^c₃)6=P(A1)P(A2).

Definition 1.5. A family of collections of events(Ai⊆F:i∈ I)is called independent, if for any finite set F⊆IandAi∈Aifor alli∈F, we have

P(∩i∈FAi) =

∏

i∈F

P(Ai).

2 Law of Total Probability

Theorem 2.1 (Law of total probability). For a probability space(_Ω,_F,P), consider a sequence of events B= (Bn∈_F:n∈_N)that partitions the sample spaceΩ, i.e. Bm∩Bn=_∅for all m6=n, and∪_n∈_NBn=Ω. Then, for any event A∈_{F, we have}

P(A) =

∑

n∈N

P(A∩Bn).

Proof. We can expand any eventA∈Fin terms of any partitionBof the sample spaceΩas A=A∩Ω=A∩(∪_n∈_NBn) =∪_n∈_N(A∩Bn).

From the mutual disjointness of the events(Bn∈_F:n∈_N), it follows that the sequence(A∩Bn∈_F:n∈_N) is mutually disjoint. The result follows from the countable additivity of probability of disjoint events.

3 Conditional Probability

Consider Ntrials of a random experiment over an outcome space Ωand an event spaceF. Letωn ∈_Ω denote the outcome of the experiment of thenth trial. Consider two eventsA,B∈_Fand denote the number of times eventAand eventBoccurs byN(A)andN(B)respectively. We denote the number of times both eventsAandBoccurred byN(A∩B). Then, we can write these numbers in terms of indicator functions as

N(A) =

∑

N n=1

1{ω_n∈A}, N(B) =

∑

N n=1

1{ω_n∈B}, N(A∩B) =

∑

N n=1

1{ω_n∈A∩B}.

We denote the relative frequency of eventsA,B,A∩BinNtrials by ^N(A)_N ,^N(B)_N ,^N(A∩B)_N respectively. We can find the relative frequency of eventsA, on the trials whereBoccurred as

N(A∩B) N N(B)

N

= ^N(A∩B) N(B) ^.

2

(3)

Inspired by the relative frequency, we define the conditional probability function conditioned on events.

Definition 3.1. Fix an event B∈Fsuch that P(B)>0, we can define the conditional probability P(·|B): F→[0, 1]of any eventA∈Fconditioned on the eventBas

P(_A|B) =^P(A∩B) P(B) ^.

Lemma 3.2 (Conditional probability). For any event B∈_F such that P(B)>0, the conditional probability P(·|B):F→[0, 1]is a probability measure on space(_Ω,_F).

Proof. We will show that the conditional probability satisfies all four axioms of a probability measure.

Non-negativity: For all eventsA∈_{F, we have}P(A|B)>0 sinceP(A∩B)>0.

σ-additivity: For an infinite sequence of mutually disjoint events (Ai∈F:i∈N)such that Ai∩Aj=∅ for alli6=j, we have P(∪i∈NA_i|B) =_∑_i∈_NP(A_i|B). This follows from disjointness of the sequence (A_i∩B∈F:i∈N)_.

Certainty: SinceΩ∩B=B, we haveP(Ω|B) =1.

Remark 2. For two independent events A,B∈F such that P(A∩B)>0, we have P(A|B) =P(A) and P(B|A) =P(B). If eitherP(A) =0 orP(B) =0, thenP(A∩B) =0.

Remark3. For any partitionBof the sample spaceΩ, ifP(Bn)>0 for alln∈N, then from the law of total probability and the definition of conditional probability, we have

P(A) =

∑

n∈N

P(A|Bn)P(Bn).

4 Conditional Independence

Definition 4.1 (Conditional independence of events). For a probability space(_Ω,F,P), a family of events (A_i∈F:i∈I)is said to be conditionally independent given an eventC∈Fsuch thatP(C)>0, if for any finite setF⊆I, we have

P(∩_i∈FA_i|C) =

∏

i∈F

P(A_i|C).

Remark 4. Let C∈_F be an event such that P(C)>0 Two events A,B∈ _F are said to be conditionally independent given eventC, if

P(A∩B|C) =P(A|C)P(B|C). If the eventC=Ω, it implies thatA,Bare independent events.

Remark5. Two events may be independent, but not conditionally independent and vice versa.

Example 4.2. Consider two independent events A,B∈Fsuch that P(A∩B)>0 andP(A∪B)<1.

Then the eventsAandBare not conditionally independent givenA∪B. To see this, we observe that

P(A∩B|A∪B) =^P((A∩B)∩(A∪B))

P(A∪B) = ^P(A∩B)

P(A∪B)= ^P(A)P(B)

Example 4.3. Consider two non-independent events A,B∈_Fsuch thatP(A)>0. Then the events A andBare conditionally independent givenA. To see this, we observe that

P(A∩B|A) = ^P(A∩B)

P(A) =P(B|A)P(A|A).

3