Lecture-02: Probability Function

Lecture-02: Probability Function

1 Indicator Functions

Definition 1.1 (Indicator functions). Consider a random experiment over an outcome space Ω and an event spaceF. The indicator of an eventB∈Fis denoted by

1B(ω) =

(1, ω∈B, 0, ω∈/B.

Example 1.2. Consider a roll of dice that has an outcome spaceΩ= [6]and event spaceF=P(Ω). For an eventO≜{1, 3, 5}that represents odd outcomes for dice roll, we observe that1O(1) =1 and1O(2) =0.

Example 1.3. ConsiderNtrials of a random experiment over outcome spaceΩand the event spaceF. Let ωn∈_Ωdenote the outcome of the experiment of thenth trial. For each eventB∈F, we define an indicator function

1B(ωn) =

(1, ωn∈B, 0, ωn∈/B.

For any eventB∈F, the number of times the eventBoccurs inNtrials is denoted byN(B) =_∑n=1^N _1B(ωn). We denote the relative frequency of an eventAinNtrials by^N(B)_N .

Definition 1.4 (Disjoint events). Let(_Ω,F)be a pair of sample and event space, and a sequence of events A∈_F^N. The sequence ismutually disjointifAn∩Am=_∅for anym̸=n∈_{N. If}∪_n∈NAn=_{Ω, then}Ais apartitionof sample spaceΩ.

Exercise 1.5. Consider the sample spaceΩ={H,T}^N and event spaceF=σ(_An:n∈N) _where An≜{ω∈Ω:ω_i=Hfor somei∈[n]}. Construct a sequence of non-trivial disjoint events.

Exercise 1.6. Let(_Ω,F)be a pair of sample and event space. For any sequence of events A∈_F^N, show that

1. 1∩_n∈NAn(ω) =_∏n∈N1An(ω),

2. 1∪_n∈NAn(ω) =_∑n∈N1An(ω)if the sequenceAis mutually disjoint.

Example 1.7. We observe the following properties of the relative frequency.

1. For all eventsB∈F, we have 0⩽^N(B)_N ⩽1. This follows from the fact that 0⩽N(B)⩽Nfor any event B∈_F.

2. LetA∈F^Nbe a sequence of mutually disjoint events, then^N(∪i_N^∈NAi)=_∑i∈N^N(A_Nⁱ⁾. This follows from the fact that for mutually disjoint event sequenceA, we have

1∪_i∈NA_i(ωn) =

∑

i∈N1A_i(ωn).

1

3. For the certain eventΩ, we have^N(_N^Ω)=1. This follows from the fact thatN(_Ω) =N.

Since the relative frequency is positive and bounded, it may converge to a real number asN grows very large, and the limit limN→∞N(B)

N may exist.

2 Probability axioms

Inspired by the relative frequency, we list the following axioms for a probability functionP:F→[0, 1]. Axiom 2.1 (Axioms of probability). We define a probability measure on sample spaceΩand event space Fby a functionP:F→[0, 1]which satisfies the following axioms.

Non-negativity: For all eventsB∈F, we haveP(B)⩾0.

σ-additivity: For an infinite sequenceA∈_F^Nof mutually disjoint events, we haveP(∪_i∈NAi) =_∑i∈NP(Ai). Certainty: P(_Ω) =_1.

Definition 2.2 (Probability space). A sample spaceΩ, an event spaceF⊆ P(Ω), and a probability measure P:F→[0, 1], together define a probability space(Ω,F,P).

3 Properties of Probability

Theorem 3.1. For any probability space(Ω,F,P), we have the following properties of probability measure.

impossibility: P(_∅) =0.

finite additivity: For mutually disjoint events A∈Fⁿ, we have P(∪ⁿ_i=1Ai) =_∑ⁿ_i=1P(Ai). monotonicity: If events A,B∈_Fsuch that A⊆B, then P(A)⩽P(B).

inclusion-exclusion: For any events A,B∈F, we have P(A∪B) =P(A) +P(B)−P(A∩B).

continuity: For a sequence of events A∈_F^N)such thatlimnAnexists, we have P(limn→∞An) =limn→∞P(An). Proof. We consider the probability space(_Ω,F,P).

1. We take disjoint eventsE∈F^NwhereE₁=_ΩandE_i=_∅fori⩾2. It follows that∪i∈NE_i=_ΩandEis a collection of mutually disjoint events. From the countable additivity axiom of probability, it follows that

P(_Ω) =P(_Ω) +

∑

i⩾2

P(E_i). SinceP(E_i)⩾0, it implies thatP(_∅) =0.

2. We see that finite additivity follows from the countable additivity. We consider disjoint eventsA1, . . . ,An, and take Ai=∅for alli>n. It follows that the sequence of sets A∈F^N is mutually disjoint, and sinceP(∅) =0, it follows that

P(∪_i=1ⁿ A_i) =P(∪_i∈NA_i) =

∑

n i=1

P(A_i) +

∑

i>n

P(_∅) =

∑

n i=1

P(A_i).

3. For eventsA,B∈_Fsuch thatA⊆B, we can take disjoint eventsE₁=AandE₂=B\A. From closure under complements and intersection, it follows that E₂∈F. From non-negativity of probability, we haveP(E₂)⩾0. Finally, the result follows from finite additivity of disjoint events

P(B) =P(E₁∪E2) =P(E₁) +P(E2)⩾P(A). 2

4. For any two eventsA,B∈F, we can write the following events as disjoint unions

A= (A\B)∪(A∩B), B= (B\A)∪(A∩B), A∪B= (A\B)∪(A∩B)∪(B\A). The result follows from the finite additivity of probability of disjoint events.

5. To show the continuity of probability in events, we first need to understand the limits of events. We show the continuity of probability in the next section.

Example 3.2. Consider a single coin tosse with the sample spaceΩ={H,T}, an event spaceF=P(Ω). The probability measureP:F→[0, 1]is defined as

P(_∅) =_0, P({H}) =p P({T}) =1−p P({H,T}) =1.

Can you verify thatPis a probability function?

Example 3.3. ConsiderNcoin tosses with the sample spaceΩ={H,T}^[N], an event spaceF=P(_Ω). For each outcomeω∈Ω, we define the number of heads asNH(ω)≜∑n∈[N]1{H}(ωn)and the number of tails asN_T(ω)≜N−N_H(ω). The probability measureP:F→[_{0, 1}]is defined for each eventA∈Fas

P(A)≜

∑

ω∈A

p^NH(ω)(1−p)^NT(ω). Can you verify thatPis a probability function?

4 Limits of Sets

Definition 4.1 (Limits of monotonic sets). For a sequence of non-decreasing sets (An :n∈N), we can define the limit as

n→lim∞An≜∪n∈NAn.

Similarly, for a sequence of non-increasing sets(An:n∈_N), we can define the limit as

n→lim∞An≜∩_n∈NAn.

Example 4.2 (Monotone sets). Consider a monotonically increasing sequencea∈_R^Ndefined asan≜−¹_n for alln∈N, which converges to the limit 0. We consider monotone sequence of setsA,B∈_B(_R)^N de- fine as An≜[−2,−_n¹]andBn≜[−2,¹_n]for alln∈N, which are monotonically increasing and decreasing respectively. We can verify the following limits

limn An=∪_n∈NAn= [−2, 0), lim

n Bn=∩_n∈NBn= [−2, 0].

Definition 4.3 (Limits of sets). For a sequence of sets(An:n∈N), we can define the limit superior and limit inferior of this sequence of sets as

lim sup

n→∞ An≜∩n∈N∪_m⩾n Am= lim

n→∞∪_m⩾nAm, lim inf

n→∞ An≜∪n∈N∩_m⩾nAm= lim

n→∞∩_m⩾nAm. Lemma 4.4. For a sequence of sets(An:n∈N), we havelim infn→∞An⊆lim sup_n→∞An.

3

Proof. For eachn∈N, we defineEn≜∪_m⩾nAmandFn≜∩_m⩾nAm. Consider a fixedn, thenF₁,F2, . . . ,Fn⊆ An, andFm⊆Amfor allm⩾n. Therefore, we can write∪_n∈NFn⊆ ∪_m⩾nAmfor eachn∈N, and hence the result follows.

Definition 4.5. If the limit superior and limit inferior of any sequence of sets(An:n∈N)are equal, then the sequence of sets has a limitA_∞, which is defined as

A_∞≜ lim

n→∞An=lim sup

n→∞ An=lim inf

n→∞ An.

Example 4.6 (Sequence of sets with different limits). We consider sequence of sets(An= [−2,(−1)ⁿ+¹_n]: n∈_N). It follows thatFn=∩_m⩾nAm= [−2,−1]and

En=∪_m⩾nAm=

([−2, 1+ _n+1¹ ], nodd, [−2, 1+ ¹_n], neven.

We can verify the following limits lim inf

n An=∪n∈NFn= [−2,−1]_{, lim sup}

n

An=∩n∈NEn= [−2, 1]_.

4.1 Proof of continuity of probability

Continuity for increasing sets.Let(_An∈F:n∈N)be a non-decreasing sequence of events, then limn→∞An=

∪n∈NAn. This implies that (P(An)_:n∈N) is a non-negative non-decreasing bounded sequence, and hence has a limit. It remains to show that limn→∞P(An) =P(∪_n∈NAn). To this end, we observe that (A₁,A₂\A₁, . . . ,An\An−1)is a partition of the event An, and P(A_i\Ai−1) =P(A_i)−P(Ai−1)for each i∈N. From finite additivity ofPfor mutually disjoint events, we can write for eachn∈_N

P(An) =P(A1) +

n−1

∑

i=1

P(A_i+1\A_i) =P(A1) +

n−1

∑

i=1

(P(A_i+1)−P(A_i)).

Fromσ-additivity ofPfor sequence of mutually disjoint events, we can write for limn→∞An=∪n∈NAn, P(∪n∈NAn) =P(A₁) +

∑

i∈N

(P(A_i+1)−P(A_i)) =P(A₁) + _lim

n→∞ n−1

∑

i=1

(P(A_i+1)−P(A_i)) = _lim

n→∞P(An)_. Continuity for decreasing sets.Similarly, for a non-increasing sequence of sets(Bn∈_F:n∈_N), we can find the non-decreasing sequence of sets(B^c_n∈_F:n∈_N). By the first part, we have

P(lim

n→∞Bn) =P(∩n∈NBn) =1−P(∪n∈NB^c_n) =1−P(lim

n→∞B^c_n) =1− lim

n→∞P(B_n^c) = lim

n→∞P(Bn). Continuity for general sequence of sets. We can similarly prove the general result for a sequence of sets(An∈_F:n∈_N)such that the limits limnAn exists. We can define non-increasing sequences of sets (En=∪_m⩾nAm∈_F:n∈_N)and non-decreasing sequences of sets(Fn=∩_m⩾nAm∈_F:n∈_N). From the continuity of probability for the monotonic sets, we have

P(_{lim sup}

n

An) =P(∩n∈NEn) = _lim

n→∞P(En)_, P(_{lim inf}

n An) =P(∪n∈NFn) = _lim

n→∞P(Fn)_. From the definition of two sequences of sets, we obtain

P(En)⩾sup

m⩾n

P(Am), P(Fn)⩽ inf

m⩾nP(Am). Therefore taking limsup and liminf, we obtain

lim sup

n∈N P(En)⩾ inf

n∈Nsup

m⩾n

P(Am)⩾sup

n∈N inf

m⩾nP(Am)⩾lim inf

n∈N P(Fn). SinceP(limnAn) =limnP(En) =limnP(Fn)exists, the result follows.

4