Lecture-01: Probability Review

Lecture-01: Probability Review

1 Probability Review

Definition 1.1. A probability space(Ω,F,P)consists of set of all possible outcomes called a sample space and denoted byΩ, a collection of subsetsFof sample space calledevent space, and a non-negative set function probabilityP:F→[0,1], with the following properties.

1. Event spaceFis aσ-algebra, that is it contains an empty set and is closed under complements and countable unions.

2. Set functionPsatisfiesP(Ω) =1, and is additive for countably disjoint events.

An element of the sample space is called an outcome and an element of event space is called an event.

Example 1.2 (Discreteσ-algebra). For a finite sample spaceΩ, the event spaceF={A:A⊆Ω}consists of all subsets of sample spaceΩ.

Example 1.3 (Borelσ-algebra). If the sample spaceΩ=R, then aBorelσ-algebra is generated by half- open intervals by complements and countable unions. That is, B=σ({(−∞,x]:x∈R}). We make the following observations.

1. From closure under complements, the open interval(x,∞)belong toBfor eachx∈R.

2. From closure under countable unions, the open interval(−∞,x) =∪n∈N(−∞,x−¹_n]belongs toBfor eachx∈R.

3. From closure under countable intersections, the singleton{x}=^T_n∈N([x−¹_n,∞)∩(−∞,x+¹_n])belongs toBfor eachx∈R.

There is a natural order of inclusion on sets through which we can define monotonicity of probability set functionP. To define continuity of this set function, we define limits of sets.

Definition 1.4. For a sequence of sets(A_n:n∈N), we definelimit superiorandlimit inferiorof this set sequence respectively as

lim sup

n

A_n= ^\

n∈N [

k>n

A_k, lim inf

n A_n= ^[

n∈N

\

k>n

A_k.

It is easy to check that lim infA_n⊆lim supA_n. We say that limit of set sequence exists if lim supA_n⊆lim infA_n, and the limit of the set sequence in this case is lim supA_n.

Theorem 1.5. Probability set function is monotone and continuous.

Proof. Consider two eventsA⊆Bboth elements ofF, then from the additivity of probability over disjoint events AandB\A, we have

P(B) =P(A∪B\A) =P(A) +P(B\A)>P(A).

Monotonicity follows from non-negativity of probability set function, that is sinceP(B\A)>0. For continuity from below, we take an increasing sequence of sets(A_n:n∈N), such thatAn⊆An+1for alln. Then, it is clear thatAn↑A=∪_nAn. We can define disjoint sets(E_n:n∈N), where

E₁=A₁, E_n=A_n\An−1,n>2.

1

The disjoint sets E_n’s satisfy ∪ⁿ_i=1E_i=A_n for alln∈Nand∪_nE_n=∪_nA_n. From the above property and the additivity of probability set function over disjoint sets, it follows that

P(A) =P(∪_nEn) =

∑

n∈N

P(E_n) =lim

n∈N n i=1

∑

P(E_i) =lim

n∈N

P∪ⁿ_i=1Ei=lim

n∈N

P(A_n).

For continuity from below, we take decreasing sequence of sets(A_n:n∈N), such thatA_n+1⊆A_nfor alln. We can form increasing sequence of sets (B_n:n∈N) whereB_n=A^c_n. Then, the continuity from above follows from continuity from above. Continuity of probability for general sequence of converging sets follows from the definition of limsup and liminf of sequence of sets and the continuity of probability function from above and below.

1.1 Independence

Definition 1.6. For a probability space(Ω,F,P), two eventsA,B∈Fareindependent eventsif

P(A∩B) =P(A)P(B). (1)

Definition 1.7. A collection of eventsE⊆Fis called a sub-event space if it is aσ-algebra.

Definition 1.8. Two sub-event spacesGandHare called independent if any pair of events(G,H)∈G×Hare independent. That is,

P(G∩H) =P(G)P(H), G∈G,H∈H.

1.2 Conditional Probability

Definition 1.9. Let(Ω,F,P)be a probability space. For eventsA,B∈Fsuch that 1>P(B)>0, the conditional probability of eventAgiven eventBis defined as

P(A|B) =P(A∩B) P(B) .

KnowingP(A|B), we also knowP(A|B^c)ifP(B)<1. Indeed, we can computeP(A|B^c)in terms ofP(A|B)as P(A|B^c) =P(A∩B^c)

P(B^c) =P(A)−P(A∩B)

1−P(B) =P(A)−P(B)P(A|B)

1−P(B) . (2)

We can check thatP(A∩Ω) =P(A)P(Ω)andP(A∩/0) =P(A)P(/0). Therefore, we can defineP(A|Ω) =P(A) andP(A|/0) =P(A). Hence, if we know the conditional probability on an eventB∈F, we know the conditional probability on the event subspace{/0,B,B^c,Ω}.

Definition 1.10. We can define conditional probability of event A on a sub-event space E by the collection (P(A

E):E∈E).

2 Random variables

Definition 2.1. A real valuedrandom variableXon a probability space(Ω,F,P)is a functionX:Ω→Rsuch that for everyx∈R, we have

X⁻¹(−∞,x],{ω∈Ω:X(ω)6x} ∈F.

Recall that the collection((−∞,x]:x∈R)generates the Borelσ-algebraB(R). Therefore, it follows that X⁻¹(B)⊆F, since set inverse mapX⁻¹preserves complements, unions, and intersections.

Definition 2.2. For a random variableXdefined on the probability space(Ω,F,P), we defineσ(X)is the smallest σ-algebra formed by inverse mapping of Borel sets, i.e.

σ(X),σ(X⁻¹(−∞,x]:x∈R).

Note thatσ(X)is a sub-event space ofFand hence probability is defined for each element ofσ(X).

Definition 2.3. For a random variableX defined on probability space(Ω,F,P), Thedistribution functionF : R→[0,1]for this random variableXis defined as

F(x) = (P◦X⁻¹)(−∞,x], for allx∈R. 2

Theorem 2.4. Distribution function F of a random variable X is non-negative, monotone increasing, continuous from the right, and has countable points of discontinuities. Further, if P◦X⁻¹(R) =1, then

x→−∞lim F(x) =0, lim

x→∞F(x) =1.

Proof. Non-negativity and monotonicity of distribution function follows from non-negativity and monotonicity of probability set function, and the fact that forx1<x2

X⁻¹(−∞,x1]⊆X⁻¹(−∞,x2].

Letx_n↓xbe a decreasing sequence of real numbers. We take decreasing sets(A_n∈F:n∈N), where A_n= X⁻¹(−∞,x_n]∈F. The right continuity of distribution function follows from the continuity from above of proba- bility set functions.

Example 2.5. One of the simplest family of random variables are indicator functions1:F×Ω→(0,1). For each eventA∈F, we can define an indicator function as

1A(ω) =

(1, ω∈A, 0, ω∈/A.

We make the following observations.

1. 1Ais a random variable for eachA∈F. This follows from the fact that

1⁻¹_A (−∞,x] =







/0, x<0, A^c, x∈[0,1), Ω,x>1.

2. The distribution functionFfor the random variable1Ais given by

F(x) =







0, x<0, P(A^c), x∈[0,1), 1, x>1.

2.1 Expectation

Letg:R→Rbe a Borel measurable function, i.e. g⁻¹(−∞,x]∈Bfor allx∈R. Then, theexpectationofg(X) for a random variableX with distribution functionFis defined as

Eg(X) = Z

x∈R

g(x)dF(x).

Remark 1. Recall that probabilities are defined only for events. For a random variableX, the probabilities are defined for generating eventsX⁻¹(−∞,x]∈FbyF(x) =P◦X⁻¹(−∞,x].

Remark 2. The expectation is only defined for random variables. For an eventA, the probability P(A)equals expectation of the indicator random variable1A.

2.2 Random Vectors

Definition 2.6. IfX1, . . . ,X_nare random variables defined on the same probability space(Ω,F,P), then the vector

X,(X₁, . . . ,Xn)is a random mappingΩ→Rⁿand is called a random vector. Since eachXiis a random variable,

the joint event∩_i∈[n]X_i⁻¹(−∞,xi]∈F, and the joint distribution of random vectorXis defined as F_X(x₁, . . . ,x_n) =P ∩_i∈[n]X_i⁻¹(−∞,x_i]

, for allx∈Rⁿ.

Definition 2.7. A random variableX isindependentof the event subspaceE, ifσ(X)andE are independent event subspaces.

3

Remark3. Sinceσ(X)is generated by the collection(X⁻¹(−∞,x]:x∈R), it follows thatXis independent ofE if and any if for allx∈Rand eventE∈E,

E[1{X6x}1E] =P({X6x} ∩E) =P({X6x})P(E) =E1{X6x}E1A.

Definition 2.8. Two random variablesX,Y are independent ifσ(X)andσ(Y)are independent event subspaces.

Remark4. Sinceσ(X)andσ(Y)are generated by collections(X⁻¹(−∞,x]:x∈R)and(Y⁻¹(−∞,y]:y∈R), it follows that the random variablesX,Y are independent if and only if for allx,y∈R, we have

F_X,Y(x,y) =F_X(x)F_Y(y).

Definition 2.9. A random vectorX:Ω→Rⁿis independent if the joint distribution is product of marginals. That is,

F_X(x) =

n

∏

i=1

F_Xi(x_i), for allx∈Rⁿ.

4