Probability Models

Sample Space

The sample space of an experiment is the set

Ω

whose elements are the possible outcomes of the experiment.

Event

An event is a subset of the sample space. If the realized outcome

ω \in Ω

lies in

A \subseteq Ω

, then the event

A

occurs.

Impossible and Certain Events

The empty set

\emptyset

is the impossible event, and the whole sample space

Ω

is the certain event.

Increasing Sequence of Events

A sequence of events

A_{1}, A_{2}, \dots

is increasing if

A_{1} \subseteq A_{2} \subseteq A_{3} \subseteq \dots .

Decreasing Sequence of Events

A sequence of events

A_{1}, A_{2}, \dots

is decreasing if

A_{1} \supseteq A_{2} \supseteq A_{3} \supseteq \dots .

Event Sequence Limits

A_{1}, A_{2}, \dots

is an increasing sequence of events and

A = ⋃_{n = 1}^{\infty} A_{n}

, then we say that

A_{n}

increases to

A

and write

A_{n} ↗ A

If $A_{1}, A_{2}, \dots$ is a decreasing sequence of events and $A = ⋂_{n = 1}^{\infty} A_{n}$ , then we say that $A_{n}$ decreases to $A$ and write $A_{n} ↘ A$ .

Probability Measure

Let

ℱ

be a

σ

-algebra on

Ω

. A function

P : ℱ \to [0,1]

is a probability measure if

P (Ω) = 1

and, for every pairwise disjoint sequence

A_{1}, A_{2}, \dots \in ℱ

P (⋃_{n = 1}^{\infty} A_{n}) = \sum_{n = 1}^{\infty} P (A_{n}) .

Probability Space

A probability space is a triple

(Ω, ℱ, P)

, where

Ω

is a sample space,

ℱ

is a

σ

-algebra of events, and

P

is a probability measure on

ℱ

Finite Uniform Distribution

For any specific

N \in ℕ

, the random variable

X

is said to have a (finite discrete) uniform distribution on the sample space

Ω = {1, \dots, N}

- denoted:

X \sim

unif

{1, \dots, N}

iff

P (X = k) = 1 / N, k = 1, \dots, N .

Conditional Probability

Let

A, B \subseteq Ω

be events, with

P (B) \neq 0

. The conditional probability of

A

given

B

is defined as

P (A | B) = \frac{P (A \cap B)}{P (B)} .

Intersection as Conditional Probability

Suppose that

P (A), P (B) \neq 0

. By the definition of conditional probability,

P (A \cap B) = P (A | B) P (B)

Finite Intersection as Conditional Probability

Suppose we have events

E_{1}, \dots, E_{n}

. Then

P (E_{1} \cap E_{2} \cap \dots \cap E_{n}) = P (E_{n} | (E_{1} \cap \dots \cap E_{n - 1})) P (E_{1} \cap \dots \cap E_{n - 1})

whenever the conditional probability is defined.

Let

A : = E_{1} \cap \dots \cap E_{n - 1}

and

B : = E_{n}

. From intersection as conditional probability,

P (A \cap B) = P (B | A) P (A) .

Substituting the definitions of

A

and

B

gives

P ((E_{1} \cap E_{2} \cap \dots \cap E_{n - 1}) \cap E_{n}) = P (E_{n} | (E_{1} \cap E_{2} \cap \dots \cap E_{n - 1})) \cdot P (E_{1} \cap \dots \cap E_{n - 1}) .

Product Rule

Let

E_{1}, \dots, E_{n}

be events. Repeatedly applying finite intersection as conditional probability gives

\begin{matrix} P (E_{1} \cap E_{2} \cap \dots \cap E_{n}) & = P (E_{n} | E_{1} \cap E_{2} \cap \dots \cap E_{n - 1}) \\ \cdot P (E_{n - 1} | E_{1} \cap \dots \cap E_{n - 2}) \\ \dots P (E_{3} | E_{1} \cap E_{2}) \cdot P (E_{2} | E_{1}) P (E_{1}) . \end{matrix}

Proof by induction and using the previous lemma during the induction step and the "intersection as conditional probability" for the base case.

As stated it is notationally unwieldy. We can clean it up using the following: $P (E_{1} \cap E_{2} \cap \dots \cap E_{n}) = P (E_{1}) (\prod_{j = 2}^{n} P (E_{j} | E_{1} \cap \dots \cap E_{j - 1}))$

Symmetric Conditional Equation

P (A)

and

P (B)

are nonzero, then

P (A | B) P (B) = P (B | A) P (A)

By conditional probability,

P (A | B) P (B) = P (A \cap B) .

Since

A \cap B = B \cap A

, we have

P (A \cap B) = P (B \cap A) = P (B | A) P (A) .

Therefore

P (A | B) P (B) = P (B | A) P (A)

Bayes Formula

If the relevant conditional probabilities are defined, then

P (A | B) = \frac{P (B | A) P (A)}{P (B)} .

By the symmetric conditional equation,

P (A | B) P (B) = P (B | A) P (A) .

Since

P (B) \neq 0

, divide both sides by

P (B)

to get

P (A | B) = \frac{P (B | A) P (A)}{P (B)} .

Probability of Set Difference

Suppose that

A, B \subseteq Ω

are events. Then

P (A ∖ B) = P (A) - P (A \cap B)

By set difference decomposition, the event

A

decomposes as the disjoint union

A = (A ∖ B) \cup (A \cap B) .

Since

A ∖ B

and

A \cap B

are disjoint, the probability measure gives

P (A) = P (A ∖ B) + P (A \cap B) .

Subtracting

P (A \cap B)

from both sides gives

P (A ∖ B) = P (A) - P (A \cap B) .

Probability Outside an Event

Suppose that

A \subseteq Ω

is an event. Then

P (Ω ∖ A) = 1 - P (A) .

By probability of set difference,

P (Ω ∖ A) = P (Ω) - P (Ω \cap A) .

Since

A \subseteq Ω

, we have

Ω \cap A = A

. Also,

P (Ω) = 1

by the definition of a probability measure. Therefore

P (Ω ∖ A) = 1 - P (A) .

Probability of the Impossible Event

The impossible event has probability zero:

P (\emptyset) = 0.

Since

Ω ∖ Ω = \emptyset

, probability outside an event gives

P (\emptyset) = P (Ω ∖ Ω) = 1 - P (Ω) = 0.

Probability is Nonnegative

A

is an event, then

P (A) \geq 0.

By the definition of a probability measure,

P

has codomain

[0,1]

, so

P (A) \in [0,1]

Monotonicity of Probability

Suppose that

A \subseteq B \subseteq Ω

are events. Then

P (A) \leq P (B) .

By set difference decomposition,

B = (B ∖ A) \cup (B \cap A)

, and since

A \subseteq B

, we have

B \cap A = A

. Thus

B

is the disjoint union of

B ∖ A

and

A

. Therefore

P (B) = P (B ∖ A) + P (A) \geq P (A)

because

P (B ∖ A) \geq 0

by probability is nonnegative. Thus

P (B) \geq P (A) .

Continuity of Probability

Let

P

be a probability measure. If

A_{n} ↗ A

, then

\lim_{n \to \infty} P (A_{n}) = P (A) .

A_{n} ↘ A

, then

\lim_{n \to \infty} P (A_{n}) = P (A) .

Suppose first that

A_{n} ↗ A

. Define

B_{1} : = A_{1}

and

B_{n} : = A_{n} ∖ A_{n - 1}

for

n \geq 2

. The events

B_{1}, B_{2}, \dots

are pairwise disjoint,

A = ⋃_{n = 1}^{\infty} B_{n}

, and

A_{N} = ⋃_{n = 1}^{N} B_{n}

. By countable additivity,

P (A) = \sum_{n = 1}^{\infty} P (B_{n}),

while

P (A_{N}) = \sum_{n = 1}^{N} P (B_{n})

. Hence

\lim_{N \to \infty} P (A_{N}) = P (A) .

Now suppose that $A_{n} ↘ A$ . Then $A_{1} ∖ A_{n} ↗ A_{1} ∖ A$ . By the increasing case and probability of set difference,

\begin{matrix} \lim_{n \to \infty} P (A_{n}) & = \lim_{n \to \infty} (P (A_{1}) - P (A_{1} ∖ A_{n})) \\ = P (A_{1}) - P (A_{1} ∖ A) \\ = P (A) . \end{matrix}

Probability of an Event through a Partition

Suppose that

𝒳 = {X_{1}, X_{2}, \dots}

is a partition of the sample space

Ω

. Then for any event

E

we have

P (E) = \sum_{i \in ℕ_{1}} P (E \cap X_{i})

Observe that $E = E \cap Ω = E \cap (⊔_{i \in ℕ_{1}} X_{i}) = ⊔_{i \in ℕ_{1}} (E \cap X_{i})$ , therefore $P (E) = \sum_{i \in ℕ_{1}} P (E \cap X_{i})$

It follows as an easy corollary that it holds for finite partitions by setting the rest of the partition to be the empty set.

Law of Total Probability

Suppose that

𝒳 = {X_{1}, X_{2}, \dots}

is a partition of

Ω

. Combining probability through a partition with conditional probability gives

P (E) = \sum_{i \in ℕ_{1}} P (E | X_{i}) P (X_{i})

Independence of Events

Given two events

A, B \subseteq Ω

, we say that

A

and

B

are independent iff

P (A \cap B) = P (A) P (B) .

Independence is Symmetric

Suppose that

A

and

B

are independent. Then

B

and

A

are independent.

Since

A \cap B = B \cap A

P (B \cap A) = P (A \cap B) = P (A) P (B) = P (B) P (A) .

Independent Family of Events

A family of events

{A_{i} : i \in I}

is independent if, for every finite

J \subseteq I

P (⋂_{j \in J} A_{j}) = \prod_{j \in J} P (A_{j}) .

Pairwise Independent Events

A family of events

{A_{i} : i \in I}

is pairwise independent if each pair is independent, so

P (A_{i} \cap A_{j}) = P (A_{i}) P (A_{j})

whenever

i \neq j

. Pairwise independence does not, by itself, imply independence of the whole family.

Independence is Preserved by Sample-Space Difference

A

and

B

are independent events, then

Ω ∖ A

and

B

A

and

Ω ∖ B

, and

Ω ∖ A

and

Ω ∖ B

are independent.

First note that we will use probability outside an event. Now,

\begin{matrix} P ((Ω ∖ A) \cap B) & = P (B ∖ A) & (ambient difference) \\ = P (B) - P (A \cap B) & (probability of set difference) \\ = P (B) - P (A) P (B) & (independence) \\ = (1 - P (A)) P (B) & (algebra) \\ = P (Ω ∖ A) P (B) . & (probability outside event) \end{matrix}

Therefore

Ω ∖ A

and

B

are independent. Similarly,

\begin{matrix} P (A \cap (Ω ∖ B)) & = P (A ∖ B) & (ambient difference) \\ = P (A) - P (A \cap B) \\ = P (A) - P (A) P (B) \\ = P (A) (1 - P (B)) \\ = P (A) P (Ω ∖ B) . \end{matrix}

Therefore

A

and

Ω ∖ B

are independent. Finally,

\begin{matrix} P ((Ω ∖ A) \cap (Ω ∖ B)) & = P ((Ω ∖ A) ∖ B) & (intersection of differences) \\ = P (Ω ∖ A) - P ((Ω ∖ A) \cap B) \\ = P (Ω ∖ A) - P (Ω ∖ A) P (B) \\ = P (Ω ∖ A) (1 - P (B)) \\ = P (Ω ∖ A) P (Ω ∖ B) . \end{matrix}

Therefore

Ω ∖ A

and

Ω ∖ B

are independent.

Product Probability Space

(Ω_{1}, ℱ_{1}, P_{1})

and

(Ω_{2}, ℱ_{2}, P_{2})

are probability spaces, their independent product has sample space

Ω_{1} \times Ω_{2}

, events generated by rectangles

A_{1} \times A_{2}

, and probability determined on rectangles by

P (A_{1} \times A_{2}) = P_{1} (A_{1}) P_{2} (A_{2}) .