Discrete Distributions

Probability Mass Function

Suppose that

X

is a discrete random variable. The probability mass function of

X

is defined by the function

p_{X} : ℝ \to [0, 1]

given by

p_{X} (x) = P (X = x)

Probability Mass Function Properties

A function

p : ℝ \to [0,1]

is the probability mass function of a discrete random variable exactly when its support is countable and

\sum_{x} p (x) = 1.

p

is a mass function, countable additivity over the events

{X = x}

gives the sum

1

. Conversely, a non-negative countably supported function with total mass

1

defines a probability distribution by summing over subsets of its support.

Joint Probability Mass Function

X

and

Y

are discrete random variables, their joint probability mass function is

p_{X, Y} (x, y) : = P ({X = x} \cap {Y = y}) .

Discrete Marginal Mass Functions

p_{X, Y}

is the joint mass function of

X

and

Y

, then

p_{X} (x) = \sum_{y} p_{X, Y} (x, y), p_{Y} (y) = \sum_{x} p_{X, Y} (x, y) .

Sum the disjoint events

{X = x} \cap {Y = y}

over all possible values of the other variable.

Discrete Independence by Joint Mass Function

Discrete random variables

X

and

Y

are independent if and only if

p_{X, Y} (x, y) = p_{X} (x) p_{Y} (y)

for all

x, y \in ℝ

Suppose first that

X

and

Y

are independent random variables. For fixed

x, y \in ℝ

, the singleton sets

{x}

and

{y}

are Borel sets, so the events

{X = x}

and

{Y = y}

are independent. Therefore, by the definition of independence of events and the definition of the joint probability mass function,

p_{X, Y} (x, y) = P ({X = x} \cap {Y = y}) = P (X = x) P (Y = y) = p_{X} (x) p_{Y} (y) .

Conversely, suppose that $p_{X, Y} (x, y) = p_{X} (x) p_{Y} (y)$ for all $x, y \in ℝ$ . Let $A, B \subseteq ℝ$ be Borel sets. Since $X$ and $Y$ are discrete random variables, the sets of possible values $S_{X} : = X (Ω)$ and $S_{Y} : = Y (Ω)$ are countable. We have the countable union

{X \in A} \cap {Y \in B} = ⋃_{x \in A \cap S_{X}} ⋃_{y \in B \cap S_{Y}} ({X = x} \cap {Y = y}) .

To see the equality, suppose $ω \in {X \in A} \cap {Y \in B}$ . Then $X (ω) \in A \cap S_{X}$ and $Y (ω) \in B \cap S_{Y}$ , so $ω$ belongs to the union term indexed by $x = X (ω)$ and $y = Y (ω)$ . Conversely, every union term is contained in ${X \in A} \cap {Y \in B}$ .

The union is pairwise disjoint: if $ω$ belongs to both ${X = x} \cap {Y = y}$ and ${X = x^{'}} \cap {Y = y^{'}}$ , then $X (ω) = x = x^{'}$ and $Y (ω) = y = y^{'}$ . Thus two different pairs $(x, y) \neq (x^{'}, y^{'})$ cannot have overlapping events. Hence, by the countable additivity in the definition of a probability measure,

\begin{matrix} P ({X \in A} \cap {Y \in B}) & = \sum_{x \in A \cap S_{X}} \sum_{y \in B \cap S_{Y}} P ({X = x} \cap {Y = y}) \\ = \sum_{x \in A \cap S_{X}} \sum_{y \in B \cap S_{Y}} p_{X, Y} (x, y) \\ = \sum_{x \in A \cap S_{X}} \sum_{y \in B \cap S_{Y}} p_{X} (x) p_{Y} (y) \\ = (\sum_{x \in A \cap S_{X}} p_{X} (x)) (\sum_{y \in B \cap S_{Y}} p_{Y} (y)) \\ = P (X \in A) P (Y \in B) . \end{matrix}

Since this holds for every pair of Borel sets

A

and

B

X

and

Y

are independent by the definition of independent random variables.

Conditional Mass Function

X

and

Y

are discrete and

P (X = x) > 0

, the conditional mass function of

Y

given

X = x

is the discrete version of conditional probability:

p_{Y | X} (y | x) : = P (Y = y | X = x) = \frac{p_{X, Y} (x, y)}{p_{X} (x)} .

Conditional Expectation Given a Value

X

and

Y

are discrete and

P (X = x) > 0

, then using the conditional mass function,

E (Y | X = x) : = \sum_{y} y p_{Y | X} (y | x),

whenever the sum is defined.

Law of Total Expectation

X

and

Y

are discrete and the expectations exist, then

E (Y) = \sum_{x} E (Y | X = x) P (X = x) .

Substitute the definition of conditional expectation and use

p_{Y | X} (y | x) P (X = x) = p_{X, Y} (x, y)

, then sum over

x

and

y

Discrete Convolution Formula

X

and

Y

are independent integer-valued random variables, then

P (X + Y = z) = \sum_{x} P (X = x) P (Y = z - x) .

The event

{X + Y = z}

is the disjoint union of the events

{X = x} \cap {Y = z - x}

. Sum their probabilities and use independence.

Indicator Random Variable

For an event

A

, the indicator random variable

𝟏_{A}

is defined by

𝟏_{A} (ω) = 1

ω \in A

, and

𝟏_{A} (ω) = 0

otherwise.

Expectation of Indicator

For any event

A

, the indicator random variable satisfies

E (𝟏_{A}) = P (A) .

The indicator is

1

with probability

P (A)

and

0

with probability

1 - P (A)

Bernoulli Random Variable

We say that a discrete random variable

X

is a Bernoulli random variable iff there is some

p \in [0, 1]

such that

P (X = 1) = p and P (X = 0) = 1 - p .

We denote this symbolically as

X \sim bern (p)

Expected Value of a Bernoulli Random Variable

Suppose

X \sim bern (p)

. By expected value,

E (X) = p

E (X) = P (X = 1) \cdot 1 + P (X = 0) \cdot 0 = p \cdot 1

Binomial Distribution

The random variable

X

is said to have a binomial distribution on

n

trials, with probability of success per trial

p

iff

X \overset{d}{=} Z_{1} + Z_{2} + \dots + Z_{n}

where

Z_{1}, \dots, Z_{n}

are independent identically distributed Bernoulli random variables with

Z_{i} \sim bern (p)

Expected Value of the Binomial Distribution

Suppose that

X \sim binom (n, p)

. Since a binomial variable is a sum of Bernoulli random variables,

E (X) = n p

Probability Mass Function of the Binomial Distribution

Suppose

X \sim binom (n, p)

. Its probability mass function is

p_{X} (k) = (\binom{n}{k}) p^{k} {(1 - p)}^{n - k}

Note that $p_{X} (k)$ represents the probability of getting exactly $k$ successes in $n$ independent Bernoulli trials each with the same probability.

Coin Counting

A coin is tossed 12 times and 7 heads occur

Determine the probability that the 7th head occurs on the 12th toss given that there are exactly 7 heads in these 12 tosses
Now determine the probability that the 6th head occurs on the 9th toss given that there are exactly 7 heads in the first 12 tosses
Finally determine the probability that the 2nd head toss occurs on the 4th toss and the 6th head occurs on the 9th toss given that there are exactly 7 heads in the first 12 tosses

Let $A$ be the event that the 7th head is on the 12th toss, and $B$ the event that there are exactly 7 heads in these 12 tosses. In this case our goal is to determine $P (A | B) = \frac{P (A \cap B)}{P (B)}$ , note that the event $A$ is equal to $X = k$ where $X \sim binom (12, \frac{1}{2})$ , and thus $P (X = 7) = (\binom{12}{7}) {(\frac{1}{2})}^{7} {(\frac{1}{2})}^{5}$ .

Let's now try to compute $P (A \cap B)$ which is the probability of getting the 7th head on the 12th toss and exactly 7 heads in those 12 tosses. Note that by getting the 7th head on the 12th toss means that in the first 11 tosses exactly 6 heads occurred in any order, and then independently of that you do your last toss so that $P (A \cap B) = ((\binom{11}{6}) {(\frac{1}{2})}^{6} {(\frac{1}{2})}^{5}) \cdot \frac{1}{2}$

Now that we've computed the values of $P (A \cap B)$ along with $P (B)$ we divide them to get the answer.

Let's take a similar approach where this time $C, D$ are the events that the 6th head is on 9th toss and there exactly 7 heads in these 12 tosses respectively, so that our goal is to compute $P (C | D)$ . Recall that we can compute that via $P (C | D) = \frac{P (C \cap D)}{P (D)}$ , but we know what $P (D)$ is from our previous answer, as that was $P (B)$ , therefore we just need a way to compute $P (C \cap D)$ .

$P (C \cap D)$ is the probability that the 6th head is on the 9th toss and there are exactly 7 heads in 12 tosses. We can break this up into two parts, the first being that you have to get exactly 5 heads in the first 8 tosses, then land another head on the 9th toss and then get exactly 1 head in the last 3 tosses, therefore this probability is given by $((\binom{8}{5}) {(\frac{1}{2})}^{5} {(\frac{1}{2})}^{3}) \cdot \frac{1}{2} \cdot ((\binom{3}{1}) {(\frac{1}{2})}^{1} {(\frac{1}{2})}^{2})$

Similarly to the previous two questions, we use the definition of conditional probability to write it as a fraction, we will only need to focus on the numerator which is the probability that the 2nd head occurs on the 4th toss and the 6th head occurs on the 9th toss and that there are exactly 7 heads in the first 12 tosses. It can be handled similarly to the above by splitting it into separate binomial computations.

Consider flipping a coin that has probability $θ$ of coming up heads and $1 - θ$ of coming up tails, then the random variable which evalutes to 1 if the coin is heads, and 0 if the coin is tails has the bernouilli distribution

Poisson Distribution

The random variable

N

is said to have a poisson distribution with average number of successes

λ \in ℝ^{> 0}

iff

P (N = k) = e^{- λ} \frac{λ^{k}}{k!}

where

k \in ℕ_{0}

. We write

N \sim pois (λ)

Poisson as a Limit of Binomial

For any

i \in ℕ

suppose we have some

p_{i} \in [0, 1]

, then define

λ_{i} = i \cdot p_{i}

. Moreover suppose we have some

X_{i} \sim binom (i, p_{i})

. If

\lim_{k \to \infty} λ_{k}

exists denote it by

λ

and we have:

P (N = k) = \lim_{n \to \infty} P (X_{i} = k)

where

N \sim pois (λ)

\begin{matrix} P (X_{n} = k) & = (\binom{n}{k}) p_{n}^{k} {(1 - p_{n})}^{n - k} \\ = (\binom{n}{k}) {(\frac{λ_{n}}{n})}^{k} {(1 - \frac{λ_{n}}{n})}^{n - k} \\ = \frac{n (n - 1) \dots (n - k + 1)}{k (k - 1) \dots 1} {(\frac{λ_{n}}{n})}^{k} {(1 - \frac{λ_{n}}{n})}^{n - k} \\ = {(1 - \frac{λ_{n}}{n})}^{n} \frac{λ_{n}^{k}}{k!} \cdot (\frac{\frac{n (n - 1) \dots (n - k - 1)}{n^{k}}}{{(1 - \frac{λ_{n}}{n})}^{k}}) \\ = {(1 - \frac{λ_{n}}{n})}^{n} \frac{λ_{n}^{k}}{k!} \cdot (\frac{(1 - \frac{1}{n}) \dots (1 - \frac{k - 1}{n})}{(1 - \frac{λ_{n}}{n}) \dots (1 - \frac{λ_{n}}{n})}) \end{matrix}

Note that as

n \to \infty

we see that

1 - \frac{λ_{n}}{n} \to 1

and

1 - \frac{x}{n} \to 1

where

x \in ℝ

so that

(\frac{(1 - \frac{1}{n}) \dots (1 - \frac{k - 1}{n})}{(1 - \frac{λ_{n}}{n}) \dots (1 - \frac{λ_{n}}{n})}) \to 1

therefore as

n \to \infty

we have that

P (X_{n} = k) \to e^{- λ} \frac{λ^{k}}{k!}

This also allows us to compute a binomial with a large $n$ by approximating it with a poisson.

Number of Trials until Success as a Random Variable

Suppose we have

(Z_{n}), n \in ℕ_{1}

such that

Z_{i} \sim bern (p)

, then we may recursively define the function

T_{i} : ℕ_{1} \to ℕ_{1}

T_{1} = \min ({n \in ℕ : Z_{n} = 1})

and for any

k \geq 2

T_{k} = \min ({n \in ℕ_{1} : Z_{n} = 1} ∖ {T_{1}, \dots T_{k - 1}})

where

T_{k}

is said to count the number of trials required until and include the

k

-th success.

To best understand the above definition consider the sequence $(Z_{n} = 1), n \in ℕ_{1}$ which may be of the form $(0, 0, 1, 0, 1, 0, 0, 1, \dots)$ and note that $T_{1} = \min ({3, 5, 8}) = 3$ , to determine $T_{2}$ we have $\min ({3, 5, 8} ∖ {3}) = 5$ and so on. In this way the formula removes the previous $k - 1$ instances to find the $k$ th instance.

T_{k} = n

Equivalence

For any

n \in ℕ_{1}

we have:

T_{k} = n ⟺ \sum_{i = 1}^{n - 1} Z_{i} = k - 1 \land Z_{n} = 1

Negative Binomial

The random variable

Y

is said to have a negative binomial distribution on

k

successes each with probability

p

iff

Y \overset{d}{=} T_{k}

and we denote this as

Y \sim negbin (k, p)

Probability Mass Function of the Negative Binomial Distribution

Suppose that

Y \sim negbin (k, p)

for

k \in ℕ_{1}

p \in (0, 1]

then for any

n \in {k, k + 1, k + 2, \dots}

P (Y = n) = (\binom{n - 1}{k - 1}) p^{k} {(1 - p)}^{n - k}

Geometric Distribution

We say that the random variable

W

has a geometric distribution denoted by

W \sim geo (p)

where

p \in (0, 1]

when

W \overset{d}{=} T_{1}

Large Number of Tossed Coins

A coin is tossed 1000 statistically independent times and exactly 3 heads occurs.

If we toss it 1000 times more, and let $N$ denote the number of heads we might get this time, estimate the probability that $N$ is anywhere between 1 and 5. [hint: $e^{3} \approx 20.1$ ]
Suppose now we toss the coin $1000 x$ times and let $N_{x}$ denote the number of heads in $1000 x$ tosses and $T_{1}$ the random number of trials until we obtain the first head. How large does $x$ have to be so that $P (T_{1} > x) = 1 / 2$ ?
Now determine the probability that the $2 n d$ head occurs between 750 and 1250 tosses.

This situation is perfectly modelled by the binomial distribution, ( $N \sim binom (1000, \frac{1}{2})$ ) it's just that it's hard to compute the binomial distribution for large values. Therefore we can employ an approximation using the poisson distribution as discussed previously, we want to know $P (N \in {1, 2, 3, 4, 5}) = P (N = 1) + P (N = 2) + P (N = 3) + P (N = 4) + P (N = 5)$ and we can approximate each such value, using $λ = 1000 \cdot \frac{1}{2} = 500$

$P (N = 1) \approx e^{- 500} \frac{500^{1}}{1}$
$P (N = 2) \approx e^{- 500} \frac{500^{2}}{2}$
...
$P (N = 5) \approx e^{- 500} \frac{500^{5}}{5}$

Therefore by adding all these numbers together we obtain an estimation on the probability of $N$ being anywhere between 1 and 5.

We want to determine a value for $x$ such that $P (T_{1} > x) = \frac{1}{2}$ , to understand this concretely if $x = 5$ then $P (T_{1} > 5)$ asks "what is the probability of getting our first head after 5 tosses of the coin")

Poisson Process

Suppose that

(T_{n}, n \in ℕ)

is a poisson process with

T_{n} \sim G (n, λ^{- 1})

and let

U = \frac{T_{3}}{T_{8}} & V = \frac{T_{3}}{T_{8} - T_{3}} .

Determine the following.

$E U$ and $σ (U)$ .
$E V$ and $σ (V)$ .
$P (U > 1 / 2)$ .

Recall that

E (T_{n}) = \frac{n}{λ}

as this is the formula for the expected value for the gamma distribution. Also if a random varaible

X \sim gamma (p, θ)

then we have

E (X^{- 1}) = \frac{1}{θ (p - 1)}

p > 1

. Thus we have

E (\frac{T_{8}}{T_{3}}) = E (T_{8} \cdot \frac{1}{T_{3}})

Since the sequence

T_{n}

is IID (why is that idk) then we can say

E (T_{8} \cdot \frac{1}{T_{3}}) = E (T_{8}) \cdot E (\frac{1}{T_{3}}) = \frac{8}{λ} \cdot \frac{1}{\frac{1}{λ} (3 - 1)} = 4

Fundamental Properties of Covariance

Let

X, Y, Z

be random variables and

c \in ℝ

then we have

$Var (X) = Cov (X, X)$
$Cov (c X, Y) = c \cdot Cov (X, Y) = Cov (X, c Y)$
$Cov (X + Y, Z) = Cov (X, Z) + Cov (Y, Z)$
$Cov (X, Y + Z) = Cov (X, Y) + Cov (X, Z)$
$Cov (X, Y) = Cov (Y, X)$
$Cov (X, c) = 0$

Covariance Equations

For any

ℝ

-valued

X

and

Y

and any scalars

s

and

t

consider the simple difference

W = Y - (s + t X)

. But, if

E W = 0 = cov (X, W)

this will determine

s

and

t

Assuming that $E X = E Y = var X = var X = 1$ and $cov (X, Y) = 1 / 2$ determine two simultaneous equations thus to obtain the unique $α$ and $β$ such that $Y = α + β X + W and E (W) = 0 = cov (X, W) .$
Now supposing that $X \sim gamma (p = 3, θ = 2)$ and $Y = X^{- 1}$ , determine $α$ and $β$
For the assumptions in b), determine the correlation coefficient $ρ (α + β X, Y)$ .

Before doing anything else we consider the following

\begin{matrix} Cov (Y, W) & = Cov (Y, Y - (s + t X)) \\ = Cov (Y, Y) - Cov (Y, s + t X) \\ = Var (Y) - t Cov (Y, X) \\ = 1 - t \frac{1}{2} \end{matrix}

We did this because we knew that the properties of co-variance would result in some expression where some of the sub-expressions are the same as the ones we know information about. One thing that we require is that

E (W) = 0

, we see that

E (W) = E (Y - (s + t X)) = E (Y) - s - t E (X) = 0

So we conclude that

s = E (Y) - t E (X)

. We also need that

Cov (X, W) = 0

that is to say

\begin{matrix} Cov (X, W) & = Cov (X, Y - (s + t X)) \\ = Cov (X, Y) - Cov (X, s + t X) \\ = Cov (X, Y) - (0 + t Cov (X, X)) \\ = Cov (X, Y) - t Var (X) \\ = 0 \end{matrix}

in that we have

t = \frac{Cov (X, Y)}{Var (X)} = \frac{1}{2}

by our assumptions. This also allows us to deduce that

s = E (Y) - \frac{1}{2} E (X)

therefore we can now observe that

\begin{matrix} Y & = W + (s + t X) \\ = W + (E (Y) - \frac{1}{2} E (X) + (\frac{1}{2}) X) \\ = (E (Y) - \frac{1}{2} E (X)) + \frac{1}{2} X + W \end{matrix}

so that

α = (E (Y) - \frac{1}{2} E (X))

and

β = \frac{1}{2}

Sum of Bernoulli

Suppose that

Z_{i}

IID

bern (p = \frac{1}{4})

for

i \in ℕ_{1}

, determine the following

$P (Z_{1} + Z_{2} = 0 \cap Z_{3} + Z_{4} = 1 \cap Z_{5} + Z_{6} = 2)$
$P (Z_{1} = 0 \cap Z_{2} + Z_{4} = 1 | \sum_{i = 6}^{6} Z_{i} = 3)$

Recall that if $X, Y \sim bern (p)$ then we know that $X + Y \sim bin (2, \frac{1}{4})$ . Therefore we can use the binomial formula when trying to compute $P (X + Y = k)$ for some integer $k$ . In our specific question we note that the given events are independent (todo explain why), and thus we have $\begin{matrix} P (Z_{1} + Z_{2} = 0 \cap Z_{3} + Z_{4} = 1 \cap Z_{5} + Z_{6} = 2) & = P (Z_{1} + Z_{2} = 0) \cdot P (Z_{3} + Z_{4} = 1) \cdot P (Z_{5} + Z_{6} = 2) \\ = (\binom{2}{0}) {(\frac{3}{4})}^{2} \cdot (\binom{2}{1}) (\frac{1}{4}) (\frac{3}{4}) \cdot (\binom{2}{2}) {(\frac{1}{4})}^{2} \\ = \frac{9}{16} \cdot \frac{6}{16} \cdot \frac{1}{16} \\ = \frac{3^{3}}{2^{11}} \end{matrix}$

To determine this quantity we first use the definition of conditional probability and then employ the same strategy as above.

Geometric Expectation

Suppose that

X | N \sim bin (N, \frac{3}{5})

and that

N \sim geo (\frac{2}{3})

For the varaible $N$ obtain $E (N)$ and $E (N \cdot (N - 1))$

Recall that for $N \sim geo (\frac{2}{3})$ we know that $P (N = n) = p q^{n - 1}$ and an easy way to remember that is that geo is the distribution of how many trials are required before the first success. Note that in this case $p = \frac{2}{3}$ and $q = 1 - p = \frac{1}{3}$ . Now let's try to compute the expected value: $\begin{matrix} E (N) & = \sum_{n = 1}^{\infty} n \cdot P (N = n) \\ = \sum_{n = 1}^{\infty} n \cdot p q^{n - 1} \\ = p \sum_{n = 1}^{\infty} n q^{n - 1} \\ = p (\frac{d}{d q} (\frac{1}{1 - q})) \\ = p \cdot \frac{1}{{(1 - q)}^{2}} \\ = p \cdot \frac{1}{p^{2}} \\ = \frac{1}{p} \\ = \frac{3}{2} \end{matrix}$

Taking a look at $E (N \cdot (N - 1))$ then we have $\begin{matrix} E (N \cdot (N - 1)) & = \sum_{n = 2}^{\infty} n \cdot (n - 1) \cdot p q^{n - 1} \\ = p q \sum_{n = 2}^{\infty} n (n - 1) q^{n - 2} \\ = p q (\frac{d^{2}}{d q^{2}} (\frac{1}{1 - q})) \\ = p q \frac{2}{{(1 - q)}^{3}} \\ = \frac{2 q}{p^{2}} \\ = \frac{2 / 3}{{(2 / 3)}^{2}} = \frac{3}{2} \end{matrix}$

Poisson Quotient

Suppose that

T_{n}, n \in ℕ_{1}

is a poisson process where we know that

T_{n} \sim gamma (n, \frac{1}{5})

and let

U = \frac{T_{2}}{T_{5}}

and

V = \frac{T_{2}}{T_{5} - T_{2}}

Observe that the random variable $U$ satisfies $U = \frac{T_{2}}{T_{5}} = \frac{T_{2}}{T_{2} + T_{3}}$ meaning that $U \sim beta (2, 3)$ , and then we have $E (U) = E (\frac{T_{2}}{T_{5}}) = \frac{E (T_{2})}{E (T_{5})} = \frac{2 / λ}{5 / λ} = \frac{2}{5}$