Discrete Distributions

Probability Mass Function

Suppose that

X

is a discrete random varaible, then the probability mass function of

X

is defined by the function

p : ℝ \to [0, 1]

p_{X} (x) = P (X = x)

Bernoulli Random Variable

We say that a random variable

X

is a Bernoulli random varaible iff there is some

p \in [0, 1]

P (X = 1) = p and P (X = 0) = 1 - p

we denote this symbolically as

X \sim bern (p)

Expected Value of a Bernoulli Random Variable

Suppose

X \sim bern (p)

then

E (X) = p

E (X) = P (X = 1) \cdot 1 + P (X = 0) \cdot 0 = p \cdot 1

Binomial Distribution

The random varialbe

X

is said to have a binomial distribution on

n

trials, with probability of success per trial

p

iff

X \overset{d}{=} Z_{1} + Z_{2} + \dots + Z_{n}

where

Z_{1}, \dots, Z_{n} IID Z_{i} \sim bern (p)

Expected Value of the Binomial Distribution

Suppose that

X \sim binom (n, p)

then

E (X) = n p

Probability Mass Function of the Binomial Distribution

Suppose

X \sim binom (n, p)

then

p_{X} (k) = (\binom{n}{k}) p^{k} {(1 - p)}^{n - k}

Note that $p_{X} (k)$ represents the probability of getting exactly $k$ successes in $n$ independent Bernouilli trials each with the same probability.

Coin Counting

A coin is tossed 12 times and 7 heads occur

Determine the probability that the 7th head occurs on the 12th toss given that there are exactly 7 heads in these 12 tosses
Now determine the probability that the 6th head occurs on the 9th toss given that there are exactly 7 heads in the first 12 tosses
Finally determine the probability that the 2nd head toss occurs on the 4th toss and the 6th head occurs on the 9th toss given that there are exactly 7 heads in the first 12 tosses

Let $A$ be the event that the 7th head is on the 12th toss, and $B$ the event that there are exactly 7 heads in these 12 tosses. In this case our goal is to determine $P (A | B) = \frac{P (A \cap B)}{P (B)}$ , note that the event $A$ is equal to $X = k$ where $X \sim binom (12, \frac{1}{2})$ , and thus $P (X = 7) = (\binom{12}{7}) {(\frac{1}{2})}^{7} {(\frac{1}{2})}^{5}$ .

Let's now try to compute $P (A \cap B)$ which is the probability of getting the 7th head on the 12th toss and exactly 7 heads in those 12 tosses. Note that by getting the 7th head on the 12th toss means that in the first 11 tosses exactly 6 heads occurred in any order, and then independently of that you do your last toss so that $P (A \cap B) = ((\binom{11}{6}) {(\frac{1}{2})}^{6} {(\frac{1}{2})}^{5}) \cdot \frac{1}{2}$

Now that we've computed the values of $P (A \cap B)$ along with $P (B)$ we divide them to get the answer.

Let's take a similar approach where this time $C, D$ are the events that the 6th head is on 9th toss and there exactly 7 heads in these 12 tosses respectively, so that our goal is to compute $P (C | D)$ . Recall that we can compute that via $P (C | D) = \frac{P (C \cap D)}{P (D)}$ , but we know what $P (D)$ is from our previous answer, as that was $P (B)$ , therefore we just need a way to compute $P (C \cap D)$ .

$P (C \cap D)$ is the probability that the 6th head is on the 9th toss and there are exactly 7 heads in 12 tosses. We can break this up into two parts, the first being that you have to get exactly 5 heads in the first 8 tosses, then land another head on the 9th toss and then get exactly 1 head in the last 3 tosses, therefore this probability is given by $((\binom{8}{5}) {(\frac{1}{2})}^{5} {(\frac{1}{2})}^{3}) \cdot \frac{1}{2} \cdot ((\binom{3}{1}) {(\frac{1}{2})}^{1} {(\frac{1}{2})}^{2})$

Similarly to the previous two questions, we use the definition of conditional probability to write it as a fraction, we will only need to focus on the numerator which is the probability that the 2nd head occurs on the 4th toss and the 6th head occurs on the 9th toss and that there are exactly 7 heads in the first 12 tosses. It can be handled similarly to the above by splitting it into separate binomial computations.

Consider flipping a coin that has probability $θ$ of coming up heads and $1 - θ$ of coming up tails, then the random variable which evalutes to 1 if the coin is heads, and 0 if the coin is tails has the bernouilli distribution

Poisson Distribution

The random variable

N

is said to have a poisson distribution with average number of successes

λ \in ℝ^{> 0}

iff

P (N = k) = e^{- λ} \frac{λ^{k}}{k!}

where

k \in ℕ_{0}

. We write

N \sim pois (λ)

Poisson as a Limit of Binomial

For any

i \in ℕ

suppose we have some

p_{i} \in [0, 1]

, then define

λ_{i} = i \cdot p_{i}

. Moreover suppose we have some

X_{i} \sim binom (i, p_{i})

. If

\lim_{k \to \infty} λ_{k}

exists denote it by

λ

and we have:

P (N = k) = \lim_{n \to \infty} P (X_{i} = k)

where

N \sim pois (λ)

\begin{matrix} P (X_{n} = k) & a m p; = (\binom{n}{k}) p_{n}^{k} {(1 - p_{n})}^{n - k} \\ a m p; = (\binom{n}{k}) {(\frac{λ_{n}}{n})}^{k} {(1 - \frac{λ_{n}}{n})}^{n - k} \\ a m p; = \frac{n (n - 1) \dots (n - k + 1)}{k (k - 1) \dots 1} {(\frac{λ_{n}}{n})}^{k} {(1 - \frac{λ_{n}}{n})}^{n - k} \\ a m p; = {(1 - \frac{λ_{n}}{n})}^{n} \frac{λ_{n}^{k}}{k!} \cdot (\frac{\frac{n (n - 1) \dots (n - k - 1)}{n^{k}}}{{(1 - \frac{λ_{n}}{n})}^{k}}) \\ a m p; = {(1 - \frac{λ_{n}}{n})}^{n} \frac{λ_{n}^{k}}{k!} \cdot (\frac{(1 - \frac{1}{n}) \dots (1 - \frac{k - 1}{n})}{(1 - \frac{λ_{n}}{n}) \dots (1 - \frac{λ_{n}}{n})}) \end{matrix}

Note that as

n \to \infty

we see that

1 - \frac{λ_{n}}{n} \to 1

and

1 - \frac{x}{n} \to 1

where

x \in ℝ

so that

(\frac{(1 - \frac{1}{n}) \dots (1 - \frac{k - 1}{n})}{(1 - \frac{λ_{n}}{n}) \dots (1 - \frac{λ_{n}}{n})}) \to 1

therefore as

n \to \infty

we have that

P (X_{n} = k) \to e^{- λ} \frac{λ^{k}}{k!}

This also allows us to compute a binomial with a large $n$ by approximating it with a poisson.

Number of Trials until Success as a Random Variable

Suppose we have

(Z_{n}), n \in ℕ_{1}

such that

Z_{i} \sim bern (p)

, then we may recursively define the function

T_{i} : ℕ_{1} \to ℕ_{1}

T_{1} = \min ({n \in ℕ : Z_{n} = 1})

and for any

k \geq 2

T_{k} = \min ({n \in ℕ_{1} : Z_{n} = 1} ∖ {T_{1}, \dots T_{k - 1}})

where

T_{k}

is said to count the number of trials required until and include the

k

-th success.

To best understand the above definition consider the sequence $(Z_{n} = 1), n \in ℕ_{1}$ which may be of the form $(0, 0, 1, 0, 1, 0, 0, 1, \dots)$ and note that $T_{1} = \min ({3, 5, 8}) = 3$ , to determine $T_{2}$ we have $\min ({3, 5, 8} ∖ {3}) = 5$ and so on. In this way the formula removes the previous $k - 1$ instances to find the $k$ th instance.

T_{k} = n

Equivalence

For any

n \in ℕ_{1}

we have:

T_{k} = n ⟺ \sum_{i = 1}^{n - 1} Z_{i} = k - 1 \land Z_{n} = 1

Negative Binomial

The random variable

Y

is said to have a negative binomial distribution on

k

successes each with probability

p

iff

Y \overset{d}{=} T_{k}

and we denote this as

Y \sim negbin (k, p)

Probability Mass Function of the Negative Binomial Distribution

Suppose that

Y \sim negbin (k, p)

for

k \in ℕ_{1}

p \in (0, 1]

then for any

n \in {k, k + 1, k + 2, \dots}

P (Y = n) = (\binom{n - 1}{k - 1}) p^{k} {(1 - p)}^{n - k}

Geometric Distribution

We say that the random variable

W

has a geometric distribution denoted by

W \sim geo (p)

where

p \in (0, 1]

when

W \overset{d}{=} T_{1}

Large Number of Tossed Coins

A coin is tossed 1000 statistically independent times and exactly 3 heads occurs.

If we toss it 1000 times more, and let $N$ denote the number of heads we might get this time, estimate the probability that $N$ is anywhere between 1 and 5. [hint: $e^{3} \approx 20.1$ ]
Suppose now we toss the coin $1000 x$ times and let $N_{x}$ denote the number of heads in $1000 x$ tosses and $T_{1}$ the random number of trials until we obtain the first head. How large does $x$ have to be so that $P (T_{1} > x) = 1 / 2$ ?
Now determine the probability that the $2 n d$ head occurs between 750 and 1250 tosses.

This situation is perfectly modelled by the binomial distribution, ( $N \sim binom (1000, \frac{1}{2})$ ) it's just that it's hard to compute the binomial distribution for large values. Therefore we can employ an approximation using the poisson distribution as discussed previously, we want to know $P (N \in {1, 2, 3, 4, 5}) = P (N = 1) + P (N = 2) + P (N = 3) + P (N = 4) + P (N = 5)$ and we can approximate each such value, using $λ = 1000 \cdot \frac{1}{2} = 500$

$P (N = 1) \approx e^{- 500} \frac{500^{1}}{1}$
$P (N = 2) \approx e^{- 500} \frac{500^{2}}{2}$
...
$P (N = 5) \approx e^{- 500} \frac{500^{5}}{5}$

Therefore by adding all these numbers together we obtain an estimation on the probability of $N$ being anywhere between 1 and 5.

We want to determine a value for $x$ such that $P (T_{1} > x) = \frac{1}{2}$ , to understand this concretely if $x = 5$ then $P (T_{1} > 5)$ asks "what is the probability of getting our first head after 5 tosses of the coin")

Poisson Process

Suppose that

(T_{n}, n \in ℕ)

is a poisson process with

T_{n} \sim G (n, λ^{- 1})

and let

U = \frac{T_{3}}{T_{8}} & V = \frac{T_{3}}{T_{8} - T_{3}} .

Determine the following.

$E U$ and $σ (U)$ .
$E V$ and $σ (V)$ .
$P (U > 1 / 2)$ .

Recall that

E (T_{n}) = \frac{n}{λ}

as this is the formula for the expected value for the gamma distribution. Also if a random varaible

X \sim gamma (p, θ)

then we have

E (X^{- 1}) = \frac{1}{θ (p - 1)}

p > 1

. Thus we have

E (\frac{T_{8}}{T_{3}}) = E (T_{8} \cdot \frac{1}{T_{3}})

Since the sequence

T_{n}

is IID (why is that idk) then we can say

E (T_{8} \cdot \frac{1}{T_{3}}) = E (T_{8}) \cdot E (\frac{1}{T_{3}}) = \frac{8}{λ} \cdot \frac{1}{\frac{1}{λ} (3 - 1)} = 4

Fundamental Properties of Covariance

Let

X, Y, Z

be random variables and

c \in ℝ

then we have

$Var (X) = Cov (X, X)$
$Cov (c X, Y) = c \cdot Cov (X, Y) = Cov (X, c Y)$
$Cov (X + Y, Z) = Cov (X, Z) + Cov (Y, Z)$
$Cov (X, Y + Z) = Cov (X, Y) + Cov (X, Z)$
$Cov (X, Y) = Cov (Y, X)$
$Cov (X, c) = 0$

Covariance Equations

For any

ℝ

-valued

X

and

Y

and any scalars

s

and

t

consider the simple difference

W = Y - (s + t X)

. But, if

E W = 0 = cov (X, W)

this will determine

s

and

t

Assuming that $E X = E Y = var X = var X = 1$ and $cov (X, Y) = 1 / 2$ determine two simultaneous equations thus to obtain the unique $α$ and $β$ such that $Y = α + β X + W and E (W) = 0 = cov (X, W) .$
Now supposing that $X \sim gamma (p = 3, θ = 2)$ and $Y = X^{- 1}$ , determine $α$ and $β$
For the assumptions in b), determine the correlation coefficient $ρ (α + β X, Y)$ .

Before doing anything else we consider the following

\begin{matrix} Cov (Y, W) & a m p; = Cov (Y, Y - (s + t X)) \\ a m p; = Cov (Y, Y) - Cov (Y, s + t X) \\ a m p; = Var (Y) - t Cov (Y, X) \\ a m p; = 1 - t \frac{1}{2} \end{matrix}

We did this because we knew that the properties of co-variance would result in some expression where some of the sub-expressions are the same as the ones we know information about. One thing that we require is that

E (W) = 0

, we see that

E (W) = E (Y - (s + t X)) = E (Y) - s - t E (X) = 0

So we conclude that

s = E (Y) - t E (X)

. We also need that

Cov (X, W) = 0

that is to say

\begin{matrix} Cov (X, W) & a m p; = Cov (X, Y - (s + t X)) \\ a m p; = Cov (X, Y) - Cov (X, s + t X) \\ a m p; = Cov (X, Y) - (0 + t Cov (X, X)) \\ a m p; = Cov (X, Y) - t Var (X) \\ a m p; = 0 \end{matrix}

in that we have

t = \frac{Cov (X, Y)}{Var (X)} = \frac{1}{2}

by our assumptions. This also allows us to deduce that

s = E (Y) - \frac{1}{2} E (X)

therefore we can now observe that

\begin{matrix} Y & a m p; = W + (s + t X) \\ a m p; = W + (E (Y) - \frac{1}{2} E (X) + (\frac{1}{2}) X) \\ a m p; = (E (Y) - \frac{1}{2} E (X)) + \frac{1}{2} X + W \end{matrix}

so that

α = (E (Y) - \frac{1}{2} E (X))

and

β = \frac{1}{2}

Sum of Bernoulli

Suppose that

Z_{i}

IID

bern (p = \frac{1}{4})

for

i \in ℕ_{1}

, determine the following

$P (Z_{1} + Z_{2} = 0 \cap Z_{3} + Z_{4} = 1 \cap Z_{5} + Z_{6} = 2)$
$P (Z_{1} = 0 \cap Z_{2} + Z_{4} = 1 | \sum_{i = 6}^{6} Z_{i} = 3)$

Recall that if $X, Y \sim bern (p)$ then we know that $X + Y \sim bin (2, \frac{1}{4})$ . Therefore we can use the binomial formula when trying to compute $P (X + Y = k)$ for some integer $k$ . In our specific question we note that the given events are independent (todo explain why), and thus we have $\begin{matrix} P (Z_{1} + Z_{2} = 0 \cap Z_{3} + Z_{4} = 1 \cap Z_{5} + Z_{6} = 2) & a m p; = P (Z_{1} + Z_{2} = 0) \cdot P (Z_{3} + Z_{4} = 1) \cdot P (Z_{5} + Z_{6} = 2) \\ a m p; = (\binom{2}{0}) {(\frac{3}{4})}^{2} \cdot (\binom{2}{1}) (\frac{1}{4}) (\frac{3}{4}) \cdot (\binom{2}{2}) {(\frac{1}{4})}^{2} \\ a m p; = \frac{9}{16} \cdot \frac{6}{16} \cdot \frac{1}{16} \\ a m p; = \frac{3^{3}}{2^{11}} \end{matrix}$

To determine this quantity we first use the definition of conditional probability and then employ the same strategy as above.

Geometric Expectation

Suppose that

X | N \sim bin (N, \frac{3}{5})

and that

N \sim geo (\frac{2}{3})

For the varaible $N$ obtain $E (N)$ and $E (N \cdot (N - 1))$

Recall that for $N \sim geo (\frac{2}{3})$ we know that $P (N = n) = p q^{n - 1}$ and an easy way to remember that is that geo is the distribution of how many trials are required before the first success. Note that in this case $p = \frac{2}{3}$ and $q = 1 - p = \frac{1}{3}$ . Now let's try to compute the expected value: $\begin{matrix} E (N) & a m p; = \sum_{n = 1}^{\infty} n \cdot P (N = n) \\ a m p; = \sum_{n = 1}^{\infty} n \cdot p q^{n - 1} \\ a m p; = p \sum_{n = 1}^{\infty} n q^{n - 1} \\ a m p; = p (\frac{d}{d q} (\frac{1}{1 - q})) \\ a m p; = p \cdot \frac{1}{{(1 - q)}^{2}} \\ a m p; = p \cdot \frac{1}{p^{2}} \\ a m p; = \frac{1}{p} \\ a m p; = \frac{3}{2} \end{matrix}$

Taking a look at $E (N \cdot (N - 1))$ then we have $\begin{matrix} E (N \cdot (N - 1)) & a m p; = \sum_{n = 2}^{\infty} n \cdot (n - 1) \cdot p q^{n - 1} \\ a m p; = p q \sum_{n = 2}^{\infty} n (n - 1) q^{n - 2} \\ a m p; = p q (\frac{d^{2}}{d q^{2}} (\frac{1}{1 - q})) \\ a m p; = p q \frac{2}{{(1 - q)}^{3}} \\ a m p; = \frac{2 q}{p^{2}} \\ a m p; = \frac{2 / 3}{{(2 / 3)}^{2}} = \frac{3}{2} \end{matrix}$

Poisson Quotient

Suppose that

T_{n}, n \in ℕ_{1}

is a poisson process where we know that

T_{n} \sim gamma (n, \frac{1}{5})

and let

U = \frac{T_{2}}{T_{5}}

and

V = \frac{T_{2}}{T_{5} - T_{2}}

Observe that the random variable $U$ satisfies $U = \frac{T_{2}}{T_{5}} = \frac{T_{2}}{T_{2} + T_{3}}$ meaning that $U \sim beta (2, 3)$ , and then we have $E (U) = E (\frac{T_{2}}{T_{5}}) = \frac{E (T_{2})}{E (T_{5})} = \frac{2 / λ}{5 / λ} = \frac{2}{5}$