Continuous Distributions

Continuous Random Variable

We say that a random variable

X

is continuous if

P (X = x) = 0

for every

x \in ℝ

This is not the same as saying that $X$ is not a discrete random variable. A discrete random variable has a countable set of possible values. A continuous random variable has no single value with positive probability. These are different conditions.

Continuous Random Variable Has Uncountable Range

X

is a continuous random variable, then

X (Ω)

is not countable.

Suppose, toward a contradiction, that

X (Ω)

is countable. Since every outcome

ω \in Ω

has

X (ω) \in X (Ω)

Ω = ⋃_{x \in X (Ω)} {ω \in Ω : X (ω) = x} .

Because

X

is continuous, each event

{X = x}

has probability

0

. By countable additivity in the definition of a probability measure,

P (Ω) = \sum_{x \in X (Ω)} P (X = x) = 0.

This contradicts

P (Ω) = 1

, so

X (Ω)

is not countable.

Uncountable Range Does Not Imply Continuous

A random variable can have an uncountable set of possible values without being continuous.

Let

X

be a random variable with

X (Ω) = [0,1]

whose distribution function is

F_{X} (x) = {\begin{matrix} 0, & x < 0, \\ \frac{1}{2} + \frac{x}{2}, & 0 \leq x \leq 1, \\ 1, & x > 1. \end{matrix}

The set of possible values of

X

contains

[0,1]

, so it is not countable because nontrivial intervals are uncountable. Thus

X

is not a discrete random variable. However,

P (X = 0) = F_{X} (0) - \lim_{x \to 0^{-}} F_{X} (x) = \frac{1}{2},

X

is not continuous.

Random Variable Neither Discrete Nor Continuous

Give an example of a random variable that is neither discrete nor continuous.

Let

X

be a random variable with

X (Ω) = [0,1]

whose distribution function is

F_{X} (x) = {\begin{matrix} 0, & x < 0, \\ \frac{1}{2} + \frac{x}{2}, & 0 \leq x \leq 1, \\ 1, & x > 1. \end{matrix}

The set of possible values of

X

contains

[0,1]

, which is uncountable. Therefore

X (Ω)

is not countable, so

X

is not discrete.

On the other hand,

P (X = 0) = F_{X} (0) - \lim_{x \to 0^{-}} F_{X} (x) = \frac{1}{2},

so it is not true that

P (X = x) = 0

for every

x \in ℝ

. Therefore

X

is not continuous.

This is different from a die roll. If $X$ is the result of a fair six-sided die, then $P (X = 3) = 1 / 6$ , so a single exact outcome can have positive probability. Here $P (X = 3)$ is shorthand for the probability of the event determined by the binary relation $X = 3$ . For a continuous random variable, every exact value has probability $0$ ; probabilities live on intervals or regions rather than individual values.

One way to picture the transition is to imagine a fair die with $N$ equally likely faces. Each single face has probability $1 / N$ . As $N$ gets larger, $1 / N$ gets smaller. If the faces become tiny patches approximating a smooth sphere and the limiting roll is modeled as landing on a point of the sphere, then any single exact point has limiting probability $0$ , even though a region of the sphere can still have positive probability.

Continuous Uniform Distribution

The random variable

U

is said to have a (continuous) uniform distribution on the unit interval

[0,1]

denoted by

U \sim unif [0,1]

iff

P (U \leq u) = u \forall 0 \leq u \leq 1.

Probability of an Interval in the Uniform Distribution

For a continuous uniform random variable,

P (a \leq X \leq b) = b - a

Density Function

Let

X

be a random variable. A function

f : ℝ \to [0, \infty)

is a density function for

X

if, for every interval

I \subseteq ℝ

P (X \in I) = \int_{I} f (t) d t .

For example, the function $f : ℝ \to [0, \infty)$ defined by $f (t) = 1$ for $t \in [0,1]$ and $f (t) = 0$ otherwise is a density function for a random variable uniformly distributed on $[0,1]$ . For any interval $I$ , the integral $\int_{I} f (t) d t$ is the length of the part of $I$ that lies inside $[0,1]$ .

The reason an integral appears is not that we add the probabilities of the individual points. For a discrete random variable, that pointwise method works because the possible values form a countable set. The analogous object is usually called a probability mass function, not a density function. If $X$ is a fair six-sided die roll, then its mass function satisfies $p_{X} (k) = 1 / 6$ for $k \in {1,2,3,4,5,6}$ . For the interval $[2,4]$ , the probability is the sum of the masses at the possible values in that interval: $P (X \in [2,4]) = p_{X} (2) + p_{X} (3) + p_{X} (4) = \frac{3}{6} .$ For a continuous random variable, trying to keep this rule fails: if every point has probability $0$ , then summing point probabilities cannot recover a positive probability for an interval. The repair is to assign probability to the event that the random value lands in a small interval, not to the individual points. A density says that if $I$ is a small interval of length $Δ t$ near $t$ , for example $I = (t - ϵ, t + ϵ)$ , then $P (X \in I) \approx f (t) Δ t .$ These small interval probabilities are compatible with the basic additivity rule for disjoint events, because adjacent small intervals contribute by addition. When the interval is divided into finer and finer pieces, the limiting sum of these interval contributions is the integral. Thus the integral is the continuous replacement for summing masses at possible values, while the value $f (t)$ itself is not the probability of the point $t$ .

Density has Total Mass One

f

is a density function, then

\int_{- \infty}^{\infty} f (t) d t = 1.

Apply the definition of density to the interval

I = ℝ

. Since a random variable is real-valued with probability one,

P (X \in ℝ) = 1

. Therefore the integral over

ℝ

1

Density Determines Distribution Function

X

has density

f

, then its distribution function satisfies

F_{X} (x) = \int_{- \infty}^{x} f (t) d t

for every

x \in ℝ

Apply the definition of density to the interval

I = (- \infty, x]

Density Determines Interval Probabilities

X

has density

f

, then for

a < b

P (a < X \leq b) = \int_{a}^{b} f (x) d x .

Apply the definition of density to the interval

I = (a, b]

CDF Derivative is the Density

Suppose that

X

has density function

f

, and let

F_{X}

be the distribution function of

X

. If

f

is continuous at

x_{0} \in ℝ

, then

F_{X}

is differentiable at

x_{0}

and

F_{X}^{'} (x_{0}) = f (x_{0}) .

Choose

a, b \in ℝ

with

a < x_{0} < b

. For

x \in (a, b)

, density determines interval probabilities gives

F_{X} (x) - F_{X} (a) = P (a < X \leq x) = \int_{a}^{x} f (t) d t .

Hence, on

(a, b)

F_{X} (x) = F_{X} (a) + \int_{a}^{x} f (t) d t .

The constant term

F_{X} (a)

has derivative

0

, and by the Fundamental Theorem of Calculus I, the derivative of

\int_{a}^{x} f (t) d t

x_{0}

f (x_{0})

, because

f

is continuous at

x_{0}

. Therefore

F_{X}^{'} (x_{0}) = f (x_{0})

Expectation of a Continuous Random Variable

X

has density

f

, then

E (X) = \int_{- \infty}^{\infty} x f (x) d x,

whenever the integral is defined.

Expectation of an Interval Indicator

Suppose that

X

has density

f

and

a, b \in ℝ

with

a < b

. Then

E (χ_{(a, b]} (X)) = \int_{- \infty}^{\infty} χ_{(a, b]} (x) f (x) d x .

Let

A : = {ω \in Ω : a < X (ω) \leq b} .

For every

ω \in Ω

χ_{(a, b]} (X (ω)) = {\begin{matrix} 1, & ω \in A, \\ 0, & ω \notin A, \end{matrix}

χ_{(a, b]} (X) = 𝟏_{A}

by the definition of an indicator random variable. Therefore, by expectation of an indicator,

E (χ_{(a, b]} (X)) = E (𝟏_{A}) = P (A) = P (a < X \leq b) .

By density determines interval probabilities,

P (a < X \leq b) = \int_{a}^{b} f (x) d x .

Since

χ_{(a, b]} (x)

1

(a, b]

and

0

off

(a, b]

\int_{a}^{b} f (x) d x = \int_{- \infty}^{\infty} χ_{(a, b]} (x) f (x) d x .

Expectation of a Function of a Continuous Random Variable

X

has density

f

and

g : ℝ \to ℝ

is measurable, then

E (g (X)) = \int_{- \infty}^{\infty} g (x) f (x) d x,

whenever the integral is defined.

The formula holds for interval indicators by expectation of an interval indicator. By finite additivity and linearity of expectation for finite sums, the same formula holds for every non-negative simple function

s

built from finitely many intervals:

E (s (X)) = \int_{- \infty}^{\infty} s (x) f (x) d x .

For a non-negative measurable function

g

, choose an increasing sequence of non-negative simple functions

s_{n}

with

s_{n} (x) \to g (x)

. Applying the simple-function case to each

s_{n}

, then using the monotone convergence theorem on both sides, gives

E (g (X)) = \int_{- \infty}^{\infty} g (x) f (x) d x .

For a general measurable

g

, apply the non-negative case to the positive part

g^{+}

and the negative part

g^{-}

, then subtract, whenever the resulting integral is defined.

Continuous Expectation Respects Scalar Multiplication

Suppose that

X

has density

f_{X}

g : ℝ \to ℝ

is measurable, and

a \in ℝ

. If the expectations are defined, then

E (a g (X)) = a E (g (X)) .

By expectation of a function of a continuous random variable,

E (a g (X)) = \int_{- \infty}^{\infty} a g (x) f_{X} (x) d x .

Pulling the constant

a

out of the integral gives

\int_{- \infty}^{\infty} a g (x) f_{X} (x) d x = a \int_{- \infty}^{\infty} g (x) f_{X} (x) d x .

Applying expectation of a function of a continuous random variable again gives

a \int_{- \infty}^{\infty} g (x) f_{X} (x) d x = a E (g (X)) .

Continuous Expectation is Linear for Finite Sums

Suppose that

X_{1}, \dots, X_{n}

are continuous random variables and

g_{1}, \dots, g_{n} : ℝ \to ℝ

are measurable functions. If the expectations are defined, then

E (\sum_{i = 1}^{n} g_{i} (X_{i})) = \sum_{i = 1}^{n} E (g_{i} (X_{i})) .

This is the finite-sum case of linearity of expectation, applied to the random variables

g_{1} (X_{1}), \dots, g_{n} (X_{n})

Joint Density Function

Random variables

X

and

Y

have joint density

f_{X, Y}

if it is a two-variable analogue of a density function:

f_{X, Y} \geq 0

\int_{- \infty}^{\infty} \int_{- \infty}^{\infty} f_{X, Y} (x, y) d x d y = 1

, and

P ((X, Y) \in A) = \iint_{A} f_{X, Y} (x, y) d x d y

for suitable sets

A \subseteq ℝ^{2}

Continuous Marginal Density Functions

X

and

Y

have joint density

f_{X, Y}

, then their marginal densities are

f_{X} (x) = \int_{- \infty}^{\infty} f_{X, Y} (x, y) d y, f_{Y} (y) = \int_{- \infty}^{\infty} f_{X, Y} (x, y) d x .

Independent Continuous Random Variables

Continuous random variables

X

and

Y

are independent if their joint density factors as

f_{X, Y} (x, y) = f_{X} (x) f_{Y} (y)

for all

x, y

except possibly on a set of area zero.

Conditional Density Function

X

and

Y

have joint density and

f_{X} (x) > 0

, the conditional density of

Y

given

X = x

f_{Y | X} (y | x) = \frac{f_{X, Y} (x, y)}{f_{X} (x)} .

Continuous Conditional Expectation Given a Value

If the conditional density

f_{Y | X} (y | x)

is defined, then

E (Y | X = x) = \int_{- \infty}^{\infty} y f_{Y | X} (y | x) d y,

whenever the integral is defined.

Continuous Convolution Formula

X

and

Y

are independent continuous random variables with densities

f_{X}

and

f_{Y}

, then

X + Y

has density

f_{X + Y} (z) = \int_{- \infty}^{\infty} f_{X} (x) f_{Y} (z - x) d x .

Integrate the joint density

f_{X} (x) f_{Y} (y)

over the region

x + y \leq z

, then differentiate with respect to

z

Change of Variables for Joint Densities

Let

(Y_{1}, Y_{2}) = T (X_{1}, X_{2})

, where

T

is one-to-one and differentiable with differentiable inverse. If

(X_{1}, X_{2})

has joint density

f_{X}

, then

f_{Y} (y_{1}, y_{2}) = f_{X} (T^{- 1} (y_{1}, y_{2})) | \det {[D (T^{- 1}) (y_{1}, y_{2})]}_{E} | .

Density Gives Zero Probability to Points

X

has a density function, then

P (X = a) = 0

for every

a \in ℝ

Apply the definition of density to the interval

I = [a, a]

. The integral of a density over a single point is

0

, so

P (X = a) = 0

Exponential 1 Distribution

Suppose that

Z

is a random variable. We write

Z \sim \exp (1)

iff

Z = - \ln (U)

where

U

is a continuous uniform random variable on

[0,1]

Probability of an Interval in the Exponential Distribution

Suppose that

Z \sim \exp (1)

. Using uniform interval probabilities,

P (s \leq Z \leq t) = e^{- s} - e^{- t}

Observe that the following sets are equal:

\begin{matrix} s \leq Z \leq t & = s \leq - \ln (U) \leq t \\ = - t \leq \ln (U) \leq - s \\ = e^{- t} \leq U \leq e^{- s} \end{matrix}

therefore we know that

P (s \leq Z \leq t) = P (e^{- t} \leq U \leq e^{- s}) = e^{- s} - e^{- t}