Expected Value

Suppose that

X

is a discrete random variable. Then we define

E (X) : = \sum x P (X = x)

Expectation of a Function of a Discrete Random Variable

X

is a discrete random variable with probability mass function

p_{X}

and

g : ℝ \to ℝ

, then

E (g (X)) = \sum_{x} g (x) p_{X} (x),

whenever the sum is defined.

Moment

The

k

-th moment of a random variable

X

is defined using expected value as

E (X^{k})

, whenever this expectation exists.

Squared Expectation

Suppose

X

is a random variable such that

P (X = - 2) = 0.2, P (X = 1) = 0.5, P (X = 2) = 0.3

and

P (X = a) = 0

for all other values. Compute

E (X^{2})

and show it's not equal to

E {(X)}^{2}

Recall that $X^{2}$ is simply the composition of the function $X$ and the function $\cdot^{2}$ , denote $Y : = X^{2}$ so that $Y$ may only take on values $1$ and $4$ and also $P (Y = 4) = P ((X = 2) \cup (X = - 2)) = P (X = 2) + P (X = - 2) = 0.2 + 0.3 = 0.5$ and for all other values $P (Y = a) = 0$ $E (X^{2}) = E (Y) = 4 \cdot 0.5 + 1 \cdot 0.5 = 2 + .5 = 2.5$ on the other hand $E (X) = - 2 \cdot 0.2 + 1 \cdot 0.5 + 2 \cdot 0.3 = 0.7$ so that $E {(X)}^{2} = 0.49$ so $E {(X)}^{2} \neq E (X^{2})$

Expectation is Linear

Suppose that

X

is a random variable and that

a, b \in ℝ

. Then

a X + b

is also a random variable and

E (a X + b) = a E (X) + b

Expectation is Linear for Finite Sums

X_{1}, \dots, X_{n}

have finite expectations and

a_{1}, \dots, a_{n} \in ℝ

, then

E (\sum_{k = 1}^{n} a_{k} X_{k}) = \sum_{k = 1}^{n} a_{k} E (X_{k}) .

Let

Y : = \sum_{k = 1}^{n} a_{k} X_{k} .

By expectation as an integral,

E (Y) = \int_{Ω} (\sum_{k = 1}^{n} a_{k} X_{k}) d P .

We first handle two summands. If

U

and

V

have finite expectations, then by additivity of integration,

E (U + V) = \int_{Ω} (U + V) d P = \int_{Ω} U d P + \int_{Ω} V d P = E (U) + E (V) .

Also, for

a \in ℝ

E (a U) = \int_{Ω} a U d P = a \int_{Ω} U d P = a E (U),

because multiplying a function by a constant multiplies its integral by that constant. Applying these two facts repeatedly to the finite sum gives

E (\sum_{k = 1}^{n} a_{k} X_{k}) = \sum_{k = 1}^{n} E (a_{k} X_{k}) = \sum_{k = 1}^{n} a_{k} E (X_{k}) .

Variance

Let

X

be a random variable. The variance of

X

Var (X) : = E ({(X - E (X))}^{2}),

whenever this expected value is defined.

Variance as Squares of Expectation

From the definition of variance,

Var (X) = E (X^{2}) - E {(X)}^{2}

Variance is Not Linear

By the definition of variance,

Var (a X + b) = a^{2} Var (X)

Covariance

X

and

Y

have finite second moments, their covariance is

Cov (X, Y) : = E ((X - E (X)) (Y - E (Y))) .

Covariance as Expectation Product

X

and

Y

have finite second moments, then by covariance,

Cov (X, Y) = E (X Y) - E (X) E (Y) .

Expand the product

(X - E (X)) (Y - E (Y))

and use linearity of expectation.

Uncorrelated Random Variables

Random variables

X

and

Y

with finite second moments are uncorrelated if their covariance is zero, equivalently if

E (X Y) = E (X) E (Y),

equivalently

Cov (X, Y) = 0

Independent Random Variables Factor Expectation

X

and

Y

are independent and the expectations exist, then

E (X Y) = E (X) E (Y) .

In the discrete case,

E (X Y) = \sum_{x, y} x y P ({X = x} \cap {Y = y}) = \sum_{x, y} x y P (X = x) P (Y = y),

which factors into the product of the two expectations. The continuous proof is the analogous factorization of the product density or product measure.

Correlation Coefficient

Var (X) > 0

and

Var (Y) > 0

, the correlation coefficient of

X

and

Y

is the normalized covariance

ρ (X, Y) : = \frac{Cov (X, Y)}{\sqrt{Var (X) Var (Y)}} .

Cauchy Schwarz Inequality for Random Variables

X

and

Y

have finite second moments, then

{| E (X Y) |}^{2} \leq E (X^{2}) E (Y^{2}) .

E (Y^{2}) = 0

, then

Y = 0

almost surely and the result is immediate. Otherwise the quadratic

E ({(X - t Y)}^{2}) \geq 0

has non-positive discriminant as a function of

t

Correlation Coefficient is Bounded

If the correlation coefficient

ρ (X, Y)

is defined, then

| ρ (X, Y) | \leq 1.

Apply Cauchy Schwarz to the centered variables

X - E (X)

and

Y - E (Y)

Standard Deviation

Given a random variable

X

, we define the standard deviation of that random variable using variance:

σ (X) : = \sqrt{Var (X)}