Differentiation

directional derivative

Let

f : ℝ^{n} \to ℝ

and let

a, v \in ℝ^{n}

, then the directional derivative of

f

a

along

v

is defined as

\nabla_{v} f (a) : =

\lim_{h \to 0}

\frac{f (a + h v) - f (a)}{h}

Partial Derivative

Let

f : ℝ^{n} \to ℝ

and let

a \in ℝ^{n}

. The partial derivative with respect to

x_{i}

is:

\frac{\partial}{\partial x_{i}} f (a) : =

\nabla_{e_{i}} f (a)

Frozen Function

Suppose that

f : ℝ^{n} \to ℝ

and

a \in ℝ^{n}

then we can define a new function

f_{i, a} : ℝ \to ℝ

f_{i, a} (h) : = f (a + h e_{i})

The frozen function get's it's name because it can be thought of by taking the sample point $a$ and then freezing in all directions except for along the $i$ -th component of the input for $f$

Partial Derivative becomes frozen function derivative

\frac{\partial f}{\partial x_{i}} (a) = \frac{d}{d h} [f_{i, a} (h)]

TODO

Todo

Consider the function

f (x, y, z) = x^{2} + y^{2} + z^{2}

, and the point

(a_{1}, a_{2}, a_{3}) \in ℝ^{3}

then show that

f_{y, a} (h)

equals the function

g (x) = a_{1}^{2} + {(x)}^{2} + a_{3}^{2}

Let's start by looking at the definition of

f_{2, a} (h) = f (a + h e_{2})

now recall that

e_{2} = (0, 1, 0)

, so that

f (a + h e_{2}) = f ((a_{1}, a_{2}, a_{3}) + (0, h, 0)) = f (a_{1}, a_{2} + h, a_{3}) = {(a_{1})}^{2} + {(a_{2} + h)}^{2} + {(a_{3})}^{2}

Todo

compute the partial derivative with resepct to y of

f (x, y, z) = x^{2} + y^{2} + z^{2}

Leaving x and z as constants, and evaluating the derivative as

f (x, y, z) = C + y^{2} + C

, using the sum rule and the power rule, the derivative becomes

\frac{\partial}{\partial y} f (x, y, z) = 2 y

Gradient

The gradient of a function

f : ℝ^{n} \to R

at the point

p

is defined as

\nabla f (p) : = (\frac{\partial f}{\partial} (x_{1}) (p), \frac{\partial f}{\partial} (x_{2}) (p), \dots, \frac{\partial f}{\partial} (x_{n}) (p))

Todo

Suppose we have the fnction

f (x, y) = \sin (x)

, and

(p_{1}, p_{2}) \in ℝ^{2}

show that

\nabla f (p) = (\cos (p_{1}), 0)

By definition

\nabla f (p) = (\frac{\partial f}{\partial x_{1}} (p), \frac{\partial f}{\partial x_{2}} (p))

. Since we know that

\frac{\partial f}{\partial x_{1}} = \frac{d}{d h} (f_{1, p} (h))

, and we know that

f_{1, p} (h) = f (p + h e_{1})

since

h e_{1} = (h, 0)

p = (p_{1}, p_{2})

p + h e_{1} = (p_{1} + h, p_{2})

so then

f (p + h e_{1}) = \sin (p_{1} + h)

, evaluating the limit from the directional derivative, it evaluates to

\cos (p_{1})

we will show that $\frac{\partial f}{\partial x_{2}} (p) = 0$ , we know that $f_{2, p} (h) = f (p + h e_{2}) = \cos (p_{1})$ and thus it is a constant funtion, this implies tha directional derivative is 0

Therefore we can se that $\nabla f (p) = (\cos (p_{1}), 0)$

differentaibility

a function

f : A \to ℝ^{m}

with

A \subseteq ℝ^{n}

is differentiable at the point

a \in ℝ^{n}

if there exists a linear transformation

T : ℝ^{n} \to ℝ^{m}

such that

\lim_{x \to a} \frac{‖ f (x) - (f (a) + T (x - a)) ‖}{‖ x - a ‖} = 0

Quadratic Function is Differentiable at a Point

Let

f : ℝ^{2} \to ℝ

be given by

f (x, y) = x^{2} + y^{2} .

Show that

f

is differentiable at

a = (1, - 1)

Define

T : ℝ^{2} \to ℝ

T (h, k) = 2 h - 2 k .

This is a linear transformation. For

(x, y) \in ℝ^{2}

, write

(h, k) = (x, y) - (1, - 1) = (x - 1, y + 1) .

Then

\begin{matrix} f (x, y) - (f (1, - 1) + T (h, k)) & = x^{2} + y^{2} - (2 + 2 (x - 1) - 2 (y + 1)) \\ = {(x - 1)}^{2} + {(y + 1)}^{2} \\ = {‖ (h, k) ‖}^{2} . \end{matrix}

Therefore

\frac{| f (x, y) - (f (1, - 1) + T ((x, y) - (1, - 1))) |}{‖ (x, y) - (1, - 1) ‖} = ‖ (x, y) - (1, - 1) ‖ .

(x, y) \to (1, - 1)

, the right side tends to

0

. Hence

f

is differentiable at

(1, - 1)

Euclidean Norm is Not Differentiable at the Origin

Let

f : ℝ^{2} \to ℝ

be the Euclidean norm function

f (x, y) = ‖ (x, y) ‖ .

Show that

f

is not differentiable at

(0,0)

Suppose for contradiction that

f

is differentiable at

(0,0)

. Then there exists a linear transformation

T : ℝ^{2} \to ℝ

such that

\lim_{(x, y) \to (0,0)} \frac{| ‖ (x, y) ‖ - T (x, y) |}{‖ (x, y) ‖} = 0.

Taking points

(t, 0)

with

t > 0

, we get

0 = \lim_{t \to 0^{+}} \frac{| t - T (t, 0) |}{t} = | 1 - T (1,0) |,

T (1,0) = 1

. Taking points

(t, 0)

with

t < 0

, we get

0 = \lim_{t \to 0^{-}} \frac{| - t - T (t, 0) |}{- t} = | 1 + T (1,0) |,

T (1,0) = - 1

. This is impossible. Therefore no such linear transformation

T

exists, and

f

is not differentiable at

(0,0)

continuous partials implies differentiability

If all the partial derivatives of a function exist in a neighborhood of a point

x_{0}

and are continuous at the point

x_{0}

, then the function is differentiable at that point

x_{0}

ToDo

Derivative is Unique

Suppose that

f : ℝ^{n} \to ℝ^{m}

is differentiable at

a \in ℝ^{n}

. Then there is a unique linear transformation

T : ℝ^{n} \to ℝ^{m}

such that

\lim_{h \to 0} \frac{| f (a + h) - f (a) - T (h) |}{| h |} = 0.

This linear transformation is denoted

D f (a)

Derivative at a Point

Suppose that

f : ℝ^{n} \to ℝ^{m}

is differentiable at

a \in ℝ^{n}

. The derivative of $f$ at $a$ , denoted

D f (a)

, is the unique linear transformation

T : ℝ^{n} \to ℝ^{m}

from the derivative uniqueness theorem.

Thus $D f (a)$ is not a number in general. It is a linear map that takes a small displacement $h \in ℝ^{n}$ and returns the best linear approximation to the change $f (a + h) - f (a)$ .

Derivative of the Identity Function

Let

{id}_{ℝ^{n}} : ℝ^{n} \to ℝ^{n}

be the identity function. For every

a \in ℝ^{n}

D {id}_{ℝ^{n}} (a) = {id}_{ℝ^{n}} .

Equivalently, for every

v \in ℝ^{n}

D {id}_{ℝ^{n}} (a) (v) = v .

Let

T = {id}_{ℝ^{n}}

. Then

T

is a linear transformation. For every

h \neq 0

\frac{‖ {id}_{ℝ^{n}} (a + h) - {id}_{ℝ^{n}} (a) - T (h) ‖}{‖ h ‖} = \frac{‖ (a + h) - a - h ‖}{‖ h ‖} = 0.

Therefore

T

satisfies the defining limit for the derivative of

{id}_{ℝ^{n}}

a

. By derivative uniqueness,

D {id}_{ℝ^{n}} (a) = {id}_{ℝ^{n}} .

Applying both sides to

v

gives

D {id}_{ℝ^{n}} (a) (v) = v

Derivative of Squared Norm

Let

S : ℝ^{n} \to ℝ, S (x) = x \cdot x .

Show that

S

is differentiable at every

a \in ℝ^{n}

, and that for every

v \in ℝ^{n}

D S (a) (v) = 2 a \cdot v .

Fix

a \in ℝ^{n}

. Define

T : ℝ^{n} \to ℝ, T (h) = 2 a \cdot h .

This is a linear transformation. For

h \in ℝ^{n}

S (a + h) - S (a) - T (h) = (a + h) \cdot (a + h) - a \cdot a - 2 a \cdot h .

Expanding with the distributivity of the dot product and symmetry of the dot product gives

S (a + h) - S (a) - T (h) = h \cdot h .

Therefore

\frac{| S (a + h) - S (a) - T (h) |}{‖ h ‖} = \frac{| h \cdot h |}{‖ h ‖} = ‖ h ‖ \to 0

h \to 0

. Thus

T

satisfies the defining limit for the derivative of

S

a

. By the definition of derivative at a point,

D S (a) = T .

Applying both sides to

v

gives

D S (a) (v) = 2 a \cdot v .

Jacobian Matrix

f : ℝ^{n} \to ℝ^{m}

is differentiable at

a

, then

D f (a)

is a linear transformation

ℝ^{n} \to ℝ^{m}

. Its matrix with respect to the standard bases

E_{n}

and

E_{m}

is called the Jacobian matrix of

f

a

, and is denoted

{[D f (a)]}_{E_{m} \leftarrow E_{n}} .

The notation

f^{'} (a)

is also used for this matrix, but

D f (a)

itself denotes the derivative as a linear transformation.

Jacobian Entries are Partial Derivatives

Let

f : ℝ^{n} \to ℝ^{m}

be differentiable at

a

, and write

f = (f^{1}, \dots, f^{m}) .

Then the

i, j

-entry of the Jacobian matrix is the

j

-th partial derivative of the

i

-th component:

{({[D f (a)]}_{E_{m} \leftarrow E_{n}})}_{i, j} = D_{j} f^{i} (a) .

Equivalently,

{[D f (a)]}_{E_{m} \leftarrow E_{n}} = [\begin{matrix} \frac{\partial f^{1}}{\partial x_{1}} (a) & \frac{\partial f^{1}}{\partial x_{2}} (a) & \dots & \frac{\partial f^{1}}{\partial x_{n}} (a) \\ \frac{\partial f^{2}}{\partial x_{1}} (a) & \frac{\partial f^{2}}{\partial x_{2}} (a) & \dots & \frac{\partial f^{2}}{\partial x_{n}} (a) \\ ⋮ & ⋮ & ⋱ & ⋮ \\ \frac{\partial f^{m}}{\partial x_{1}} (a) & \frac{\partial f^{m}}{\partial x_{2}} (a) & \dots & \frac{\partial f^{m}}{\partial x_{n}} (a) \end{matrix}] .

Let

T = D f (a)

. By the standard matrix theorem for linear transformations, the

j

-th column of

{[T]}_{E_{m} \leftarrow E_{n}}

T (e_{j})

, where

e_{j}

is the

j

-th vector in the standard basis of

ℝ^{n}

. Therefore

{({[D f (a)]}_{E_{m} \leftarrow E_{n}})}_{i, j} = {(T (e_{j}))}_{i} .

Since

T

is the derivative of

f

a

f (a + t e_{j}) = f (a) + T (t e_{j}) + r (t), \frac{‖ r (t) ‖}{| t |} \to 0

t \to 0

. Taking the

i

-th component and using the linearity of

T

\frac{f^{i} (a + t e_{j}) - f^{i} (a)}{t} = {(T (e_{j}))}_{i} + \frac{r^{i} (t)}{t} .

The final term tends to

0

, so the limit on the left exists and equals

{(T (e_{j}))}_{i}

. By the definition of partial derivative, that limit is

D_{j} f^{i} (a)

. Hence

D_{j} f^{i} (a) = {({[D f (a)]}_{E_{m} \leftarrow E_{n}})}_{i, j} .

Since this holds for every

i

and every

j

, the displayed matrix formula follows.

Differentiability Implies Continuity

f : ℝ^{n} \to ℝ^{m}

is differentiable at

a

, then

f

is continuous at

a

Chain Rule for Total Derivatives

Suppose that

f : ℝ^{n} \to ℝ^{m}

is differentiable at

a

, and

g : ℝ^{m} \to ℝ^{p}

is differentiable at

f (a)

. Then the composition

g \circ f

is differentiable at

a

, and

D (g \circ f) (a) = D g (f (a)) \circ D f (a) .

Equivalently,

{[D (g \circ f) (a)]}_{E_{p} \leftarrow E_{n}} = {[D g (f (a))]}_{E_{p} \leftarrow E_{m}} {[D f (a)]}_{E_{m} \leftarrow E_{n}} .

Basic Derivative Rules in Euclidean Space

The following derivative rules hold:

If $f : ℝ^{n} \to ℝ^{m}$ is constant, then $D f (a) = 0$ .
If $f : ℝ^{n} \to ℝ^{m}$ is linear, then $D f (a) = f$ .
If $f : ℝ^{n} \to ℝ^{m}$ , then $f$ is differentiable at $a$ if and only if each component $f^{i}$ is differentiable at $a$ , and $D f (a) = (D f^{1} (a), \dots, D f^{m} (a))$ .
If $f, g : ℝ^{n} \to ℝ$ are differentiable at $a$ , then $D (f + g) (a) = D f (a) + D g (a)$ .
If $f, g : ℝ^{n} \to ℝ$ are differentiable at $a$ , then $D (f g) (a) = g (a) D f (a) + f (a) D g (a)$ .
If $g (a) \neq 0$ , then $D (\frac{f}{g}) (a) = \frac{g (a) D f (a) - f (a) D g (a)}{g {(a)}^{2}} .$

Mixed Partials Commute Under Continuity

D_{i, j} f

and

D_{j, i} f

are continuous on an open set containing

a

, then

D_{i, j} f (a) = D_{j, i} f (a) .

Higher-Order Partial Derivative

A partial derivative of order $k$ is obtained by applying partial derivative operators

k

times in sequence. For example,

D_{i, j} f = D_{i} (D_{j} f)

is a partial derivative of order

2

C^{\infty}

Function

A function is

C^{\infty}

if partial derivatives of all orders exist and are continuous.

Interior Extremum has Zero Partials

Let

A \subseteq ℝ^{n}

. If

f : A \to ℝ

has a maximum or minimum at a point

a

in the interior of

A

, and

D_{i} f (a)

exists, then

D_{i} f (a) = 0.

Matrix Norm

A

is an

m \times n

matrix, its norm is

| A | = \sqrt{\sum_{i = 1}^{m} \sum_{j = 1}^{n} a_{i, j}^{2}} .

Mean Value Inequality in Euclidean Space

Suppose that

A \subseteq ℝ^{n}

is open,

f : A \to ℝ^{m}

is differentiable, and the line segment from

a

b

lies in

A

. If

| D f (x) | \leq M

for every point

x

on that segment, then

| f (b) - f (a) | \leq M | b - a | .

Continuously Differentiable iff Continuous Partials

A function

f : ℝ^{n} \to ℝ^{m}

is continuously differentiable on an open set

A

if and only if all partial derivatives

D_{j} f^{i}

exist and are continuous on

A

. In this case the Jacobian matrix is

{[D f (x)]}_{E_{m} \leftarrow E_{n}} = (D_{j} f^{i} (x)) .

In this statement, continuously differentiable means that

f

is differentiable at every point of

A

, and that the derivative map

x \mapsto D f (x)

is continuous when

D f (x)

is represented by its Jacobian matrix.

First suppose $f$ is continuously differentiable on $A$ . Then $f$ is differentiable at every $x \in A$ , so $D f (x)$ exists. The $j$ -th partial derivative of the $i$ -th component function is the $i, j$ -entry of the Jacobian matrix: $D_{j} f^{i} (x) = {({[D f (x)]}_{E_{m} \leftarrow E_{n}})}_{i, j} .$ Since the matrix-valued function $x \mapsto {[D f (x)]}_{E_{m} \leftarrow E_{n}}$ is continuous, each entry function $x \mapsto D_{j} f^{i} (x)$ is continuous. Thus all partial derivatives exist and are continuous on $A$ .

Conversely, suppose all partial derivatives $D_{j} f^{i}$ exist and are continuous on $A$ . Fix $a \in A$ . Since $A$ is open, choose a small rectangle around $a$ contained in $A$ . Define a linear transformation $T_{a} : ℝ^{n} \to ℝ^{m}$ by $T_{a} h = (\sum_{j = 1}^{n} D_{j} f^{1} (a) h_{j}, \dots, \sum_{j = 1}^{n} D_{j} f^{m} (a) h_{j}) .$ We show that $T_{a}$ is the derivative of $f$ at $a$ .

Write $h = (h_{1}, \dots, h_{n})$ , and for $0 \leq j \leq n$ , set $z_{j} = a + (h_{1}, \dots, h_{j}, 0, \dots, 0) .$ For each component $f^{i}$ , telescope the change from $a$ to $a + h$ : $f^{i} (a + h) - f^{i} (a) = \sum_{j = 1}^{n} (f^{i} (z_{j}) - f^{i} (z_{j - 1})) .$ Each summand changes only the $j$ -th coordinate. By the one-variable mean value theorem applied along that coordinate segment, if $h_{j} \neq 0$ , there is a point $ξ_{j}$ on the segment from $z_{j - 1}$ to $z_{j}$ such that $f^{i} (z_{j}) - f^{i} (z_{j - 1}) = D_{j} f^{i} (ξ_{j}) h_{j} .$ If $h_{j} = 0$ , the same formula holds with any point $ξ_{j}$ on the segment.

Therefore $f^{i} (a + h) - f^{i} (a) - \sum_{j = 1}^{n} D_{j} f^{i} (a) h_{j} = \sum_{j = 1}^{n} (D_{j} f^{i} (ξ_{j}) - D_{j} f^{i} (a)) h_{j} .$ Since each $D_{j} f^{i}$ is continuous at $a$ , the differences $D_{j} f^{i} (ξ_{j}) - D_{j} f^{i} (a)$ go to $0$ as $h \to 0$ . Hence $\frac{| f (a + h) - f (a) - T_{a} h |}{| h |} \to 0.$ By the definition of differentiability, $f$ is differentiable at $a$ , with $D f (a) = T_{a}$ .

Since $a \in A$ was arbitrary, $f$ is differentiable on $A$ . The matrix of $T_{a}$ is exactly $(D_{j} f^{i} (a)),$ so ${[D f (a)]}_{E_{m} \leftarrow E_{n}} = (D_{j} f^{i} (a)) .$ The entries of this matrix are continuous functions of $a$ , so the Jacobian matrix varies continuously on $A$ . Thus $f$ is continuously differentiable on $A$ .

Projection Functions are Continuously Differentiable

Each projection function

π^{i} : ℝ^{n} \to ℝ

is continuously differentiable.

For the

i

-th projection function,

D_{j} π^{i} (x) = {\begin{matrix} 1 & j = i, \\ 0 & j \neq i . \end{matrix}

These partial derivatives exist and are constant, hence continuous. Therefore

π^{i}

is continuously differentiable by the continuous partials characterization.

Componentwise Continuously Differentiable Map

Let

A \subseteq ℝ^{n}

be open. If

f^{1}, \dots, f^{m} : A \to ℝ

are continuously differentiable, then the function

f : A \to ℝ^{m}

defined by

f (x) = (f^{1} (x), \dots, f^{m} (x))

is continuously differentiable.

By the definition of component functions, the component functions of

f

are

f^{1}, \dots, f^{m}

. Each

f^{i}

is continuously differentiable, so all partial derivatives

D_{j} f^{i}

exist and are continuous on

A

. Therefore

f

is continuously differentiable by the continuous partials characterization.

Derivative of the Vector Scalar Division Map

Let

Q : ℝ^{m} \times (ℝ ∖ {0}) \to ℝ^{m}, Q (y, t) = \frac{y}{t} .

y \in ℝ^{m}

t \neq 0

u \in ℝ^{m}

, and

s \in ℝ

, then

D Q (y, t) (u, s) = \frac{u}{t} - \frac{s}{t^{2}} y .

Write

y = (y_{1}, \dots, y_{m})

. The component functions of

Q

are

Q^{i} (y, t) = \frac{y_{i}}{t}

for

1 \leq i \leq m

. For

1 \leq j \leq m

\frac{\partial Q^{i}}{\partial y_{j}} (y, t) = {\begin{matrix} \frac{1}{t}, & i = j, \\ 0, & i \neq j, \end{matrix}

and

\frac{\partial Q^{i}}{\partial t} (y, t) = - \frac{y_{i}}{t^{2}} .

These partial derivatives are continuous wherever

t \neq 0

. Therefore, by the continuous partials characterization,

Q

is differentiable at

(y, t)

By the Jacobian entries are partial derivatives corollary, the Jacobian matrix of $Q$ at $(y, t)$ , with input coordinates ordered as $y_{1}, \dots, y_{m}, t$ , is ${[D Q (y, t)]}_{E_{m} \leftarrow E_{m + 1}} = [\begin{matrix} \frac{1}{t} & 0 & \dots & 0 & - \frac{y_{1}}{t^{2}} \\ 0 & \frac{1}{t} & \dots & 0 & - \frac{y_{2}}{t^{2}} \\ ⋮ & ⋮ & ⋱ & ⋮ & ⋮ \\ 0 & 0 & \dots & \frac{1}{t} & - \frac{y_{m}}{t^{2}} \end{matrix}] .$

Applying this matrix to the displacement

(u, s) = (u_{1}, \dots, u_{m}, s)

gives

D Q (y, t) (u, s) = (\frac{u_{1}}{t} - \frac{s y_{1}}{t^{2}}, \dots, \frac{u_{m}}{t} - \frac{s y_{m}}{t^{2}}) = \frac{u}{t} - \frac{s}{t^{2}} y .

Inverse Function Theorem

Suppose that

f : ℝ^{n} \to ℝ^{n}

is continuously differentiable in an open set containing

a

, and that the determinant of the Jacobian matrix is nonzero:

\det {[D f (a)]}_{E} \neq 0.

Then there are open sets

U

and

V

with

a \in U

f (a) \in V

, such that

f : U \to V

is one-to-one, onto, and has a continuously differentiable inverse. Moreover,

{[D (f^{- 1}) (y)]}_{E} = {({[D f (f^{- 1} (y))]}_{E})}^{- 1} .

f

C^{\infty}

, then

f^{- 1}

C^{\infty}

Let

A = {[D f (a)]}_{E}

. Since

\det A \neq 0

A

is an invertible matrix. Composing

f

with the affine change of variables

x \mapsto A^{- 1} (f (x) - f (a)) + a

reduces the proof to the case

f (a) = a

and

{[D f (a)]}_{E} = I_{n}

. After proving the theorem in this normalized case, composing back with the affine maps gives the original statement.

In the normalized case, continuity of the Jacobian matrix gives a closed rectangle $R$ with $a$ in its interior such that ${[D f (x)]}_{E}$ stays close to $I_{n}$ on $R$ . Applying the mean value inequality to $x \mapsto f (x) - x$ , after shrinking $R$ if needed, gives $| (f (x) - x) - (f (y) - y) | \leq \frac{1}{2} | x - y |$ for all $x, y \in R$ . Therefore $| f (x) - f (y) | \geq \frac{1}{2} | x - y |,$ so $f$ is one-to-one on $R$ .

The boundary of $R$ is compact, and $f (a)$ is not in the image of that boundary. Hence there is an open ball $V$ around $f (a)$ such that every $z \in V$ is closer to $f (a)$ than to any value $f (x)$ with $x$ on the boundary of $R$ .

Fix $z \in V$ , and define $Φ_{z} (x) = {| z - f (x) |}^{2}$ on $R$ . Since $R$ is a compact closed rectangle and $Φ_{z}$ is continuous, the extreme value theorem gives a point where $Φ_{z}$ is minimized. The choice of $V$ prevents this minimum from occurring on the boundary, so the minimum occurs at an interior point $x$ .

By the interior extremum theorem, all partial derivatives of $Φ_{z}$ vanish at $x$ . Written in matrix form, this says ${(z - f (x))}^{T} {[D f (x)]}_{E} = 0.$ Since ${[D f (x)]}_{E}$ is close to $I_{n}$ , it is invertible. Therefore $z - f (x) = 0$ , so $z = f (x)$ . This proves that every $z \in V$ has a preimage in the interior of $R$ . Together with injectivity, this gives a local inverse $f^{- 1} : V \to U$ , where $U = f^{- 1} (V) \cap int (R)$ .

The estimate $| f (x) - f (y) | \geq \frac{1}{2} | x - y |$ implies $| f^{- 1} (z_{1}) - f^{- 1} (z_{2}) | \leq 2 | z_{1} - z_{2} |,$ so $f^{- 1}$ is continuous.

Let $z = f (x) \in V$ . Since $f$ is differentiable at $x$ , $f (x + h) = f (x) + {[D f (x)]}_{E} h + r (h), \frac{| r (h) |}{| h |} \to 0.$ Using the continuity estimate for $f^{- 1}$ , this remainder identity implies ${[D (f^{- 1}) (z)]}_{E} = {({[D f (x)]}_{E})}^{- 1} = {({[D f (f^{- 1} (z))]}_{E})}^{- 1} .$ Since the Jacobian matrix of $f$ is continuous and matrix inversion is continuous on invertible matrices, $f^{- 1}$ is continuously differentiable. If $f$ is $C^{\infty}$ , repeated differentiation of $f (f^{- 1} (z)) = z$ shows inductively that $f^{- 1}$ is $C^{\infty}$ .

Implicit Function Theorem

Suppose that

F : ℝ^{n + m} \to ℝ^{m}

is continuously differentiable near

(a, b)

F (a, b) = 0

, and the

m \times m

matrix of partial derivatives of

F

with respect to the last

m

variables is invertible at

(a, b)

. Then near

(a, b)

, the equation

F (x, y) = 0

uniquely determines

y

as a continuously differentiable function

y = g (x)

. If

F

C^{\infty}

, then

g

C^{\infty}

Let

B = {[\frac{\partial F^{i}}{\partial y_{j}} (a, b)]}_{1 \leq i, j \leq m} .

By hypothesis,

B

is an invertible matrix. Define

H : ℝ^{n + m} \to ℝ^{n + m}, H (x, y) = (x, F (x, y)) .

Since

F

is continuously differentiable near

(a, b)

, there is an open set

U

containing

(a, b)

on which

F

is continuously differentiable. On

U

, the first

n

component functions of

H

are projection functions, and the last

m

component functions are the component functions of

F

. Hence

H

is continuously differentiable on

U

by the componentwise continuously differentiable map theorem.

The derivative matrix of $H$ at $(a, b)$ , written in the $x$ -variables followed by the $y$ -variables, has block form $H^{'} (a, b) = [\begin{matrix} I_{n} & 0 \\ C & B \end{matrix}],$ where $C$ is the matrix of partial derivatives of $F$ with respect to the $x$ -variables. This block matrix is invertible, with inverse $[\begin{matrix} I_{n} & 0 \\ - B^{- 1} C & B^{- 1} \end{matrix}] .$ Indeed, by block matrix multiplication, $[\begin{matrix} I_{n} & 0 \\ C & B \end{matrix}] [\begin{matrix} I_{n} & 0 \\ - B^{- 1} C & B^{- 1} \end{matrix}] = [\begin{matrix} I_{n} & 0 \\ C - B B^{- 1} C & B B^{- 1} \end{matrix}] = [\begin{matrix} I_{n} & 0 \\ 0 & I_{m} \end{matrix}],$ and the product in the opposite order is also the identity matrix.

By the inverse function theorem, after restricting to open neighborhoods, $H$ has a continuously differentiable inverse. Write this inverse as $H^{- 1} (s, t) = (s, k (s, t)) .$ The first component must be $s$ , because the first component of $H (x, y)$ is $x$ .

Define $g (s) = k (s, 0) .$ Then $H (s, g (s)) = (s, 0),$ so $F (s, g (s)) = 0$ .

Conversely, if $F (s, y) = 0$ for $(s, y)$ in the chosen neighborhood, then $H (s, y) = (s, 0) .$ Since $H$ is one-to-one on that neighborhood, $(s, y) = H^{- 1} (s, 0) = (s, g (s))$ . Hence $y = g (s)$ , proving uniqueness.

Since $g$ is obtained from the continuously differentiable inverse $H^{- 1}$ , it is continuously differentiable. If $F$ is $C^{\infty}$ , then $H$ is $C^{\infty}$ , so $H^{- 1}$ is $C^{\infty}$ by the smooth part of the inverse function theorem. Therefore $g$ is $C^{\infty}$ .

Lagrange Multiplier Condition

g : A \to ℝ^{p}

has full rank

p

g^{- 1} (0)

, and

f : A \to ℝ

has a constrained extremum at

a \in g^{- 1} (0)

, then there are scalars

λ_{1}, \dots, λ_{p}

such that

d f (a) = \sum_{i = 1}^{p} λ_{i} d g^{i} (a) .