1 Notation and Preliminary Concepts
In the following, unless otherwise stated, f:A\subseteq\mathbb{R}^n\to\mathbb{R}^m. The Euclidean norm is indicated with \|\cdot\|.
For x_0\in\mathbb{R}^n:
B_r(x_0)=\{x:\|x-x_0\|<r\}, \qquad \overline B_r(x_0)=\{x:\|x-x_0\|\le r\}.
A point x_0 is accumulation for A if every B_r(x_0) contains points of A different from x_0.
1.1 Interior, Closure, and Boundary
\operatorname{int}(A)=\{x:\exists r>0,\ B_r(x)\subset A\}, \operatorname{cl}(A)=\text{smallest closed set containing }A, \partial A=\operatorname{cl}(A)\setminus\operatorname{int}(A).
2 Sequences and Limits
2.1 Convergence of Sequences
A sequence x_k\in\mathbb{R}^n converging x if \forall\varepsilon>0\ \exists N:\ k\ge N\Rightarrow \|x_k-x\|<\varepsilon. It is written x_k\to x. In \mathbb{R}^n:
x_k\to x \iff (x_k)_i\to x_i\quad\forall i.
2.2 Cauchy Criterion
In \mathbb{R}^n:
x_k\text{ converges} \iff \forall\varepsilon>0\ \exists N:\ p,q\ge N\Rightarrow\|x_p-x_q\|<\varepsilon.
2.3 Bolzano–Weierstrass Theorem
Every bounded sequence in \mathbb{R}^n admits a converging subsequence.
2.4 Limit of a Function
We say \lim_{x\to x_0}f(x)=L if \forall\varepsilon>0\ \exists\delta>0: 0<\|x-x_0\|<\delta \Rightarrow \|f(x)-L\|<\varepsilon.
2.5 Sequential Characterization
x_0 is an accumulation point of A:
\lim_{x\to x_0}f(x)=L \iff f(x_k)\to L for every sequence x_k\in A\setminus\{x_0\} with x_k\to x_0.
2.6 Uniqueness of the Limit
If the limit exists, it is unique.
2.7 Algebra of Limits
f\to a and g\to b:
f+g\to a+b, \qquad fg\to ab, \frac{f}{g}\to\frac ab \quad\text{if }b\neq0\text{ and }g\neq0\text{ locally}. For vector functions the limit is component by component.
2.8 Iterated Limits and Joint Limits
In general:
\lim_{x\to x_0}\lim_{y\to y_0}f(x,y) and \lim_{(x,y)\to(x_0,y_0)}f(x,y) are different concepts.
If the joint limit exists, every limit along an admissible curve or sequence coincide with it. Limits along individual matching curves are not sufficient, in general, to ensure the existence of the joint limit.
3 Continuity
f is continuous x_0\in A if \lim_{x\to x_0}f(x)=f(x_0). Equivalently: \forall\varepsilon>0\ \exists\delta>0: \|x-x_0\|<\delta \Rightarrow \|f(x)-f(x_0)\|<\varepsilon.
3.1 Sequential Characterization
f\text{ continuous at }x_0 \iff x_k\to x_0\Rightarrow f(x_k)\to f(x_0).
3.2 Operations on Continuous Functions
Sum, product, composition and quotient with nonzero denominator of continuous functions are continuous.
3.3 Uniform Continuity
f:A\to\mathbb{R}^m is uniformly continuous if \forall\varepsilon>0\ \exists\delta>0:\ \|x-y\|<\delta \Rightarrow \|f(x)-f(y)\|<\varepsilon for each x,y\in A.
Conceptual difference:
continuity: \delta can depend on the point;
uniform continuity: \delta depends only on \varepsilon.
3.4 Heine–Cantor Theorem
K\subset\mathbb{R}^n is compact and f:K\to\mathbb{R}^m it is continuous, then f is uniformly continuous.
3.5 Weierstrass Theorem
K\subset\mathbb{R}^n is compact and f:K\to\mathbb{R} it is continuous, then f is limited and attains its maximum and minimum.
4 Lipschitz and Hölder Continuity
4.1 Lipschitz Continuity
f:A\to\mathbb{R}^m is Lipschitz if it exists L\ge0 such that \|f(x)-f(y)\|\le L\|x-y\| \qquad\forall x,y\in A. You can define \operatorname{Lip}(f;A)= \sup_{x\neq y} \frac{\|f(x)-f(y)\|}{\|x-y\|}.
Each Lipschitz function is uniformly continuous.
4.2 Local Lipschitz Continuity
f is locally Lipschitz if every point has a neighborhood on which f is Lipschitz.
4.3 A Sufficient Derivative Condition
A It is convex, f\in C^1(A) and \sup_{x\in A}\|Df(x)\|\le L, then \|f(x)-f(y)\|\le L\|x-y\|.
In a variable:
|f'(x)|\le L \quad\Rightarrow\quad |f(x)-f(y)|\le L|x-y|.
4.4 Hölder Continuity
For 0<\alpha\le1, f is Hölder of exponent \alpha if \|f(x)-f(y)\| \le C\|x-y\|^\alpha. The case \alpha=1 coincides with Lipschitz.
5 Derivatives in One Variable
For f:I\subset\mathbb{R}\to\mathbb{R}^m:
f'(x_0)= \lim_{h\to0} \frac{f(x_0+h)-f(x_0)}{h}. Equivalently:
f(x_0+h)=f(x_0)+f'(x_0)h+o(h).
5.1 Differentiability Implies Continuity
f\text{ differentiable at }x_0 \Longrightarrow f\text{ continuous at }x_0. The converse is false.
5.2 Differentiation Rules
(f+g)'=f'+g', (fg)'=f'g+fg', \left(\frac fg\right)'= \frac{f'g-fg'}{g^2}, (f\circ g)'(x)=f'(g(x))g'(x).
5.3 Rolle’s Theorem
f\in C([a,b]), differentiable on (a,b) and f(a)=f(b), then there exists \xi\in(a,b) with f'(\xi)=0.
5.4 Lagrange’s Mean Value Theorem
f\in C([a,b]) and is differentiable on (a,b), then there exists \xi\in(a,b) such that f(b)-f(a)=f'(\xi)(b-a).
5.5 Cauchy’s Mean Value Theorem
Under the usual assumptions:
[f(b)-f(a)]g'(\xi) = [g(b)-g(a)]f'(\xi).
6 Partial and Directional Derivatives
6.1 Partial Derivative
For f:\mathbb{R}^n\to\mathbb{R}^m:
\frac{\partial f}{\partial x_i}(x) = \lim_{h\to0} \frac{f(x+he_i)-f(x)}{h}.
6.2 Directional Derivative
In the direction v:
D_vf(x)= \lim_{t\to0} \frac{f(x+tv)-f(x)}{t}.
For f:\mathbb{R}^n\to\mathbb{R} differentiable: D_vf(x)=\nabla f(x)^Tv.
\|v\|=1: |D_vf(x)|\le\|\nabla f(x)\|, with maximum in the direction of gradient.
6.3 Existence of Directional Derivatives
The existence of all directional derivatives at one point does not generally imply differentiability at the point.
7 Differentiability in Several Variables
7.1 Fréchet Derivative
f:\mathbb{R}^n\to\mathbb{R}^m is differentiable in x if there is a linear application Df(x):\mathbb{R}^n\to\mathbb{R}^m such that f(x+h)=f(x)+Df(x)h+o(\|h\|). Equivalently:
\lim_{h\to0} \frac{\|f(x+h)-f(x)-Df(x)h\|}{\|h\|}=0.
Df(x) is unique.
7.2 Jacobian
For f=(f_1,\dots,f_m):
Df(x)=J_f(x)= \begin{pmatrix} \partial_1f_1&\cdots&\partial_nf_1\\ \vdots&&\vdots\\ \partial_1f_m&\cdots&\partial_nf_m \end{pmatrix}.
For m=1:
Df(x)=\nabla f(x)^T.
7.3 Differentiability and Partial Derivatives
f is differentiable, then all partial derivatives exist.
The converse is false in general.
Classic enough condition: \partial_i f_j\text{ exist in a neighborhood and are continuous at }x_0 \Longrightarrow f\text{ differentiable at }x_0. In particular:
f\in C^1 \Longrightarrow f\text{ differentiable}.
7.4 Differentiability Implies Continuity
f\text{ differentiable at }x_0 \Longrightarrow f\text{ continuous at }x_0.
8 Gâteaux versus Fréchet Differentiability
8.1 Gâteaux Derivative
The derivative of Gâteaux f in x in the direction h It is D_Gf(x)[h] = \lim_{t\to0} \frac{f(x+th)-f(x)}{t}.
If it exists for each h and is linear in h, we talk about Gâteaux's differentiability.
8.2 Relationships
\text{Fréchet differentiable} \Longrightarrow \text{Gâteaux differentiable}. In finite dimensions, under appropriate assumptions of continuity of the derivative, the differentiability of Gâteaux can imply that of Fréchet.
The only existence of directional derivatives does not guarantee neither linearity nor differentiability of Fréchet.
9 Chain Rule and Derivatives of Compositions
f:\mathbb{R}^n\to\mathbb{R}^m, \qquad g:\mathbb{R}^m\to\mathbb{R}^p are differentiable: D(g\circ f)(x)=Dg(f(x))Df(x).
For g:\mathbb{R}^m\to\mathbb{R}:
\nabla(g\circ f)(x) = Df(x)^T\nabla g(f(x)).
x=x(t):
\frac{d}{dt}f(x(t)) = Df(x(t))x'(t).
For f:\mathbb{R}^n\to\mathbb{R}:
\frac{d}{dt}f(x(t)) = \nabla f(x(t))^Tx'(t).
10 Second Derivatives and the Hessian
10.1 Hessian
For f:\mathbb{R}^n\to\mathbb{R} twice differentiable: \operatorname{Hess}f(x)=\nabla^2f(x)= \left[ \frac{\partial^2f}{\partial x_i\partial x_j} \right]_{i,j}.
10.2 Schwarz–Clairaut Theorem
If the second mixed derivatives are continuous in a neighborhood: \frac{\partial^2f}{\partial x_i\partial x_j} = \frac{\partial^2f}{\partial x_j\partial x_i}. So Hessian is symmetric.
10.3 Second Directional Derivative
f\in C^2:
D_v^2f(x)=v^T\operatorname{Hess}f(x)v.
10.4 Second Derivative of a Composition along a Curve
x=x(t) and f:\mathbb{R}^n\to\mathbb{R}:
\frac{d^2}{dt^2}f(x(t)) = x'(t)^T\operatorname{Hess}f(x(t))x'(t) +\nabla f(x(t))^Tx''(t).
11 Regularity Classes
C^0=\text{continuous functions}, C^1=\text{functions with continuous first derivatives}, C^k=\text{functions with continuous derivatives up to order }k\text{ continuous}, C^\infty=\bigcap_{k\ge0}C^k.
An analytical function locally admits a development in series of powers converging to function.
In general:
\text{analytic}\Longrightarrow C^\infty, but vice versa is false.
12 Asymptotic Notation: Little o and Big O
12.1 Little o
For x\to x_0:
f(x)=o(g(x)) if \frac{f(x)}{g(x)}\to0 when the report is defined locally.
Equivalently:
f=o(g) \iff \forall\varepsilon>0\ \exists\delta>0: |f(x)|\le\varepsilon|g(x)| for x sufficiently close to x_0.
12.2 Big O
f(x)=O(g(x)) if they exist C>0 and a neighborhood of x_0 such as |f(x)|\le C|g(x)|.
12.3 Relationship between o and O
f=o(g)\Longrightarrow f=O(g). The converse is false.
12.4 Asymptotic Equivalence
f(x)\sim g(x) \quad\Longleftrightarrow\quad \frac{f(x)}{g(x)}\to1. Equivalently:
f=g+o(g).
12.5 \Theta Notation
f=\Theta(g) if c|g(x)|\le|f(x)|\le C|g(x)| locally, for some c,C>0.
12.6 Algebraic Rules
f=o(g) and h=O(k):
fh=o(gk).
f_1=o(g)andf_2=o(g): f_1+f_2=o(g).
f=O(g)andh=O(k):
fh=O(gk).
f=o(1): 1+f=1+o(1).
12.7 Vector-Valued Version
For f:\mathbb{R}^n\to\mathbb{R}^m:
f(h)=o(\|h\|^k) means \frac{\|f(h)\|}{\|h\|^k}\to0. Similarly:
f(h)=O(\|h\|^k) \iff \|f(h)\|\le C\|h\|^k.
13 Elementary Asymptotic Expansions
For x\to0:
e^x=1+x+\frac{x^2}{2}+\frac{x^3}{6}+O(x^4), \log(1+x)=x-\frac{x^2}{2}+\frac{x^3}{3}+O(x^4), \sin x=x-\frac{x^3}{6}+O(x^5), \cos x=1-\frac{x^2}{2}+\frac{x^4}{24}+O(x^6), (1+x)^\alpha =1+\alpha x+\frac{\alpha(\alpha-1)}{2}x^2+O(x^3). In particular: \sin x\sim x, \qquad \tan x\sim x, \qquad 1-\cos x\sim\frac{x^2}{2}, e^x-1\sim x, \qquad \log(1+x)\sim x.
14 Taylor’s Theorem in One Variable
Both f sufficiently regular near to x_0. Taylor's polynomial of order p It is T_p(x;x_0) = \sum_{k=0}^p \frac{f^{(k)}(x_0)}{k!}(x-x_0)^k.
14.1 Peano Remainder
f It is p times differentiable in an appropriate sense:
f(x) = \sum_{k=0}^p \frac{f^{(k)}(x_0)}{k!}(x-x_0)^k +o(|x-x_0|^p).
14.2 Lagrange Remainder
f\in C^{p+1} between x_0 and x:
f(x)=T_p(x;x_0) + \frac{f^{(p+1)}(\xi)}{(p+1)!}(x-x_0)^{p+1} for some \xi between x_0 and x.
14.3 Integral Remainder
f^{(p+1)} is continuous:
f(x)=T_p(x;x_0) + \frac1{p!} \int_{x_0}^x f^{(p+1)}(t)(x-t)^p\,dt.
14.4 Big-O Form
f^{(p+1)} is locally limited:
f(x)=T_p(x;x_0)+O(|x-x_0|^{p+1}).
15 Taylor’s Theorem in Several Variables
Both f:\mathbb{R}^n\to\mathbb{R}. With h=x-x_0:
15.1 First Order
f is differentiable:
f(x_0+h) = f(x_0)+\nabla f(x_0)^Th+o(\|h\|).
Df is locally Lipschitz:
f(x_0+h) = f(x_0)+\nabla f(x_0)^Th+O(\|h\|^2).
15.2 Second Order
f is twice differentiable with sufficient regularity:
f(x_0+h) = f(x_0) +\nabla f(x_0)^Th +\frac12h^T\operatorname{Hess}f(x_0)h +o(\|h\|^2).
\operatorname{Hess}f is locally Lipschitz:
f(x_0+h) = f(x_0) +\nabla f(x_0)^Th +\frac12h^T\operatorname{Hess}f(x_0)h +O(\|h\|^3).
15.3 Form along a Line Segment
Defining \phi(t)=f(x_0+th), Taylor multidimensional is reduced to Taylor 1D \phi. In particular:
\phi'(t)=\nabla f(x_0+th)^Th, \phi''(t)=h^T\operatorname{Hess}f(x_0+th)h.
15.4 Second-Order Lagrange Remainder
f\in C^2 the segment between x_0 and x_0+h, exist \theta\in(0,1) that the f(x_0+h) = f(x_0)+\nabla f(x_0)^Th +\frac12h^T\operatorname{Hess}f(x_0+\theta h)h.
15.5 Multi-Index Form
For \alpha=(\alpha_1,\dots,\alpha_n)\in\mathbb{N}^n: |\alpha|=\sum_i\alpha_i, \qquad \alpha!=\prod_i\alpha_i!, h^\alpha=\prod_i h_i^{\alpha_i}, \qquad D^\alpha f = \frac{\partial^{|\alpha|}f} {\partial x_1^{\alpha_1}\cdots\partial x_n^{\alpha_n}}. Then formally: f(x_0+h) = \sum_{|\alpha|\le p} \frac{D^\alpha f(x_0)}{\alpha!}h^\alpha +R_p(h). With appropriate hypotheses:
R_p(h)=o(\|h\|^p), or, with a more locally limited derivative, R_p(h)=O(\|h\|^{p+1}).
16 Mean Value Theorem in Several Variables
For f:\mathbb{R}^n\to\mathbb{R}, if the segment [x,y]=\{x+t(y-x):0\le t\le1\} is contained in the domain and f is differentiable, exists \xi on the segment f(y)-f(x)=\nabla f(\xi)^T(y-x).
For f:\mathbb{R}^n\to\mathbb{R}^m there is generally no analogue with a single point \xi valid simultaneously for all components. The estimate is:
\|f(y)-f(x)\| \le \sup_{z\in[x,y]}\|Df(z)\|\,\|y-x\|.
17 Functions with Lipschitz Gradient
f:\mathbb{R}^n\to\mathbb{R} It is L-smooth se \|\nabla f(x)-\nabla f(y)\| \le L\|x-y\|.
f\in C^2 on a convex set: \|\operatorname{Hess}f(x)\|\le L\quad\forall x implies \nabla f It is L-Lipschitz.
17.1 Descent Lemma
\nabla f It is L-Lipschitz:
f(y) \le f(x)+\nabla f(x)^T(y-x) +\frac L2\|y-x\|^2. More precisely:
|f(y)-f(x)-\nabla f(x)^T(y-x)| \le \frac L2\|y-x\|^2.
18 Inverse Function Theorem
Both f:\mathbb{R}^n\to\mathbb{R}^n class C^1 in a neighborhood of x_0. \det Df(x_0)\neq0, then f is locally invertible near x_0 and the inverse is C^1. In addition:
D(f^{-1})(f(x_0)) = Df(x_0)^{-1}.
19 Implicit Function Theorem
Both F:\mathbb{R}^n\times\mathbb{R}^m\to\mathbb{R}^m class C^1, with F(x_0,y_0)=0. D_yF(x_0,y_0) is invertible, then locally there is a function y=\varphi(x) such that F(x,\varphi(x))=0. The derivative is D\varphi(x) = -[D_yF(x,\varphi(x))]^{-1}D_xF(x,\varphi(x)).
20 Unconstrained Extrema and the Second-Order Test
f:\mathbb{R}^n\to\mathbb{R} is differentiable and x_* is an extreme internal place:
\nabla f(x_*)=0.
f\in C^2 and \nabla f(x_*)=0:
\operatorname{Hess}f(x_*)\succ0 \Rightarrow minimum local narrow;
\operatorname{Hess}f(x_*)\prec0 \Rightarrow maximum local tight;
Hessian indefinite \Rightarrow saddle point;
Hessian semidefinite \Rightarrow inconclusive tests in general.
21 Table of Main Implications
| Assumption | Consequence |
|---|---|
| Fréchet differentiable in x_0 | continues x_0 |
| C^1 close to x_0 | differentiable in x_0 |
| Lipschitz | uniformly continuous |
| Df limited on convex set | f Lipschitz |
| \nabla f L-Lipschitz | linear rest O(\Vert h\Vert ^2) |
| \operatorname{Hess}f Lipschitz | Taylor order 2 with rest O(\Vert h\Vert ^3) |
| f=o(g) | f=O(g) |
| f\sim g | f=g+o(g) |
| f\in C^2 | Hessian symmetric |
| \det Df(x_0)\neq0 | local invertability |
22 Hierarchy of Regularity Notions
In general:
C^1 \Longrightarrow \text{differentiable} \Longrightarrow \text{continuous}.
On convex sets, if Df is limited:
C^1+\sup\|Df\|<\infty \Longrightarrow \text{Lipschitz}.
For 0<\alpha<1:
\text{Lipschitz} \Longrightarrow C^{0,\alpha}\text{ locally on bounded sets}.
But, in general, none of the inverse implications is valid without additional hypotheses.
23 Quick Reference for Taylor Expansions and Orders
| Regularity | Local expansion |
|---|---|
| Differentiated | f(x+h)=f(x)+Df(x)h+o(\Vert h\Vert ) |
| Df locally Lipschitz | f(x+h)=f(x)+Df(x)h+O(\Vert h\Vert ^2) |
| Twice differentiable | f(x+h)=f(x)+Df(x)h+\frac12D^2f(x)[h,h]+o(\Vert h\Vert ^2) |
| D^2f locally Lipschitz | f(x+h)=f(x)+Df(x)h+\frac12D^2f(x)[h,h]+O(\Vert h\Vert ^3) |
| C^{p+1} locally | f=T_p+O(\Vert h\Vert ^{p+1}) |
| Taylor-Peano order p | f=T_p+o(\Vert h\Vert ^p) |