22 Optimal Control Problems
22.1 Bolza form and first-order necessary conditions
Consider the finite-dimensional optimal control problem in Bolza form
\begin{aligned} \operatorname*{minimize}_{\bm{x},\bm{u}}\quad & J[\bm{x},\bm{u}] = \Phi\big(\bm{x}(a),\bm{x}(b)\big) + \int_a^b L\big(\bm{x}(t),\bm{u}(t),t\big)\,\mathrm{d}t,\\ \text{subject to}\quad & \bm{x}'(t)=\bm{f}\big(\bm{x}(t),\bm{u}(t),t\big),\\ & \bm{b}\big(\bm{x}(a),\bm{x}(b)\big)=\bm{0},\\ & \bm{u}(t)\in\mathcal U\quad\text{a.e. on }[a,b]. \end{aligned}
Here \bm{x}(t)\in\mathbb R^n is the state, \bm{u}(t)\in\mathbb R^m is the control, and \bm{b}\in\mathbb R^q represents endpoint constraints. Unless stated otherwise, the presentation below uses the normal minimization convention
H(\bm{x},\bm{\lambda},\bm{u},t) = L(\bm{x},\bm{u},t) +\bm{\lambda}^{\mathsf T}\bm{f}(\bm{x},\bm{u},t).
Integral constraints
An integral constraint
\bm{\Psi}\big(\bm{x}(a),\bm{x}(b)\big) +\int_a^b \bm{g}(\bm{x},\bm{u},t)\,\mathrm{d}t=\bm{0}
can be converted into endpoint constraints by adding an auxiliary state \bm{z},
\bm{z}'=\bm{g}(\bm{x},\bm{u},t),\qquad \bm{z}(a)=\bm{0},\qquad \bm{z}(b)+\bm{\Psi}\big(\bm{x}(a),\bm{x}(b)\big)=\bm{0}.
This conversion is exact. In particular, both endpoint conditions on \bm{z} are essential; adding only the differential equation does not enforce the original integral constraint.
Augmented functional and transversality
Define the endpoint Lagrangian
B(\bm{x}_a,\bm{x}_b,\bm{\omega}) =\Phi(\bm{x}_a,\bm{x}_b) +\bm{\omega}^{\mathsf T}\bm{b}(\bm{x}_a,\bm{x}_b),
and the augmented functional
\mathcal J_a =B\big(\bm{x}(a),\bm{x}(b),\bm{\omega}\big) +\int_a^b \left[ H(\bm{x},\bm{\lambda},\bm{u},t) -\bm{\lambda}^{\mathsf T}\bm{x}' \right]\,\mathrm{d}t.
Integration by parts of -\bm{\lambda}^{\mathsf T}\delta\bm{x}' gives
\begin{aligned} \delta\mathcal J_a ={}& \int_a^b \left(\bm{\lambda}'+H_{\bm{x}}\right)^{\mathsf T}\delta\bm{x}\,\mathrm{d}t +\int_a^b H_{\bm{u}}^{\mathsf T}\delta\bm{u}\,\mathrm{d}t\\ &+\int_a^b \left(H_{\bm{\lambda}}-\bm{x}'\right)^{\mathsf T}\delta\bm{\lambda}\,\mathrm{d}t +\bm{b}^{\mathsf T}\delta\bm{\omega}\\ &+\left(B_{\bm{x}_a}+\bm{\lambda}(a)\right)^{\mathsf T}\delta\bm{x}(a) +\left(B_{\bm{x}_b}-\bm{\lambda}(b)\right)^{\mathsf T}\delta\bm{x}(b). \end{aligned}
Therefore, for an unconstrained smooth control, a normal extremal satisfies
\left\{ \begin{aligned} \bm{x}' &= H_{\bm{\lambda}}=\bm{f}(\bm{x},\bm{u},t),\\ \bm{\lambda}' &= -H_{\bm{x}},\\ \bm{b}\big(\bm{x}(a),\bm{x}(b)\big)&=\bm{0},\\ B_{\bm{x}_a}+\bm{\lambda}(a)&=\bm{0},\\ B_{\bm{x}_b}-\bm{\lambda}(b)&=\bm{0},\\ H_{\bm{u}}&=\bm{0}. \end{aligned} \right. \tag{22.1}
The stationarity condition H_{\bm{u}}=\bm{0} is only a first-order necessary condition for an interior unconstrained control. It does not by itself establish that the Hamiltonian is minimized; second-order conditions or convexity may be needed.
Example 22.1 (Minimum-Energy Transfer of a Point Mass) Let’s consider the problem of a mass m sliding on a plane (coordinate x) subjected only to an external applied force F, the optimal control problem is
\begin{aligned} \text{minimize:} \qquad & \int_0^1 F^2\, \mathrm{d}t \\ \text{subject to:} \qquad & x' = v, \qquad v' = \frac F m \\ & x(0) = 0 \qquad x(1) = 1 \\ & v(0) = 0 \qquad v(1) = 0 \end{aligned}
In this case the description of the dynamical system is known from the physical domain, in fact the velocity is the derivative of the position and the acceleration (derivative of velocity) is equal to the force applied divided by the mass; the boundaries conditions are set by the problem.
At this point to solve the problem we have to compute the two function \mathcal B, \mathcal H defined as
\begin{aligned} \mathcal B & = \mu_1 \big(x(a) - 0\big) + \mu_2\big(x(b)-1\big) + \mu_3\big(v(a)-0\big) + \mu_4 \big(v(b) - 0\big) \\ \mathcal H & = F^2 + \lambda_1 v + \lambda_2 \frac F m \end{aligned}
Following the results of Equation 22.1 the boundary valued problem that has to be solved to find the minimum point is
\begin{cases} x' = v \\ v' = F/m \\ \lambda_1' = 0 \\ \lambda_2' = -\lambda_1 \\ x(0) = 0 \qquad x(1) = 1 \\ v(0) = 0 \qquad v(1) = 0 \\ \mu_1 + \lambda_1(0) = 0 \\ \mu_3 + \lambda_2(0) = 0 \\ \mu_2 - \lambda_1(1) = 0 \\ \mu_4 - \lambda_2(1) = 0 \\ 2 F + \frac{\lambda_2}{m} = 0 \end{cases}
In this case the adjoin boundary conditions are trivially solved (in fact will exists \mu_i that will satisfy the equation); considering the last equation in the expression of the velocity the system becomes
\begin{cases} x' = v \\ v' = -\frac{\lambda_2}{2m^2} \\ \lambda_1' = 0 \\ \lambda_2' = -\lambda_1 \\ x(0) = 0 \qquad x(1) = 1 \\ v(0) = 0 \qquad v(1) = 0 \\ \end{cases}
Since \lambda_1(t)=c_1 is constant and \lambda_2'= -\lambda_1, we have \lambda_2(t)=-c_1t+c_2. Therefore
\begin{aligned} v' & = \frac{c_1 t-c_2}{2m^2} \\ \xrightarrow{\int} \quad v & = \frac{c_1}{4m^2} t^2 - \frac{c_2}{2m^2}t + c_3 \\ \xrightarrow{\int} \qquad x & = \frac{c_1}{12m^2}t^3 - \frac{c_2}{4m^2}t^2 + c_3 t + c_4 \end{aligned}
Considering the boundary conditions than the integration constant are c_3 = c_4 = 0 and the endpoint conditions determine c_1=-24m^2 and c_2=-12m^2, giving
\begin{aligned} x(t) &= 3t^2-2t^3, \\ v(t) &= 6t-6t^2, \\ \lambda_1(t) &= -24m^2, \\ \lambda_2(t) &= 24m^2t-12m^2. \end{aligned}
and so the control history to determine this minimal solution is
F = - \frac{\lambda_2}{2m} = 6\big(1-2t\big) m
22.2 Pontryagin minimum principle
When the control is constrained to an admissible set \mathcal U, the interior stationarity equation H_{\bm{u}}=\bm{0} must be replaced by the pointwise Hamiltonian minimum condition. Under the usual regularity hypotheses, a normal optimal process admits a nontrivial costate \bm{\lambda} such that, almost everywhere,
\begin{aligned} \bm{x}' &= H_{\bm{\lambda}},\\ \bm{\lambda}' &= -H_{\bm{x}},\\ \bm{u}^\star(t) &\in \operatorname*{arg\,min}_{\bm{v}\in\mathcal U} H\big(\bm{x}^\star(t),\bm{\lambda}(t),\bm{v},t\big), \end{aligned}
with the endpoint constraints and transversality conditions in Equation 22.1. Compactness of \mathcal U is a common sufficient assumption ensuring that a minimum exists when H is continuous in the control; convexity of \mathcal U is not required by the Pontryagin principle itself.
If \bm{u}^\star(t) lies in the interior of \mathcal U and H is differentiable with respect to \bm{u}, the minimum condition implies
H_{\bm{u}}\big(\bm{x}^\star,\bm{\lambda},\bm{u}^\star,t\big)=\bm{0}.
At a boundary point this equation need not hold. For convex \mathcal U, the corresponding first-order variational inequality is
H_{\bm{u}}(\bm{x}^\star,\bm{\lambda},\bm{u}^\star,t)^{\mathsf T} (\bm{v}-\bm{u}^\star)\ge 0, \qquad \forall\bm{v}\in\mathcal U.
The general Pontryagin principle also allows abnormal extremals, in which the multiplier of the objective vanishes. The chapters here use the normal scaling. When abnormality is possible, one introduces a scalar multiplier \lambda_0\ge0 and writes H=\lambda_0L+\bm{\lambda}^{\mathsf T}\bm{f}, with (\lambda_0,\bm{\lambda}) not identically zero.
Examples of the Pontryagin minimum principle
Example 22.2 (Maximum-Travel Problem with Bounded Force) Let’s consider the system in example Example 22.1 where in this case the goal is to maximize the travel of the mass m having a force F \leq |1|; this means solving the following optimal control problem
\begin{aligned} \text{minimize:} \qquad & -x(1) \\ \text{subject to:} \qquad & x' = v, \qquad v' = \frac F m \\ & x(0) = 0 \\ & v(0) = 0 \qquad v(1) = 0 \\ & F \in [-1,1] \end{aligned}
In order to solve this problem it’s necessary to determine the Hamiltonian
\mathcal{H}(x,v,\lambda_1,\lambda_2, F,t) = \lambda_1 v + \lambda_2 \frac F m
and the endpoint Lagrangian
\mathcal{B}(x_a,v_a,x_b,v_b,\mu_1,\mu_2,\mu_3) = -x_b + \mu_1 x_a + \mu_2 v_a + \mu_3 v_b
The adjoint equation so becomes
\lambda_1' = - \frac{\partial\mathcal{H}}{\partial x} = 0
and
\lambda_2' = - \frac{\partial\mathcal{H}}{\partial v} = - \lambda_1
for the the adjoint boundary conditions only -1 - \lambda_1(1)=0 is reported (the other equation trivilly solves the cofficients \mu_i that gives no information for the final solution). The boundary valued problem so become
\left\{ \begin{aligned} & x' = v \\ & v' = F/m \\ & \lambda_1' = 0 \\ & \lambda_2' = - \lambda_1 \\ & -1-\lambda_1(1)= 0\\ & x(0) = v(0) = v(1) = 0 \\ & F(t) = \underset{\overline F \in [-1,1]}{\text{argmin}} \left\{ \lambda_1(t) v + \lambda_2(t) \frac {\overline F}m \right\} \end{aligned} \right.
In this case in order to compute the argument minimum associated to the Pontryagin principle we have to consider only the term related to \overline F, and in particular this means
\begin{aligned} F(t) & = \underset{\overline F \in [-1,1]}{\text{argmin}} \left\{ \lambda_2(t) \frac {\overline F}m \right\} \\& = \begin{cases} + 1 & \text{if } \lambda_2(t) < 0 \\ - 1 & \text{if } \lambda_2(t) > 0 \\ [-1,1] \qquad & \text{if } \lambda_2(t) = 0 \\ \end{cases} \\ & = - \text{sign} \lambda_2(t) \end{aligned}
and so the simplified system that determines the solution becomes
\left\{ \begin{aligned} & x' = v \\ & v' = -\frac{\text{sign}\lambda_2}{m} \\ & \lambda_1' = 0 \\ & \lambda_2' = - \lambda_1 \\ & \lambda_1(1)= -1\\ & x(0) = v(0) = v(1) = 0 \end{aligned} \right.
By integration it can be seen that \lambda_1 can be regarded as a constant c_1 that, for the adjoint boundary condition, is equal to -1 and so \lambda_2 is in the form
\lambda_2 = t + c_2
From now on the solution becomes analytically complex, however it can be found that c_2 = -\frac{1}{2} determining the solution
F(t) = \begin{cases} 1 \qquad & t \leq \frac 1 2 \\ -1 & t > \frac 1 2 \end{cases}
Example 22.3 (A Cautionary Example: Stationarity Is Not Sufficient) Given the formal optimal control problem
\begin{aligned} \text{minimize:} \qquad & \mathcal{F} (x,u) = \int_0^2 2x- u^2 \, \mathrm{d}t \\ \text{subject to:} \qquad & x' = 1-2u \\ & x(0) = 1 \qquad x(2) = 0 \end{aligned}
As stated, this problem does not possess a minimum over an unconstrained control class such as L^2(0,2). Indeed, integrating the dynamics gives the single mean-value constraint
\int_0^2 u(t)\,\mathrm{d}t=\frac{3}{2}.
The negative quadratic term -\int_0^2u(t)^2\,\mathrm{d}t can then be made arbitrarily negative by adding zero-mean oscillations of increasing L^2 norm while preserving the endpoint condition. Hence \inf J=-\infty.
The equation H_u=0 would only identify a stationary extremal; because H is concave in u, it cannot certify a pointwise minimum. This example is therefore useful as a warning: Pontryagin stationarity is necessary only under the relevant hypotheses and is never, by itself, a sufficiency test.
To obtain a well-posed minimization problem one may, for example, replace -u^2 by +u^2 or impose a compact control bound u\in[u_{\min},u_{\max}].
Example 22.4 (Integral Constraint and Bounded Control) To solve the optimal control problem
\begin{aligned} \text{minimize:} \qquad & \int_0^1 t + x - y^2 \, \mathrm{d}t \\ \text{subject to:} \qquad & x' = x+y \qquad y' = x + yu \\ & \int_0^1\sin y\, \mathrm{d}t = \gamma\\ & x(0) = 1 \qquad y(1) = 0 \\ & u \in [-2,3] \end{aligned}
where necessarily |\gamma|\le 1, because |\sin y|\le1 on an interval of length one. The value 3 sometimes appearing in earlier versions of this example makes the problem infeasible. We remove the integral constraint by introducing one auxiliary state:
\begin{aligned} \text{minimize:} \qquad & \int_0^1 t + x - y^2 \, \mathrm{d}t \\ \text{subject to:} \qquad & x' = x+y \\ & y' = x + yu \\ & z' = \sin y \\ & x(0) = 1, \qquad & z(0) = 0 \\ & y(1) = 0, \qquad & z(1) = \gamma \\ & u \in [-2,3] \end{aligned}
The associated Hamiltonian \mathcal{H} and endpoint Lagrangian \mathcal{B} of the problem are
\begin{aligned} \mathcal{H}(x,y,z,\lambda_1, \lambda_2, \lambda_3) & = t + x - y^2 \\ & + \lambda_1 (x+y) + \lambda_2 (x+yu) + \lambda_3 \sin y \\ \mathcal{B}(x_0, y_1, z_0, z_1, \mu_{1\dots 4}) & = -\mu_1 \big(x_0 - 1\big) - \mu_2 y_1 \\ &- \mu_3 z_0 - \mu_4 \big(z_1 -\gamma\big) \end{aligned}
With that stated the co-equations are computed as
\begin{aligned} \lambda_1' & = - \frac{\partial\mathcal{H}}{\partial x} = -1-\lambda_1-\lambda_2, \\ \lambda_2' & = - \frac{\partial\mathcal{H}}{\partial y} = 2y-\lambda_1-\lambda_2u-\lambda_3\cos y \\ \lambda_3' & = - \frac{\partial\mathcal{H}}{\partial z} = 0 \end{aligned}
The unconstrained endpoint components are x(1) and y(0), hence transversality gives \lambda_1(1)=0 and \lambda_2(0)=0. The bounded control is obtained directly from the Hamiltonian minimum condition. The resulting BVP is
\begin{cases} x'=x+y,\\ y'=x+yu,\\ z'=\sin y,\\ \lambda_1'=-1-\lambda_1-\lambda_2,\\ \lambda_2'=2y-\lambda_1-\lambda_2u-\lambda_3\cos y,\\ \lambda_3'=0,\\ x(0)=1,\qquad y(1)=0,\\ z(0)=0,\qquad z(1)=\gamma,\\ \lambda_1(1)=0,\qquad \lambda_2(0)=0,\\ u^\star(t)\in \operatorname*{arg\,min}_{\overline u\in[-2,3]} \left\{\lambda_2(t)y(t)\,\overline u\right\}. \end{cases}
Therefore
u^\star(t)=\begin{cases} -2, & \lambda_2(t)y(t)>0,\\ [-2,3], & \lambda_2(t)y(t)=0,\\ 3, & \lambda_2(t)y(t)<0. \end{cases}
The set-valued middle case is a candidate singular situation and requires further analysis; it cannot be assigned arbitrarily to one endpoint of the control interval.
Example 22.5 (Bang-Bang Control with a Mayer Term) Given the optimal control problem
\begin{aligned} \text{minimize:} \qquad & \mathcal{F} (x,y,u) = x(0) + \int_0^1 x^2 + y^2 + u \, \mathrm{d}t \\ \text{subject to:} \qquad & x' = y - u, \qquad y' = x+u \\ &x(1) = 2 \\ & u(t) \in [-1,2] \end{aligned}
the solution can be obtained by firstly computing the Hamiltonian \mathcal{H} and the endpoint Lagrangian \mathcal{B} of the problems resulting in
\begin{aligned} \mathcal{H}\big(x,y,u,\lambda_1,\lambda_2\big) & = x^2 + y^2 + u + \lambda_1 \big(y-u\big) + \lambda_2 \big(x + u\big) \\ & = x^2 + y^2 + u \big(1 - \lambda_1 + \lambda_2\big) + \lambda_1 y + \lambda_2 x \\ \mathcal{B}\big(x_0,x_1, \mu\big) & = x_0 + \mu \big( x_1 - 2\big) \end{aligned}
We can so explicit: the adjoint ordinary differential equations
\begin{aligned} \lambda_1' &= -\frac{\partial\mathcal{H}}{\partial x} = - 2x - \lambda_2 \\ \lambda_2' &= - \frac{\partial\mathcal{H}}{\partial y} = -2y - \lambda_1 \end{aligned}
the adjoint boundary conditions
\begin{aligned} \lambda_1(0) & = - \frac{\partial\mathcal{B}}{\partial x_0} = - 1 \\ \lambda_1(1) &= \frac{\partial\mathcal{B}}{\partial x_1} = \mu \\ \lambda_2(0) & = -\frac{\partial\mathcal{B}}{\partial y_0} = 0 \\ \lambda_2(1) & = \frac{\partial\mathcal{B}}{\partial y_1} = 0 \end{aligned}
The switching function is
\sigma(t)=\frac{\partial\mathcal{H}}{\partial u}=1-\lambda_1(t)+\lambda_2(t).
Because the control is bounded, \sigma=0 must not be imposed identically. The control is obtained from the Hamiltonian minimum condition. With the original differential equations, the BVP becomes
\begin{cases} x' = y - u \\ y' = x+u \\ \lambda_1' = -2x - \lambda_2 \\ \lambda_2' = -2y - \lambda_1 \\ \lambda_1(0) = -1 \\ \cancel{\lambda_1(1) = \mu} \qquad \qquad \text{: trivially satisfied} \\ \lambda_2(0) = \lambda_2(1) = 0 \\ u = \underset{\overline u \in [-1,2]}{\text{argmin}} \left\{ \overline u \big(1 - \lambda_1 + \lambda_2\big) \right\} \end{cases}
In this case the solution of the Pontryagin minimum principle is reduced to the form
u^\star(t)=\begin{cases} -1, & \sigma(t)>0,\\ [-1,2], & \sigma(t)=0,\\ 2, & \sigma(t)<0. \end{cases}
22.3 What the Necessary Conditions Do Not Prove
The Pontryagin conditions characterize extremals; they do not by themselves prove existence, uniqueness, or optimality. Existence typically requires compactness/coercivity and appropriate continuity and convexity hypotheses. Sufficiency can follow in special convex problems, or from a verification theorem based on the Hamilton–Jacobi–Bellman equation. Second-order conditions are needed to distinguish minima from maxima or saddle extremals when convexity is absent.
The Hamilton–Jacobi–Bellman approach uses the value function V(\bm{x},t) and, formally, the PDE
-V_t(\bm{x},t)=\min_{\bm{u}\in\mathcal U} \left\{L(\bm{x},\bm{u},t)+\nabla_{\bm{x}}V(\bm{x},t)^{\mathsf T}\bm{f}(\bm{x},\bm{u},t)\right\}.
Where V is differentiable along an optimal trajectory, the costate is related to the value-function gradient. This provides a useful conceptual bridge between indirect methods based on PMP and dynamic programming.
22.4 Linear Controls and Singular Arcs
In this section, we examine problems in which the Hamiltonian is a linear function of the control variable u. Although linear problems are generally expected to be easier to handle, this is not always the case. The complexity arises from the fact that the necessary condition for minimizing the Hamiltonian with respect to the control along an optimal trajectory does not yield a well-defined expression for synthesizing the optimal control.
Another challenge associated with this class of problems is the occurrence of discontinuities in the control, which can be difficult to identify. Additionally, there may be an infinite number of switching points that can accumulate at a specific time instant. Given that general results and theorems regarding existence and uniqueness are limited in this context, it becomes necessary to introduce additional relationships by manipulating other necessary conditions.
Consequently, problems with a Hamiltonian that is linear in terms of the control u represent an almost independent area of research within optimal control theory. Most theoretical results in this domain are derived using a geometric approach, involving fiber bundles and symmetry groups rather than traditional variational techniques.
To maintain clarity and simplicity in notation, we will focus on problems with a single control variable and fixed endpoints. The formulation of these problems is as follows:
Definition 22.1 Minimize the cost functional J subject to the dynamic \bm{x}' with the control u constrained in |u|\leq 1 and the initial state \bm{x}(0)=\bm{x}_0, where
\begin{aligned} J(u)&= \int_0^T f_0(\bm{x})+b_0(\bm{x})u \,\mathrm{d}t\\ \bm{x}'(t) &= \bm{f}(\bm{x})+\bm{b}(\bm{x})u\qquad \mathrm{i.e.} \end{aligned} \tag{22.2}
f_0(\bm{x}) and b_0(\bm{x}) are scalar functions of the state \bm{x}, u is the scalar control.
To further simplify the computations, we consider the scalar equation of the functional as an extra state variable posing
x_0'= f_0(\bm{x})+b_0(\bm{x})u, \qquad x_0(0)=0,
(we have already seen that this is not a loss of generality), but the key point here is that the control appears only linearly in both dynamics and cost functional. We can write the Hamiltonian for this system, namely
\mathcal{H}(\bm{x},\bm{\lambda},t) = \sum_{i=0}^n f_i(\bm{x})\lambda_i+u\sum_{i=0}^n b_i(\bm{x})\lambda_i.
It is convenient to rename the two addends as
\begin{aligned} A(\bm{x},\bm{\lambda},t)&=\sum_{i=0}^n f_i(\bm{x})\lambda_i,\\ B(\bm{x},\bm{\lambda},t)&=\sum_{i=0}^n b_i(\bm{x})\lambda_i, \end{aligned}
so that the Hamiltonian becomes
\mathcal{H}(\bm{x},\bm{\lambda},t) = A(\bm{x},\bm{\lambda},t)+B(\bm{x},\bm{\lambda},t)u. \tag{22.3}
The adjoint variables of the costate are given by the usual relation
\lambda_i' = -\frac{\partial\mathcal{H}}{\partial x_i}= -\sum_{j=0}^n\lambda_j\frac{\partial f_j(\bm{x})}{\partial x_i}-u\sum_{j=0}^n\lambda_j\frac{\partial b_j(\bm{x})}{\partial x_i}.
We state now the necessary conditions for problem Equation 22.2.
Theorem 22.1 If u^\star(t) is an optimal control and if \bm{x}^\star(t) is the corresponding optimal trajectory, then there are a costate \lambda_i^\star(t) such that
for i=1,\ldots,n \begin{aligned} x_{i}'^{\star} &= f_i(\bm{x}^{\star})+b_i(\bm{x}^{\star})u^{\star},\\ \lambda_i'^{\star} &= -\sum_{j=0}^n\lambda_j^\star\frac{\partial f_j(\bm{x}^\star)}{\partial x_i^\star}- u^\star\sum_{j=0}^n\lambda_j^\star(t)\frac{\partial b_j(\bm{x}^\star)}{\partial x_{i}^\star}; \end{aligned}
for all t\in[0,T] and all u(t) s.t. |u(t)|\leq 1, holds A(\bm{x}^\star,\lambda^\star,t)+u^\star(t)B(\bm{x}^\star, \lambda^\star,t)\leq A(\bm{x}^\star,\lambda^\star,t)+u(t)B(\bm{x}^\star,\lambda^\star,t);
If T is free, then for all t\in[0,T]
\mathcal{H}^\star(\bm{x}^\star, \bm{\lambda}^\star,t) := A(\bm{x}^\star, \bm{\lambda}^\star,t)+u^\star(t)B(\bm{x}^\star, \bm{\lambda}^\star,t)=0 , \tag{22.4}
while, if T is fixed, then
\mathcal{H}^\star(\bm{x}^\star, \bm{\lambda}^\star,t) := A(\bm{x}^\star, \bm{\lambda}^\star,t) +u^\star(t)B(\bm{x}^\star, \bm{\lambda}^\star,t)=c , \tag{22.5} for a real constant c.
It follows easily from Equation 22.3 with the PMP, that the optimal control is given by
u^\star(t)=-\mathrm{sign}\{B(\bm{x}^\star, \bm{\lambda}^\star,t)\}=-\mathrm{sign}\left\{\sum_{i=0}^n b_i(\bm{x}^\star(t))\lambda^\star_i(t)\right\},
as long as the function B(\bm{x}^\star,\bm{\lambda}^\star,t) is not zero. If B becomes zero, then the sign function is not defined. If B is zero only in a point we speak of bang-bang controls, if B is identically zero for an entire interval t\in(t_1,t_2], the controls are called singular. In this case, the associated trajectory \bm{x}(t) is called a singular arc.
Singular controls
If problem Equation 22.2 results singular, controls, costate and trajectory have the following property: there is at least one half-open interval (t_1,t_2]\subset[0,T] such that
\sum_{i=0}^n \lambda^\star_i(t)b_i(\bm{x}^\star(t))=0, \qquad t\in(t_1,t_2]. \tag{22.6}
As before, the existence of an extremal singular control, does not imply that the optimal control is singular: we need additional information as uniqueness to conclude its optimality. The function Equation 22.6 is often called switching function.
For a singular arc, vanishing of the switching function and its time derivatives is only part of the analysis. If the control first appears explicitly in the 2r-th derivative \sigma^{(2r)}, a standard higher-order necessary condition for a minimizing singular arc is the generalized Legendre–Clebsch condition
(-1)^r\frac{\partial}{\partial u} \left(\frac{\mathrm d^{2r}}{\mathrm dt^{2r}}H_u\right)\ge0.
Failure of this condition rules out local optimality of the singular extremal.
Test for singular controls
Suppose we are in the free terminal time case, so that Equation 22.4 holds, we test if it is possible to have a singular control as follows. First we need to assume that the switching function is zero in an unknown interval (t_1,t_2], then, because of the free end time hypothesis, for t\in(t_1,t_2],
\begin{aligned} \mathcal{H}(\bm{x},\bm{\lambda},t) = \sum_{i=0}^n f_i(\bm{x})\lambda_i+u\sum_{i=0}^n b_i(\bm{x})\lambda_i &= 0\\ & \Downarrow\\ \sum_{i=0}^n f_i(\bm{x})\lambda_i&=0. \end{aligned}
Moreover, for each k\in\mathbb{N}, we have that
\frac{\mathrm{d}^k}{\mathrm{d}t^k} \sum_{i=0}^n \lambda_i(t)b_i(\bm{x})=0, \quad \left\{\begin{aligned} k & =1,2,3,\ldots, \\ t & \in(t_1,t_2], \end{aligned}\right. \tag{22.7}
and similarly,
\frac{\mathrm{d}^k}{\mathrm{d}t^k}\sum_{i=0}^n \lambda_i f_i(\bm{x})=0, \quad \left\{\begin{aligned} k & =1,2,3,\ldots, \\ t & \in(t_1,t_2], \end{aligned}\right. \tag{22.8}
However, the canonical equations (we omit explicit time dependence) are
\begin{aligned} x_i' &= f_i+ub_i\\ \lambda_i' &= -\sum_{j=0}^n \lambda_j(t)\frac{\partial f_j}{\partial x_i} -u \sum_{j=0}^n \lambda_j \frac{\partial b_j}{\partial x_i}, \quad \left\{\begin{aligned} i,j & =1,2,\ldots,n \\ k & \in\mathbb{N}, \end{aligned}\right. \end{aligned}
Let k=1 in Equation 22.7, then by the chain rule we have
\frac{\mathrm{d}}{\mathrm{d}t}\sum_{i=0}^n \lambda_i b_i = \sum_{i=0}^n\left(\lambda_i'b_i+\lambda_i \sum_{j=0}^n \frac{\partial b_j}{\partial x_i}x'_j\right)=0.
Substituting the canonical equations in the previous one, yields
\sum_{i=0}^n\sum_{j=0}^n\left( f_j\lambda_i\frac{\partial b_i}{\partial x_j}-b_i\lambda_j\frac{\partial f_j}{\partial x_i}\right) +u \sum_{i=0}^n\sum_{j=0}^n\left( \lambda_i b_j\frac{\partial b_i}{\partial x_j}- \lambda_j b_i\frac{\partial b_j}{\partial x_i}\right)=0.
We notice that the coefficient of u is zero, and we conclude that
\sum_{i=0}^n\sum_{j=0}^n\lambda_i \left( b_j\frac{\partial f_i}{\partial x_j}-f_j\frac{\partial b_i}{\partial x_j}\right)=0. \tag{22.9}
Applying the same argument to Equation 22.8, gives
\begin{aligned} 0 &= \frac{\mathrm{d}}{\mathrm{d}t}\sum_{i=0}^n \lambda_if_i(\bm{x}) \\ &= \sum_{i=0}^n\left(\lambda_i'f_i+\lambda_i\sum_{j=0}^n\frac{\partial f_i}{\partial x_j}x_j'\right)\\ &= \sum_{i=0}^n\sum_{j=0}^n\left( \lambda_if_i\frac{\partial f_i}{\partial x_j}-\lambda_jf_i\frac{\partial f_j}{\partial x_i}\right) +u \sum_{i=0}^n\sum_{j=0}^n\left( \lambda_i b_j\frac{\partial f_i}{\partial x_j}- \lambda_j f_i\frac{\partial b_j}{\partial x_i}\right)\\ &= u \sum_{i=0}^n\sum_{j=0}^n\lambda_i\left( b_j\frac{\partial f_i}{\partial x_j}-f_j\frac{\partial b_i}{\partial x_j}\right)=0. \end{aligned}
This last line implies that either u=0 or
\sum_{i=0}^n\sum_{j=0}^n\lambda_i\left( b_j\frac{\partial f_i}{\partial x_j}-f_j\frac{\partial b_i}{\partial x_j}\right)=0.
Noticing that the previous line is equal to Equation 22.9, we can not conclude that u=0, and this is true for all k\in\mathbb{N}.\ In conclusion, a necessary but not sufficient test for an extremal singular control is that in the interval (t_1,t_2] the following relations are satisfied: \left.\begin{aligned} \displaystyle\sum_{i=0}^n b_i\lambda_i=0\\ \displaystyle\sum_{i=0}^n f_i\lambda_i=0\\ \displaystyle\sum_{i=0}^n\sum_{j=0}^n\lambda_i\left( b_j\frac{\partial f_i}{\partial x_j}-f_j\frac{\partial b_i}{\partial x_j}\right)=0 \end{aligned}\right\}
We show now how to compute the explicit expression for the singular control, we return back to the equation of the switching function Equation 22.7 and let k=2,3,\ldots. It is shown in [1:pag.487] that those derivatives for k=2,3,\ldots require extensive manipulations but have all the following same structure:
\sum_{i=0}^n\lambda_i\psi_{ki}(\bm{x})+u\sum_{i=0}^n\lambda_i\phi_{ki}(\bm{x})=0.
If \sum \lambda_i\phi_{ki}(\bm{x})=0 we can perform successive derivatives of Equation 22.7, in general there will be a finite value of k=m for which \sum \lambda_i\phi_{mi}(\bm{x})\neq 0 and then the singular control u can be solved:
u= -\frac{\displaystyle\sum_{i=0}^n\lambda_i\psi_{mi}(\bm{x})}{\displaystyle\sum_{i=0}^n\lambda_i\phi_{mi}(\bm{x})}.
Up to now we have dealt with free final time, if the final time is fixed the we have to take in account the presence of the constant c in the Hamiltonian, but Equation 22.7 and Equation 22.8 still hold and hence also the conclusions.\ In practical problems, we must check if
\sum \lambda_i\phi_{ki}(\bm{x})=0
for all k=1,\ldots,m-1 and
\sum \lambda_i\phi_{mi}(\bm{x})\neq 0
If any of the relations are violated then this represent a violation of the necessary conditions and so singular controls can not occur, if these conditions are all met, then there may be a singular extremal control.