18 Optimal Control problems
Considering the problem
\begin{aligned} \textrm{minimize:} \quad & \underbrace{ \overbrace{ \phi\big(\bm{x}(a), \bm{x}(b)\big)}^\textrm{Mayer} + \overbrace{\int_a^b L (\bm{x}, \bm{u},t)\, \mathrm{d}t}^\textrm{ Lagrange}}_\textrm{Bolza} \quad && \textrm{: target} \\ \textrm{subject to:} \quad & \bm{x}' = \bm{f}\big(\bm{x},\bm{u},t\big) && \textrm{: dynamical system} \\ & \bm{b}\big(\bm{x}(a),\bm{x}(b)\big) = 0 && \textrm{: boundary conditions} \\ & \int_a^b \bm{g}(\bm{x},\bm{u}, t)\, \mathrm{d}t = \bm{g}_0 && \textrm{: integral constraints} \end{aligned}
where \bm{x} is the state vector and \bm{u} is the control vector. An integral constraint can be converted into a boundary condition by adding the state \bm{z} defined by \bm{z}'=\bm{g}(\bm{x},\bm{u},t). With \bm{z}(a)=\bm{0}, integration gives
z(a) = 0 \qquad z(b) = g_0
With that said the minimization problem becomes
\begin{aligned} \textrm{minimize:} \qquad & \phi\big(\bm{x}(a), \bm{x}(b)\big) + \int_a^b L (\bm{x}, \bm{u},t)\, \mathrm{d}t \\ \textrm{subject to:} \qquad & \bm{x}' = \bm{f}\big(\bm{x},\bm{u},t\big) \\ & \bm{z}' = \bm{g}(\bm{x},\bm{u},t) \\ & \bm{b}\big(\bm{x}(a),\bm{x}(b)\big) = 0 \\ & \bm{z}(a) = 0 \quad \bm{z}(b) = \bm{g}_0 \end{aligned}
Compacting the vector of states \bm{x} and added variable \bm{z} we can define a new variable vector \bm{w}, and similarly all the dynamical system can be described by a multi-variable function F and boundary conditions B and so the problem can be compacted as
\begin{aligned} \textrm{minimize:} \quad & \psi\big(\bm{w}(a), \bm{w}(b)\big) + \int_a^b \mathcal M(\bm{w}, \bm{u},t)\, \mathrm{d}t \\ \textrm{subject to:} \quad & \bm{w}' = \bm{F}\big(\bm{w},\bm{u},t\big) \\ & \bm{B}\big(\bm{w}(a),\bm{w}(b)\big) = 0 \end{aligned}
To solve the problem as here state it’s necessary to build the function of the boundary conditions \bm{\mathcal{B}} (depending by the Lagrange multipliers \bm{\mu}), also known as utility function, and the Hamiltonian \mathcal{H} (depending by the multipliers \bm{\lambda}):
\begin{split} \bm{\mathcal B}\big(\bm{w}(a),\bm{w}(b),\bm{\mu}\big) & = \psi\big(\bm{w}(a),\bm{w}(b)\big) + \bm{\mu}\cdot \bm{B}\big( \bm{w}(a),\bm{w}(b) \big) \\ \mathcal H(\bm{w},\bm{\lambda},\bm{u},t) & = \mathcal M(\bm{w}, \bm{u},t) + \bm{\lambda} \cdot \bm{F}(\bm{w},\bm{u}, t) \end{split}
At this point it’s possible to compute the lagrangian \mathcal{L} on which the variations can be performed:
\begin{aligned} \mathcal{L}\big( \bm{w},\bm{u},\bm{\lambda}, \bm{\mu} ,t \big) & = \bm{\mathcal B}\big(\bm{w}(a),\bm{w}(b),\bm{\mu}\big) + \mathcal H(\bm{w},\bm{\lambda},\bm{u},t) \\ \delta\mathcal{L} & = \int_a^b \Big( A \delta w + B \delta u + C \delta\lambda \Big)\, \mathrm{d}t \\ & + D \, \delta w(a) + E \, \delta w(b) - \bm{B} \big(w(a),w(b)\big) \delta\mu \end{aligned}
where
A = \lambda' + \frac{\partial H}{\partial w} \qquad B = \frac{\partial H}{\partial u} \qquad C = f(w,u,t) - x'
D = \frac{\partial B}{\partial w(a)}\Big|_{a} + \lambda(a) \qquad \qquad E = \frac{\partial B}{\partial w(b)}\Big|_{b} -\lambda(b)
With that said the resulting boundary valued problem becomes
\left\{\begin{aligned} &w' = f(x,u,t) && \textrm{: original ODE} \\ &\lambda' = - \frac{\partial H}{\partial w} && \textrm{: adjoint ODE} \\ &B\big(w(a),w(b)\big) = 0 && \textrm{: original boundary condition} \\ &\left. \begin{aligned} \frac{\partial B}{\partial w(a)}\Big|_{a}+ \lambda(a) = 0\\ \frac{\partial B}{\partial w(b)}\Big|_{b} - \lambda(b) = 0 \end{aligned} \qquad \right\} &&\textrm{: adjoint boundary conditions} \\ & \frac{\partial H}{\partial u} = 0 && \textrm{: control equation} \end{aligned} \right. \tag{18.1}
Example 18.1 (optimal control problem) Let’s consider the problem of a mass m sliding on a plane (coordinate x) subjected only to an external applied force F, the optimal control problem is
\begin{aligned} \textrm{minimize:} \qquad & \int_0^1 F^2\, \mathrm{d}t \\ \textrm{subject to:} \qquad & x' = v, \qquad v' = \frac F m \\ & x(0) = 0 \qquad x(1) = 1 \\ & v(0) = 0 \qquad v(1) = 0 \end{aligned}
In this case the description of the dynamical system is known from the physical domain, in fact the velocity is the derivative of the position and the acceleration (derivative of velocity) is equal to the force applied divided by the mass; the boundaries conditions are set by the problem.
At this point to solve the problem we have to compute the two function \mathcal B, \mathcal H defined as
\begin{aligned} \mathcal B & = \mu_1 \big(x(a) - 0\big) + \mu_2\big(x(b)-1\big) + \mu_3\big(v(a)-0\big) + \mu_4 \big(v(b) - 0\big) \\ \mathcal H & = F^2 + \lambda_1 v + \lambda_2 \frac F m \end{aligned}
Following the results of equation Equation 18.1 the boundary valued problem that has to be solved to find the minimum point is
\begin{cases} x' = v \\ v' = F/m \\ \lambda_1' = 0 \\ \lambda_2' = \lambda_1 \\ x(0) = 0 \qquad x(1) = 1 \\ v(0) = 0 \qquad v(1) = 0 \\ \mu_1 + \lambda_1(0) = 0 \\ \mu_3 + \lambda_2(0) = 0 \\ \mu_2 - \lambda_1(1) = 0 \\ \mu_4 - \lambda_2(1) = 0 \\ 2 F + \frac{\lambda_2}{m} = 0 \end{cases}
In this case the adjoin boundary conditions are trivially solved (in fact will exists \mu_i that will satisfy the equation); considering the last equation in the expression of the velocity the system becomes
\begin{cases} x' = v \\ v' = -\frac{\lambda_2}{2m^2} \\ \lambda_1' = 0 \\ \lambda_2' = \lambda_1 \\ x(0) = 0 \qquad x(1) = 1 \\ v(0) = 0 \qquad v(1) = 0 \\ \end{cases}
Considering that \lambda_1 is a constant c_1 (in order to have null derivative), then it means that \lambda_2 is in the form c_1 t + c_2 and so the expression of the acceleration and velocity becomes
\begin{aligned} v' & = -\frac{c_1 t + c_2}{2m^2} \\ \xrightarrow{\int} \quad v & = - \frac{c_1}{4m^2} t^2 - \frac{c_2}{2m^2}t + c_3 \\ \xrightarrow{\int} \qquad x & = - \frac{c_1}{12m^2}t^3 - \frac{c_2}{4m^2}t^2 + c_3 t + c_4 \end{aligned}
Considering the boundary conditions than the integration constant are c_3 = c_4 = 0 and the solution of the linear system given by -c_1 - 2c_2 = 0 and -c_1 -3c_2 = 12m^2 that determines the last full solution
\begin{aligned} x & = - 6t^2 + 6 t \\ v & = - 2t^3 + 3 t^2 \\ \lambda_1 & = 24m^2 \\ \lambda_2 & = 24m^2 t - 12m^2 \end{aligned}
and so the control history to determine this minimal solution is
F = - \frac{\lambda_2}{2m} = 6\big(1-2t\big) m
18.1 Pontryagin maximum (minimum) principle
The Pontryagin maximum (minimum) principle has been developed in order to solve problem where the controls must stay bounded to a certain range due to the physical implementation of the system (if, as example, a motor is powered with 20W of power, than it cannot move object that require more than that power).
Mathematically this kind of problem is described by
\begin{aligned} \textrm{minimize:} \qquad & \phi\big(x(a),x(B)\big) + \int_a^b L (x,u,t) \, \mathrm{d}t \\ \textrm{subject to:} \qquad & x' = f(x,u,t) \\ & b\big(x(a), x(b)\big) = 0 \\ & u(t) \in \mathcal U \end{aligned}
where \mathcal U is the domain of the controls that, in order for the principle to work, must be convex and compact.
To solve this problem we compute the Hamiltonian \mathcal H and the utility function \mathcal{B} as in the previous cases:
\begin{aligned} \mathcal{H}(x,u, \lambda,t) & = L(x,u,t) + \lambda f(x,u,t) \qquad \\ \mathcal{B}(x_a,x_b,\mu) & = \phi(x_a,x_b) + \mu b(x_a,x_b) \end{aligned}
With that stated the boundary valued problem becomes similar to the formulation in Equation 18.1 with the addition of the Pontryagin minimum principle:
\left\{ \begin{aligned} & x' = f(x,u,t) = \frac{\partial\mathcal H}{\partial\lambda} && \textrm{: original ODE} \\ &\lambda' = - \frac{\partial\mathcal H}{\partial x} && \textrm{: adjoint ODE - co-equations} \\ &b\big(w(a),w(b)\big) = 0 && \textrm{: original BC} \\ &\left. \begin{aligned} \frac{\partial\mathcal{B}}{x_a} + \lambda(a) = \bm{0} \\ \frac{\partial\mathcal{B}}{x_b} - \lambda(b) = \bm{0} \end{aligned} \qquad \right\} &&\textrm{: adjoint BC} \\ & u(t) = \underset{\overline u \in \mathcal U}{\textrm{argmin}} \big\{ \mathcal{H}\big(x, \overline u,\lambda, t\big) \big\} && \textrm{: Pontryagin min principle} \\ & \frac{\partial\mathcal{H}}{\partial u} = 0 && \textrm{: control equation} \end{aligned} \right.
Examples with Pontryagin maximum (minimum) principle
Example 18.2 (maximum travel optimal control problem) Let’s consider the system in example Example 18.1 where in this case the goal is to maximize the travel of the mass m having a force F \leq |1|; this means solving the following optimal control problem
\begin{aligned} \textrm{minimize:} \qquad & -x(1) \\ \textrm{subject to:} \qquad & x' = v, \qquad v' = \frac F m \\ & x(0) = 0 \\ & v(0) = 0 \qquad v(1) = 0 \\ & F \in [-1,1] \end{aligned}
In order to solve this problem it’s necessary to determine the Hamiltonian
\mathcal{H}(x,v,\lambda_1,\lambda_2, F,t) = \lambda_1 v + \lambda_2 \frac F m
and the utility function
\mathcal{B}(x_a,v_a,x_b,v_b,\mu_1,\mu_2,\mu_3) = -x_b + \mu_1 x_a + \mu_2 v_a + \mu_3 v_b
The adjoint equation so becomes
\lambda_1' = - \frac{\partial\mathcal{H}}{\partial x} = 0
and
\lambda_2' = - \frac{\partial\mathcal{H}}{\partial v} = - \lambda_1
Among the adjoint boundary conditions, only -1-\lambda_1(1)=0 constrains the state–costate trajectory; the other equations determine the multipliers \mu_i. The boundary-value problem therefore becomes
\left\{ \begin{aligned} & x' = v \\ & v' = F/m \\ & \lambda_1' = 0 \\ & \lambda_2' = - \lambda_1 \\ & -1-\lambda_1(1)= 0\\ & x(0) = v(0) = v(1) = 0 \\ & F(t) = \underset{\overline F \in [-1,1]}{\textrm{argmin}} \left\{ \lambda_1(t) v + \lambda_2(t) \frac {\overline F}m \right\} \end{aligned} \right.
In this case in order to compute the argument minimum associated to the Pontryagin principle we have to consider only the term related to \overline F, and in particular this means
\begin{aligned} F(t) & = \underset{\overline F \in [-1,1]}{\textrm{argmin}} \left\{ \lambda_2(t) \frac {\overline F}m \right\} \\& = \begin{cases} + 1 & \textrm{if } \lambda_2(t) < 0 \\ - 1 & \textrm{if } \lambda_2(t) > 0 \\ [-1,1] \qquad & \textrm{if } \lambda_2(t) = 0 \\ \end{cases} \\ & = - \textrm{sign} \lambda_2(t) \end{aligned}
and so the simplified system that determines the solution becomes
\left\{ \begin{aligned} & x' = v \\ & v' = -\frac{\textrm{sign}\lambda_2}{m} \\ & \lambda_1' = 0 \\ & \lambda_2' = - \lambda_1 \\ & \lambda_1(1)= -1\\ & x(0) = v(0) = v(1) = 0 \end{aligned} \right.
By integration it can be seen that \lambda_1 can be regarded as a constant c_1 that, for the adjoint boundary condition, is equal to -1 and so \lambda_2 is in the form
\lambda_2 = t + c_2
From now on the solution becomes analytically complex, however it can be found that c_2 = -\frac{1}{2} determining the solution
F(t) = \begin{cases} 1 \qquad & t \leq \frac 1 2 \\ -1 & t > \frac 1 2 \end{cases}
Example 18.3 (optimal control problem) Given the optimal control problem
\begin{aligned} \textrm{minimize:} \qquad & \mathcal{F} (x,u) = \int_0^2 2x- u^2 \, \mathrm{d}t \\ \textrm{subject to:} \qquad & x' = 1-2u \\ & x(0) = 1 \qquad x(2) = 0 \end{aligned}
the solution can be obtained by computing the hamiltonian \mathcal{H} and the utility function \mathcal{B} related to the problem:
\begin{aligned} \mathcal{H}(x,u,\lambda) & = 2x - u^2 - \lambda \big(1-2u\big) \\ \mathcal{B}(x_0,x_2, \mu_1, \mu_2) & = - \mu_1 \big(x_0-1) - \mu_2 x_2 \end{aligned}
With that stated, the lonely resulting co-equation is \lambda' = \frac{\partial\mathcal{H}}{\partial x} = 2, determining so that the solution of \lambda(t) is in the form 2t + c_1 (where c_1 is a constant depending from the boundary conditions). The adjoint boundary conditions aren’t useful to determine the solution because they trivially solves the parameters \mu_1,\mu_2. We still have the control equation that states
\frac{\partial\mathcal{H}}{\partial u} = -2u + 2\lambda = 0 \qquad \implies\quad u(t) = \lambda(t)
Using the original ordinary differential equation we have that x' = 1 - 4t - 2c_1 that by integration becomes
x(t) = -2t^2 + \big(1-2c_1\big) t + c_2
From the boundary condition x(0) = 1 we obtain c_2 = 1, while considering x(2) = 0 we have c_1 = -5/4 and so the final control law is
u(t) = 2t - \frac{5}{4}
Example 18.4 (optimal control problem) To solve the optimal control problem
\begin{aligned} \textrm{minimize:} \qquad & \int_0^1 t + x - y^2 \, \mathrm{d}t \\ \textrm{subject to:} \qquad & x' = x+y \qquad y' = x + yu \\ & \int_0^1\sin y\, \mathrm{d}t = 3\\ & x(0) = 1 \qquad y(1) = 0 \\ & u \in [-2,3] \end{aligned}
the first thing to do is removing the integral constraint by transforming him in a differential equation with 2 boundary condition as follows:
\begin{aligned} \textrm{minimize:} \qquad & \int_0^1 t + x - y^2 \, \mathrm{d}t \\ \textrm{subject to:} \qquad & x' = x+y \\ & y' = x + yu \\ & z' = \sin y \\ & x(0) = 1, \qquad & z(0) = 0 \\ & y(1) = 0, \qquad & z(1) = 3 \\ & u \in [-2,3] \end{aligned}
The associated hamiltonian \mathcal{H} and utility function \mathcal{B} of the problem are
\begin{aligned} \mathcal{H}(x,y,z,\lambda_1, \lambda_2, \lambda_3) & = t + x - y^2 \\ & - \lambda_1 (x+y) - \lambda_2 (x+yu) - \lambda_3 (\sin y) \\ \mathcal{B}(x_0, y_1, z_0, z_1, \mu_{1\dots 4}) & = -\mu_1 \big(x_0 - 1\big) - \mu_2 y_1 \\ &- \mu_3 z_0 - \mu_4 \big(z_1 -3\big) \end{aligned}
With that stated the co-equations are computed as
\begin{aligned} \lambda_1' & = - \frac{\partial\mathcal{H}}{\partial x} = -1 + \lambda_1 + \lambda_2 \\ \lambda_2' & = - \frac{\partial\mathcal{H}}{\partial y} = 2y + \lambda_1 + \lambda_2 u + \lambda_3 \cos y \\ \lambda_3' & = - \frac{\partial\mathcal{H}}{\partial z} = 0 \end{aligned}
Evaluating the adjoint boundary conditions for the point that present no explicit constraints (so x_1, y_0) we retrieve \lambda_1(1) = 0 and \lambda_2(0) = 0. With the definition of the control equation \frac{\partial\mathcal{H}}{\partial u} = - \lambda_2 y the resulting boundary value problem of the optimal control one is
\begin{cases} x' = x+y \\ y' = x + yu \\ z' = \sin y \\ \lambda_1' = \lambda_1 + \lambda_2 - 1 \\ \lambda_2' = 2y + \lambda_1 + \lambda_2 y + \lambda_3 \cos y \\ \lambda_3' = 0 \\ x(0) = 1 \qquad \lambda_1(1) = 0 \\ y(1) = 0 \qquad \lambda_2(0) = 0 \\ z(0) = 0 \qquad z(1) = 3 \\ -\lambda_2 = 0 \\ u = \underset{\overline u \in [-2,3]}{\textrm{argmin}} \left\{ -\lambda_2 y \overline u \right\} \end{cases}
where the solution of the Pontryagin minimum principle is
u(t) = \begin{cases} -2 \qquad & \lambda_2 y < 0 \\ 3 & \lambda_2 y \geq 0 \end{cases}
Example 18.5 (optimal control problem) Given the optimal control problem
\begin{aligned} \textrm{minimize:} \qquad & \mathcal{F} (x,y,u) = x(0) + \int_0^1 x^2 + y^2 + u \, \mathrm{d}t \\ \textrm{subject to:} \qquad & x' = y - u, \qquad y' = x+u \\ &x(1) = 2 \\ & u(t) \in [-1,2] \end{aligned}
the solution can be obtained by firstly computing the hamiltonian \mathcal{H} and the utility function \mathcal{B} of the problems resulting in
\begin{aligned} \mathcal{H}\big(x,y,u,\lambda_1,\lambda_2\big) & = x^2 + y^2 + u + \lambda_1 \big(y-u\big) + \lambda_2 \big(x + u\big) \\ & = x^2 + y^2 + u \big(1 - \lambda_1 + \lambda_2\big) + \lambda_1 y + \lambda_2 x \\ \mathcal{B}\big(x_0,x_1, \mu\big) & = x_0 + \mu \big( x_1 - 2\big) \end{aligned}
We can so explicit: the adjoint ordinary differential equations
\begin{aligned} \lambda_1' &= -\frac{\partial\mathcal{H}}{\partial x} = - 2x - \lambda_2 \\ \lambda_2' &= - \frac{\partial\mathcal{H}}{\partial y} = -2y - \lambda_1 \end{aligned}
the adjoint boundary conditions
\begin{aligned} \lambda_1(0) & = - \frac{\partial\mathcal{B}}{\partial x_0} = - 1 \\ \lambda_1(1) &= \frac{\partial\mathcal{B}}{\partial x_1} = \mu \\ \lambda_2(0) & = -\frac{\partial\mathcal{B}}{\partial y_0} = 0 \\ \lambda_2(1) & = \frac{\partial\mathcal{B}}{\partial y_1} = 0 \end{aligned}
and the control equation
\frac{\partial\mathcal{H}}{\partial u} = 1 - \lambda_1 + \lambda_2 = 0
With the addition of the original differential equation the boundary value problem becomes
\begin{cases} x' = y - u \\ y' = x+u \\ \lambda_1' = -2x - \lambda_2 \\ \lambda_2' = -2y - \lambda_1 \\ \lambda_1(0) = -1 \\ \cancel{\lambda_1(1) = \mu} \qquad \qquad \textrm{: trivially satisfied} \\ \lambda_2(0) = \lambda_2(1) = 0 \\ 1 + \lambda_1 - \lambda_2 = 0 \\ u = \underset{\overline u \in [-1,2]}{\textrm{argmin}} \left\{ \overline u \big(1 + \lambda_1 - \lambda_2\big) \right\} \end{cases}
In this case the solution of the Pontryagin minimum principle is reduced to the form
u(t) = \begin{cases} -1 \quad & 1 + \lambda_1 - \lambda_2 > 0 \\ 2 & 1 + \lambda_1 - \lambda_2 < 0 \end{cases}
18.2 Problems with linear controls and singular arc
In this section, we examine problems in which the Hamiltonian is a linear function of the control variable u. Although linear problems are generally expected to be easier to handle, this is not always the case. The complexity arises from the fact that the necessary condition for minimizing the Hamiltonian with respect to the control along an optimal trajectory does not yield a well-defined expression for synthesizing the optimal control.
Another challenge associated with this class of problems is the occurrence of discontinuities in the control, which can be difficult to identify. Additionally, there may be an infinite number of switching points that can accumulate at a specific time instant. Given that general results and theorems regarding existence and uniqueness are limited in this context, it becomes necessary to introduce additional relationships by manipulating other necessary conditions.
Consequently, problems with a Hamiltonian that is linear in terms of the control u represent an almost independent area of research within optimal control theory. Most theoretical results in this domain are derived using a geometric approach, involving fiber bundles and symmetry groups rather than traditional variational techniques.
To maintain clarity and simplicity in notation, we will focus on problems with a single control variable and fixed endpoints. The formulation of these problems is as follows:
Definition 18.1 Minimize the cost functional J subject to the dynamic \bm{x}' with the control u constrained in |u|\leq 1 and the initial state \bm{x}(0)=\bm{x}_0, where
\begin{aligned} J(u)&= \int_0^T f_0(\bm{x})+b_0(\bm{x})u \,\mathrm{d}t\\ \bm{x}'(t) &= \bm{f}(\bm{x})+\bm{b}(\bm{x})u\qquad \mathrm{i.e.} \end{aligned} \tag{18.2}
f_0(\bm{x}) and b_0(\bm{x}) are scalar functions of the state \bm{x}, u is the scalar control.
To further simplify the computations, we consider the scalar equation of the functional as an extra state variable posing
x_0'= f_0(\bm{x})+b_0(\bm{x})u, \qquad x_0(0)=0,
(we have already seen that this is not a loss of generality), but the key point here is that the control appears only linearly in both dynamics and cost functional. We can write the Hamiltonian for this system, namely
\mathcal{H}(\bm{x},\bm{\lambda},t) = \sum_{i=0}^n f_i(\bm{x})\lambda_i+u\sum_{i=0}^n b_i(\bm{x})\lambda_i.
It is convenient to rename the two addends as
\begin{aligned} A(\bm{x},\bm{\lambda},t)&=\sum_{i=0}^n f_i(\bm{x})\lambda_i,\\ B(\bm{x},\bm{\lambda},t)&=\sum_{i=0}^n b_i(\bm{x})\lambda_i, \end{aligned}
so that the Hamiltonian becomes
\mathcal{H}(\bm{x},\bm{\lambda},t) = A(\bm{x},\bm{\lambda},t)+B(\bm{x},\bm{\lambda},t)u. \tag{18.3}
The adjoint variables of the costate are given by the usual relation
\lambda_i' = -\dfrac{\partial\mathcal{H}}{\partial x_i}= -\sum_{j=0}^n\lambda_j\dfrac{\partial f_j(\bm{x})}{\partial x_i}-u\sum_{j=0}^n\lambda_j\dfrac{\partial b_j(\bm{x})}{\partial x_i}.
We state now the necessary conditions for problem Equation 18.2.
Theorem 18.1 If u^\star(t) is an optimal control and if \bm{x}^\star(t) is the corresponding optimal trajectory, then there are a costate \lambda_i^\star(t) such that
for i=1,\ldots,n \begin{aligned} x_{i}'^{\star} &= f_i(\bm{x}^{\star})+b_i(\bm{x}^{\star})u^{\star},\\ \lambda_i'^{\star} &= -\sum_{j=0}^n\lambda_j^\star\dfrac{\partial f_j(\bm{x}^\star)}{\partial x_i^\star}- u^\star\sum_{j=0}^n\lambda_j^\star(t)\dfrac{\partial b_j(\bm{x}^\star)}{\partial x_{i}^\star}; \end{aligned}
for all t\in[0,T] and all u(t) s.t. |u(t)|\leq 1, holds A(\bm{x}^\star,\lambda^\star,t)+u^\star(t)B(\bm{x}^\star, \lambda^\star,t)\leq A(\bm{x}^\star,\lambda^\star,t)+u(t)B(\bm{x}^\star,\lambda^\star,t);
If T is free, then for all t\in[0,T]
\mathcal{H}^\star(\bm{x}^\star, \bm{\lambda}^\star,t) := A(\bm{x}^\star, \bm{\lambda}^\star,t)+u^\star(t)B(\bm{x}^\star, \bm{\lambda}^\star,t)=0 , \tag{18.4}
while, if T is fixed, then
\mathcal{H}^\star(\bm{x}^\star, \bm{\lambda}^\star,t) := A(\bm{x}^\star, \bm{\lambda}^\star,t) +u^\star(t)B(\bm{x}^\star, \bm{\lambda}^\star,t)=c , \tag{18.5} for a real constant c.
It follows easily from equation Equation 18.3 with the PMP, that the optimal control is given by
u^\star(t)=-\mathrm{sign}\{B(\bm{x}^\star, \bm{\lambda}^\star,t)\}=-\mathrm{sign}\left\{\sum_{i=0}^n b_i(\bm{x}^\star(t))\lambda^\star_i(t)\right\},
as long as the function B(\bm{x}^\star,\bm{\lambda}^\star,t) is not zero. If B becomes zero, then the sign function is not defined. If B is zero only in a point we speak of bang-bang controls, if B is identically zero for an entire interval t\in(t_1,t_2], the controls are called singular. In this case, the associated trajectory \bm{x}(t) is called a singular arc.
Singular controls
If problem Equation 18.2 results singular, controls, costate and trajectory have the following property: there is at least one half-open interval (t_1,t_2]\subset[0,T] such that
\sum_{i=0}^n \lambda^\star_i(t)b_i(\bm{x}^\star(t))=0, \qquad t\in(t_1,t_2]. \tag{18.6}
As before, the existence of an extremal singular control, does not imply that the optimal control is singular: we need additional information as uniqueness to conclude its optimality. The function Equation 18.6 is often called switching function.
Test for singular controls
Suppose we are in the free terminal time case, so that Equation 18.4 holds, we test if it is possible to have a singular control as follows. First we need to assume that the switching function is zero in an unknown interval (t_1,t_2], then, because of the free end time hypothesis, for t\in(t_1,t_2],
\begin{aligned} \mathcal{H}(\bm{x},\bm{\lambda},t) = \sum_{i=0}^n f_i(\bm{x})\lambda_i+u\sum_{i=0}^n b_i(\bm{x})\lambda_i &= 0\\ & \Downarrow\\ \sum_{i=0}^n f_i(\bm{x})\lambda_i&=0. \end{aligned}
Moreover, for each k\in\mathbb{N}, we have that
\dfrac{\mathrm{d}^k}{\mathrm{d}t^k} \sum_{i=0}^n \lambda_i(t)b_i(\bm{x})=0, \quad \left\{\begin{aligned} k & =1,2,3,\ldots, \\ t & \in(t_1,t_2], \end{aligned}\right. \tag{18.7}
and similarly,
\dfrac{\mathrm{d}^k}{\mathrm{d}t^k}\sum_{i=0}^n \lambda_i f_i(\bm{x})=0, \quad \left\{\begin{aligned} k & =1,2,3,\ldots, \\ t & \in(t_1,t_2], \end{aligned}\right. \tag{18.8}
However, the canonical equations (we omit explicit time dependence) are
\begin{aligned} x_i' &= f_i+ub_i\\ \lambda_i' &= -\sum_{j=0}^n \lambda_j(t)\dfrac{\partial f_j}{\partial x_i} -u \sum_{j=0}^n \lambda_j \dfrac{\partial b_j}{\partial x_i}, \quad \left\{\begin{aligned} i,j & =1,2,\ldots,n \\ k & \in\Bbb{N}, \end{aligned}\right. \end{aligned}
Let k=1 in Equation 18.7, then by the chain rule we have
\dfrac{\mathrm{d}}{\mathrm{d}t}\sum_{i=0}^n \lambda_i b_i = \sum_{i=0}^n\left(\lambda_i'b_i+\lambda_i \sum_{j=0}^n \dfrac{\partial b_j}{\partial x_i}x'_j\right)=0.
Substituting the canonical equations in the previous one, yields
\sum_{i=0}^n\sum_{j=0}^n\left( f_j\lambda_i\dfrac{\partial b_i}{\partial x_j}-b_i\lambda_j\dfrac{\partial f_j}{\partial x_i}\right) +u \sum_{i=0}^n\sum_{j=0}^n\left( \lambda_i b_j\dfrac{\partial b_i}{\partial x_j}- \lambda_j b_i\dfrac{\partial b_j}{\partial x_i}\right)=0.
We notice that the coefficient of u is zero, and we conclude that
\sum_{i=0}^n\sum_{j=0}^n\lambda_i \left( b_j\dfrac{\partial f_i}{\partial x_j}-f_j\dfrac{\partial b_i}{\partial x_j}\right)=0. \tag{18.9}
Applying the same argument to equation Equation 18.8, gives
\begin{aligned} 0 &= \dfrac{\mathrm{d}}{\mathrm{d}t}\sum_{i=0}^n \lambda_if_i(\bm{x}) \\ &= \sum_{i=0}^n\left(\lambda_i'f_i+\lambda_i\sum_{j=0}^n\dfrac{\partial f_i}{\partial x_j}x_j'\right)\\ &= \sum_{i=0}^n\sum_{j=0}^n\left( \lambda_if_i\dfrac{\partial f_i}{\partial x_j}-\lambda_jf_i\dfrac{\partial f_j}{\partial x_i}\right) +u \sum_{i=0}^n\sum_{j=0}^n\left( \lambda_i b_j\dfrac{\partial f_i}{\partial x_j}- \lambda_j f_i\dfrac{\partial b_j}{\partial x_i}\right)\\ &= u \sum_{i=0}^n\sum_{j=0}^n\lambda_i\left( b_j\dfrac{\partial f_i}{\partial x_j}-f_j\dfrac{\partial b_i}{\partial x_j}\right)=0. \end{aligned}
This last line implies that either u=0 or
\sum_{i=0}^n\sum_{j=0}^n\lambda_i\left( b_j\dfrac{\partial f_i}{\partial x_j}-f_j\dfrac{\partial b_i}{\partial x_j}\right)=0.
Noticing that the previous line is equal to equation Equation 18.9, we can not conclude that u=0, and this is true for all k\in\mathbb{N}.\ In conclusion, a necessary but not sufficient test for an extremal singular control is that in the interval (t_1,t_2] the following relations are satisfied: \left.\begin{aligned} \displaystyle\sum_{i=0}^n b_i\lambda_i=0\\ \displaystyle\sum_{i=0}^n f_i\lambda_i=0\\ \displaystyle\sum_{i=0}^n\sum_{j=0}^n\lambda_i\left( b_j\dfrac{\partial f_i}{\partial x_j}-f_j\dfrac{\partial b_i}{\partial x_j}\right)=0 \end{aligned}\right\}
We show now how to compute the explicit expression for the singular control, we return back to the equation of the switching function Equation 18.7 and let k=2,3,\ldots. It is shown in [1:pag.487] that those derivatives for k=2,3,\ldots require extensive manipulations but have all the following same structure:
\sum_{i=0}^n\lambda_i\psi_{ki}(\bm{x})+u\sum_{i=0}^n\lambda_i\phi_{ki}(\bm{x})=0.
If \sum \lambda_i\phi_{ki}(\bm{x})=0 we can perform successive derivatives of Equation 18.7, in general there will be a finite value of k=m for which \sum \lambda_i\phi_{mi}(\bm{x})\neq 0 and then the singular control u can be solved:
u= -\dfrac{\displaystyle\sum_{i=0}^n\lambda_i\psi_{mi}(\bm{x})}{\displaystyle\sum_{i=0}^n\lambda_i\phi_{mi}(\bm{x})}.
Up to now we have dealt with free final time, if the final time is fixed the we have to take in account the presence of the constant c in the Hamiltonian, but Equation 18.7 and Equation 18.8 still hold and hence also the conclusions.\ In practical problems, we must check if
\sum \lambda_i\phi_{ki}(\bm{x})=0
for all k=1,\ldots,m-1 and
\sum \lambda_i\phi_{mi}(\bm{x})\neq 0
If any of the relations are violated then this represent a violation of the necessary conditions and so singular controls can not occur, if these conditions are all met, then there may be a singular extremal control.