AIMS Series on Applied Mathematics
Volume 2
Introduction to the
Mathematical Theory of Control
Alberto Bressan and Benedetto Piccoli
With 102 figures and 107 exercises
A 1 M s American Institute of Mathematical Sciences

EDITORIAL COMMITTEE
Editor in Chief: Alberto Bressan (USA)
Members: H. Mete Soner (Turkey), Eitan Tadmor (USA),
Luigi Ambrosio (Italy), Peter S. Constantin (USA).
Alberto Bressan
Department of Mathematics
Penn State University
University Park, Pa. 16802 USA
E-mail: bressan@math.psu.edu
Benedetto Piccoli
Istituto per le Applicazioni del Calcolo ’’Mauro Picone”
Viale del Policlinico 137
00161 Roma (Italy)
E-mail: b.piccoli@iac.cnr.it
AMS 2000 subject classifications: 49J15, 49J30, 49N05, 49N25, 93B05, 93B52,
70Q05, 34K35, 35B37
ISBN-10: 1-60133-002-2; ISBN-13: 978-1-60133-002-4
© 2007 by the American Institute of Mathematical Sciences. All rights reserved.
This work may not be translated or copied in whole or part without the written
permission of the publisher (AIMS, P.O. Box 2604, Springfield, MO 65801-2604,
USA), except for brief excerpts in connection with reviews or scholarly analysis.
Use in connection with any form of information storage and retrieval, electronic
adaptation, computer software, or by similar or dissimilar methodology now known
or hereafter developed is forbidden.
The use in this publication of trade names, trademarks, service marks, and
similar terms, even if they are not identified as such, is not to be taken as an
expression of opinion as to whether or not they are subject to proprietary rights.
Printed in the United States of America.
aimsciences.org

To Wen and Alessia

PREFACE
The present book originated from a set of lecture notes developed by the
first author, at S.I.S.S.A. and at Penn State University. Its primary aim is
to provide an introduction to the mathematical theory of nonlinear control
systems.
Care has been taken to make the exposition as self-contained as possible.
A preliminary chapter covers the basic theory of O.D.E’s with coefficients
measurable w.r.t. time, while an extended appendix collects several results
from functional analysis, geometry and measure theory. All this background
material could be previously found scattered in the literature. Readers will
find these results conveniently collected, and supplied with concise proofs.
The theory of finite-dimensional, deterministic control has been largely
developed in the years 1960-1980’s, and has now reached a “mature” stage.
This classical theory makes the content of Chapters 1 to 8.
After the introduction and a review of O.D.E. theory, Chapter 3 develops
the main concepts and properties of nonlinear control systems. In particular,
we discuss the relationship between control systems and differential inclusions,
the properties of the control-to-trajectory map, the structure of reachable
sets, and various local and global controllability results. The last two sections
introduce the notion of chattering controls, and provide a proof of the well
known “bang-bang theorem” for linear systems.
Chapter 4 contains some basic results on asymptotic stabilization. We
first review the method of Lyapunov functions, to analyze the stability of
dynamical systems described by O.D.E’s. Then we discuss the existence of
a stabilizing feedback control, in the case of a linear control system. The
last section contains a result on the local stabilization of non-linear systems,
obtained by a linearization method.
The next chapters deal with optimal control problems. Chapter 5 intro-
duces the main types of optimization problems, with terminal cost and with
running cost. We prove here some fundamental results on the existence of
optimal controls, under two different types of assumptions: either the system
VIII
has linear dynamics, or, in the nonlinear case, we assume that the sets of
admissible velocities are convex.
The famous Pontryagin Maximum Principle is discussed at length in Chap-
ter 6, together with other related necessary conditions for optimality. Proofs
of the basic theorems are supplemented by several pictures, together with
various examples where optimal controls can be explicitly computed.
The following two chapters deal with sufficient conditions for optimal-
ity. In Chapter 7 we derive the first order P.D.E. of dynamic programming,
describing how the minimal cost changes as a function of the initial state.
Assuming the regularity of this “optimal value function”, or the existence of a
regular feedback synthesis, one derives conditions which guarantee the global
optimality of a given trajectory.
Chapter 8 provides a concise introduction to the theory of viscosity so-
lutions for Hamilton-Jacobi equations. A key result states that, even with
minimal regularity assumptions, the optimal value function can be character-
ized as the unique viscosity solution to the appropriate H-J equation. This
represents an alternative way to obtain sufficient conditions for optimality.
The last two chapters present some recent topics which were never before
included in a textbook. These can be used as optional material, providing a
flavor of current research.
Chapter 9 develops the theory of patchy feedbacks for asymptotic sta-
bilization and optimal control. After some basic definitions and examples,
we describe the construction of stabilizing feedbacks, prove their robustness
w.r.t. inner and outer perturbations, and show the existence of nearly opti-
mal patchy feedbacks. Considerable effort was made here in order to polish the
exposition, resulting in much shorter and more transparent proofs, compared
with the original papers [2], [3], and [4].
Finally, Chapter 10 contains an introduction to the theory of impulsive
control of Lagrangian systems. This theory, initiated in the 1980’s indepen-
dently by Aldo Bressan [20] and Charles Marie [66], is concerned with me-
chanical systems which are controlled not by applying external forces but by
implementing some frictionless constraints. In this case, the equations of mo-
tion contain also the time derivative of the control function. When this control
is discontinuous, the motion thus has an impulsive character.
A key issue here is to understand what is the correct dynamics corre-
sponding to discontinuous controls, and how to reduce these systems to a
more tractable form, such as a differential inclusion. When the derivative of
the control enters linearly, under a crucial commutativity assumption one can
follow a construction of H. Sussmann [83] and “integrate” the equations, thus
eliminating the singularities.
On the other hand, without commutativity assumptions, the trajectory de-
termined by a discontinuous control t i—> u(t) is not uniquely defined. Indeed,
at each time т where и has a jump, one should additionally specify a contin-
uous path joining the left and right limits u(t—), u(t+). This leads to the
basic concept of “graph completion” introduced in [16], which has now found
IX
diverse applications also outside the realm of control theory [37]. In a last
section, for systems where the derivative of the control enters also quadrati-
cally, we show that the dynamics can be described by a suitable differential
inclusion.
In the References, we made no attempt to collect the extremely vast lit-
erature published on the theory of control during the past 50 years. Our list
is thus restricted to some major treatises, and to a small number of seminal
papers which may claim historical relevance. An exception is made for the
last two chapters, where we quote several recent papers on topics of current
research.
The book can be used for a one- or two-semesters course on control theory,
at a beginning graduate level. Having only one semester at disposal, one can
cover the Introduction, Chapters 3-6 (possibly skipping the more advanced
sections 3.8-3.10), and sections 7.3-7.4 in Chapter 7. Results from O.D.E.
theory given in Chapter 2, as well as the background material collected in
the Appendix, can be presented during the course, when needed. Chapters 9
and 10 may provide topics for individual students’ projects, at the end of the
semester.
The text is supplemented by a large collection of figures, which help the
reader understanding the key geometric ideas and building intuition. Several
homework problems are listed at the end of each chapter.
For science or engineering students, this book provides a richly illustrated
overview of the basic techniques and results in the theory of nonlinear control.
More mathematically oriented students can use this text as a useful introduc-
tion, before tackling more advanced monographs on geometric control theory
[1], [56], or the theory of control for infinite-dimensional systems, described
by partial differential equations [63], [64], [65].

Contents
1	Introduction....................................................... 1
2	Review of Differential Equations.................................. 13
2.1	Fundamental theory........................................... 14
2.2	Linear systems............................................... 21
2.3	Differentiability with respect to initial data............... 26
2.4	A transversality theorem..................................... 30
Pro	blems ...................................................... 32
3	Control Systems................................................... 35
3.1	An equivalent differential inclusion......................... 36
3.2	Fundamental properties of trajectories ...................... 37
3.3	Closure ..................................................... 44
3.4	Density ..................................................... 47
3.5	Reachable sets............................................... 51
3.6	Linear systems............................................... 56
3.7	Local controllability of nonlinear systems................... 59
3.8	Lie brackets and controllability............................. 61
3.9	Chattering controls.......................................... 65
3.10	The Bang-Bang theorem....................................... 67
Prob	lems ...................................................... 69
4	Asymptotic stabilization.......................................... 75
4.1	Lyapunov stability........................................... 75
4.2	Stabilization of linear control systems...................... 79
4.3	Stabilization of nonlinear systems........................... 83
Pro	blems ...................................................... 85
5	Existence of Optimal Controls..................................... 87
5.1	Mayer problems............................................... 87
5.2	The problem of Bolza......................................... 93
XII
Contents
Problems....................................................... 95
6	Necessary conditions .......................................... 99
6.1	The Mayer problem with free terminal	point................100
6.2	Computation of optimal controls...........................104
6.3	The Mayer problem with terminal constraints...............110
6.4	Variable terminal time....................................115
6.5	The problem of Bolza......................................119
6.6	Linear-quadratic optimal control..........................125
Problems.......................................................127
7	Sufficient	Conditions........................................133
7.1	Existence T PMP...........................................134
7.2	Convexity + PMP...........................................135
7.3	Dynamic Programming.......................................137
7.4	Relations between the P.M.P.
and the P.D.E. of Dynamic Programming.....................148
7.5	Linear-quadratic case.....................................150
7.6	Optimal syntheses.........................................154
Problems.......................................................161
8	Viscosity solutions for Hamilton-Jacobi equations..............165
8.1	The method of characteristics.............................166
8.2	One-sided differentials...................................170
8.3	Viscosity solutions.......................................174
8.4	Stability properties......................................176
8.5	Comparison theorems.......................................178
8.6	Dynamic programming (revisited)...........................185
8.7	The Hamilton-Jacobi-Bellman equation .....................189
8.8	Infinite horizon problems.................................192
Problems.......................................................197
9	Patchy Feedbacks...............................................199
9.1	Patchy vector fields......................................201
9.2	Asymptotic feedback stabilization.........................206
9.3	Robustness................................................211
9.4	Nearly optimal patchy feedbacks ..........................219
Problems.......................................................229
10	Impulsive Control Systems .....................................233
10.1	Mechanical systems controlled by moving constraints......235
10.2	Generalized trajectories for commuting vector fields.....241
10.3	The non-commutative case: graph completions..............247
10.4	Systems with quadratic impulses..........................252
10.5	Optimization problems for commutative impulsive systems. . . . 257
Contents XIII
Problems.....................................................259
A Appendices.....................................................263
A.l Normed spaces............................................263
A.2	Banach’s contraction mapping theorem....................265
A.3	Brouwer’s fixed point theorem ..........................267
A.4	A compactness theorem...................................273
A.5	Review of Lebesgue measure theory.......................274
A.6	Differentiability of Lipschitz continuous functions.....277
A.7	Multifunctions..........................................279
A.8	Convex sets.............................................283
A.9	Convex cones............................................288
A. 10 Lie brackets and Frobenius’ theorem....................294
Problems.....................................................300
References.......................................................305
Index............................................................311

1
Introduction
Since the beginnings of Calculus, differential equations have provided an effec-
tive mathematical model for a wide variety of physical phenomena. Consider
a system whose state can be described by a finite number of real-valued pa-
rameters, say x = (xi, ••• If the rate of change x = dx/dt is entirely
determined by the state x itself, then the evolution of the system can be
modelled by the ordinary differential equation
i = g(x').	(1.1)
If the state of the system is known at some initial time to, the future behavior
for t > to can then be determined by solving a Cauchy problem, consisting of
(1.1) together with the initial condition
nr(to) = ^o-	(1-2)
We are here taking a spectator’s point of view: the mathematical model al-
lows us to understand a portion of the physical world and predict its future
evolution, but we have no means of altering its behavior in any way. Celestial
mechanics provides a typical example of this situation. We can accurately cal-
culate the orbits of moons and planets and exactly predict time and locations
of eclipses, but we cannot change them in the slightest amount.
Control theory provides a different paradigm. We now assume the pres-
ence of an external agent, i.e. a “controller”, who can actively influence the
evolution of the system. This new situation is modelled by a control system,
namely
x = f(x,u),	(1.3)
where U is a family of admissible control functions. In this case, the rate of
change x(t) depends not only on the state x itself, but also on some external
parameters, say и = (ui, • • • , um), which can also vary in time. The control
function n(-), subject to some constraints, will be chosen by a controller in
order to modify the evolution of the system and achieve certain preassigned
2
1 Introduction
goals — steer the system from one state to another, maximize the terminal
value of one of the parameters, minimize a certain cost functional, etc...
In a standard setting, we are given a set of control values U C lRm. The
family of admissible control functions is defined as
Z7 = < u : IR i—> IRm ; и measurable, u(t) G U for a.e. t .	(1-4)
The system (1.1) can then be written as a differential inclusion, namely
x G Т’(ж)
where the set of possible velocities is given by
F(x) = {y; y = f(x, u) for some и G U
(1-5)
(1.6)
Clearly, every admissible trajectory of the control system (1.3) is also a solu-
tion of (1.5). Under some regularity assumptions on f, it turns out that the
converse is also true: given any absolutely continuous trajectory t »—> x(t) of
(1.5), one can select a measurable control function t >—► u(t) G U such that
i(t) = /(x(f),u(f))
at almost every time t. Differential inclusions often provide a convenient ap-
proach for the analysis of control systems.
Fig. 1.1. A differential equation vs. a differential inclusion.
Figure 1.1 illustrates the basic difference between an O.D.E and a differen-
tial inclusion. In the first case, we have a deterministic model: to each initial
state Xq there corresponds one single trajectory t h-> x(t). On the other hand,
the evolution described by (1.5) is non-deterministic. Given an initial state
#0, several different trajectories t1—► x(t) are possible.
Remark 1.1 Differential inclusions are sometimes used as non-deterministic
models, when the future behavior of a system cannot be predicted due to lack
of information. It should be clear, however, that is not the point of view of
1 Introduction
3
control theory. Here the non-determinacy reflects the possible different strate-
gies of a rational controller, who will make his choices in order to achieve a
specific goal.
The control law can be assigned in two basically different ways. In “open
loop” form, as a function of time: t u(t), and in “closed loop” or feedback,
as a function of the state: x i—► ufx). Implementing an open loop control
и = u(t) is in a sense easier, since the only information needed is provided
by a clock, measuring time. On the other hand, to implement a closed loop
control и = u[x) one constantly needs to measure the state x of the system.
Designing a feedback control, however, yields some distinct advantages. In
particular, feedback controls can be more robust in the presence of random
perturbations. For example, assume that we seek a control u(-) which steers
the system from an initial state P to the origin. If the behavior of the system is
exactly described by (1.1), this can be achieved, say, by the open loop control
t i—> u(t). In many practical situations, however, the evolution is influenced
by additional disturbances which cannot be predicted in advance. The actual
behavior of the system will thus be governed by
x = f(x,u) +	(1.7)
where t i—► //(£) is a perturbation term. In this case, if the open loop control
и = u(t) steers the system (1.1) to the origin, this same control function may
not accomplish the same task in connection with (1.7), when a perturbation is
present. In Figure 1.2 (left) the solid line depicts the trajectory of the system
(1.1), while the dotted line illustrates a perturbed trajectory x(•). We assumed
here that the disturbance tj(-) is active during a small time interval [£i, hs
presence puts the system “off course”, so that the origin is never reached.
Alternatively, one can solve the problem of steering the system to the
origin by means of a closed loop control. In this case, we would seek a control
function и = u(x) such that all trajectories of the O.D.E.
z = g(x) = f(x,u(x))	(1.8)
approach the origin as t —► oo. This approach is less sensitive to the presence
of external disturbances. As illustrated in Figure 1.2 (right), in the presence
of an external disturbance ??(•), the trajectory of the system does change, but
our eventual goal - steering the system to the origin would still be attained.
Various examples of control system are described below.
Example 1.1 (boat on a river). Consider a river with straight course.
Using a set of planar coordinates, assume that it occupies the horizontal strip
5 = {(a? 1,2:2) : — 00 <	< 00, — 1 < Х2 < 1}.
Moreover, assume that speed of the water is given by the velocity vector
v(ii,z2) = (1 -	0).
4
1 Introduction
Fig. 1.2. The effect of a perturbation on an open loop and on a feedback control.
If a boat on the river is merely dragged along by the current, its position
will be determined by the differential equation
(±1,±2) = (1 - ^2’ 0).
On the other hand, if the boat is powered by an engine, then its motion can
be modelled by the control system
(a?i,±2) = v + u = (1 - X2 T , u2),
(1.9)
where the vector u = (1x1,112) describes the velocity of the boat relative to the
water. The set U of admissible controls consists of all measurable functions
u : IR IR2 taking values inside the closed disc
U = < (а?], (J2) •
(1-Ю)
The constant M accounts for the maximum speed (in any direction) that can
be produced by the engine.
Given an initial condition (xi,#2)(0) = (±i,t2), solving (1.9) one finds
Xi(t) = Xi T t + / ui(s)ds— / {X2+ U2(r)drl ds,
Jo	Jo \ Jo /
#2 (*) = #2 + / 112(5) ds	(-1 < X2 < 1).
Jo
In particular, the constant control u = (1/1,112) = (—2/3,1) takes the boat
from a point (xi,— 1) on one side of the river to the point (^i,l) on the
opposite side, in two units of time. It is not difficult to show that if M > 0
the boat can be steered from any point P on the river to any other point Q.
Observe that for the system (1.9)-(1.10) the admissible trajectories coin-
cide with the solutions to the differential inclusion
1 Introduction
5
Fig. 1.3. Velocities of the water and of the boat.
(ii,dr2) € F(xi,x2) = < (yi,t/2) :	- 1 + ^)2 + y% < M
Example 1.	2 (fishery management). Consider a fish population living in
a lake. A simple model describing how its size x(t) varies in time is provided
by the O.D.E.
x = x(a — x).	(1-11)
Here the constant a describes the maximum sustainable amount of fish which
can be present in the lake.
Next, assume that some fish is harvested from the lake, at rate и = u(t).
For example, one may think of и as the number of fishermen active at time t.
In this case, the evolution of the fish population is described by
x = x(a — x) — xu.	(1.12)
This provides another example of a control system. In a realistic situation,
one may select the harvesting rate и = u(t) in order to maximize the total
amount of fish caught during a given time interval. Notice that if we adopt a
constant harvesting rate u(t) = й < о, the fish population will approach the
asymptotic limit x = a — й. As t oc, the choice u(t) = a/2 maximizes the
average amount of fish caught in unit time. Indeed
xu = (a — й\й = max (а—	.
cv>0
In several situations, the optimal harvesting of natural resources leads to
control problems of similar type.
Example 1.	3 (cart on a rail). Consider a cart which can move without
friction along a straight rail (Figure 1.4). For simplicity, assume that it has
unit mass. Let ?/(0) = у be its initial position and г;(0) = v be its initial
velocity. If no forces are present, its future position is simply given by
6
1 Introduction
y(t) = y-^-vt.
Next, assume that a controller is able to push the cart, with an external force
и = u(t). The evolution of the system is then determined by the second order
equation
y(t) = u(t).	(1.13)
Calling Xi (£) = y(t) and = v(t) respectively the position and the velocity
of the cart at time we can rewrite (1.13) as a first order control system:
(ii,i2) = (x2,u).
(1-14)
Given the initial condition a?i(0) = у, .r2(0) = v, solving (114) one finds
Assuming that the force satisfies the constraint
|u(t)| < 1,
the control system (1.14) is equivalent to the differential inclusion
(^1,^2) C F(x\,x2) = {(^2,tu); - 1 < w < 1} .
Fig. 1.4. A cart moving along a straight, frictionless rail.
We now consider the problem of steering the system to the origin. More
precisely, we want the cart to be at the origin with zero speed. For example,
if the initial condition is (7/, v) = (2,2), this goal is achieved by the open-loop
control
&(£) — 1
if 0 < t < 4,
if 4 < t < 6,
if t > 6.
A direct computation shows that (j?i (t), a^W) = (0?0) for t > 6. Notice, how-
ever, that the above control would not accomplish the same task in connection
1 Introduction
7
with any other initial data (i/, v) different from (2,2). This is a consequence
of the backward uniqueness of solutions to the differential equation (1.14).
A related problem is that of asymptotic stabilization. In this case, we seek
a feedback control function и = u(j?i,j?2) such that, for every initial data
(^, v), the corresponding solution of the Cauchy problem
(±1,£2) = (^2, и(Х1,Х2У),	(.rbz2)(0) = (у, v)
approaches the origin as t —> oo, i.e.
lim (rri,X2)(t) = (0,0).
t—>oc
There are several feedback controls which accomplish this task. For example,
one can take u(xi,X2) = —x\ — a?2-
Because of backward uniqueness, it is clear that there cannot be any Lip-
schitz continuous feedback и = u(xi,X2) which steers every initial condition
exactly to the origin within finite time. This goal, however, can be accom-
plished by the discontinuous feedback law
-1
The multifunction
«(^l,^) =
1
0
if X2 > 0, jq >	or if X2 < 0, Xi > #2/2,
if X2 < 0, x\ < x|/2 or if X2 > 0, x\ < -#2/2,
if x\ = X2 = 0.
(1.15)
F(xi,x2) = j (^2,^); w € [-1,1
and the trajectories of the corresponding equation
(±i, ±2) = (^2, u(zi,z2))
are shown in figure 1.5.
Fig. 1.5. A discontinuous feedback steering every initial point to the origin.
Example 1.	4 (car steering). We consider here a mathematical model de-
scribing the motion of a car in a large parking lot. At a given time, the position
8
1 Introduction
of a car is determined by three scalar parameters: the coordinates (re, y) of
its barycenter В € 1R2 and the angle 0 giving its orientation, as in Figure
1.6. The driver controls the motion of the car by acting on the gas pedal and
on the steering wheel. The control function thus has two scalar components:
speed u(t) of the car and the turning angle a(t). The motion is thus described
by the control system
' ii = и cos 0,
±2 = U sin f),	(1-16)
в -au.
It is reasonable here to impose bounds on speed of the car and on the steering
angle, say
u(t) E [—m, M],	o(t) 6 [—a, a].
A frequently encountered problem is the following: given the initial po-
sition, steer the car into a parking spot. The typical maneuver needed for
parallel parking is illustrated in Figure 1.6.
Fig. 1.6. Car parking.
In connection with a control system of the general form (1.3), a wide range
of mathematical questions can be formulated.
A first set of problems is concerned with the dynamics of the system. Given
an initial state ж, one would like to determine which other states x e IRn can
be reached using the various admissible controls utU. More precisely, given a
control function и = u(i), call 11—► a?(Z, u) t he solution to the Cauchy problem
±(t) = /(z(t), ?/(£)),	ж(0) = ж,
and define the reachable .set at time t as
R(t) = {x(t, u); и e Z/}.
For general nonlinear systems, explicit formulas describing R(t) are not avail-
able. However, one can analyze several topological and geometric properties
1 Introduction
9
of this reachable set. The closure, boundedness, convexity, and the dimension
of the set R(t) provide useful information on the control system.
In addition, it is interesting to study whether R(t) is a neighborhood of
the initial point x for all t > 0. In the positive case, the system is said to be
small time locally controllable at x. Another important case is when the union
of all reachable sets R(t) as t—->oc includes the entire space IR". We then say
that the system is globally controllable.
The dependence of the reachable set R(t) on the time t and on the set
of controls U is also a subject of investigation. For example, if U is defined
by (1.4), one may ask whether the same points in R(f) can be reached by
using controls which are piecewise constant, or take values within the set
of extreme points of U. Being able to perform the same tasks by means of
a smaller set of control functions, easier to implement, is quite relevant in
practical applications.
Different kind of problems arise in connection with controls in feedback
form. Here one basic goal is to construct a feedback control и = u(x) such
that the resulting dynamics determined by the differential equation (1.8) has
certain desired properties. For example, one could seek a control which steers
every initial state asymptotically toward the origin, or stabilizes the system
in a neighborhood of a periodic orbit, etc...
The regularity of the feedback control is often a major issue of investiga-
tion. Ideally, one would like the function x h-> ?i(x) to be smooth, or at least
continuous. However, for some nonlinear systems it turns out that certain
tasks cannot be accomplished by any continuous feedback law. This raises the
question of what kind of discontinuities can be allowed in a feedback control,
and how to interpret the solution to the resulting O.D.E. (1.8) when the right
hand side is a discontinuous function of the state x.
A further key issue related to feedback control is robustness. In general, the
differential equation (1.3) provides only an approximate description of reality.
External disturbances may affect the evolution of the system. Since these
cannot be predicted in advance, it is important to design a control such that
the system’s behavior will not be much affected by these small perturbations.
Continuous feedback laws are usually robust, but the problem can become
quite delicate when discontinuous feedbacks are implemented.
A second, very important area of control theory is concerned with opti-
mal control. In many applications, among all strategies which accomplish a
certain task, one seeks an optimal one, based on a given performance criterion.
In mathematical terms, a performance criterion can be defined by an integral
functional of the form
J= L{t,x,u)dt.	(1.17)
Jo
The value of J will have to be optimized among all admissible trajectories of
(1.3), with a number of initial or terminal constraints.
10
1 Introduction
For example, among all controls which steer the system from the initial
point x to some point on a target set J? at time T, we may seek the one that
minimizes the cost functional (1.17). This problem is formulated as
min I L(t,x,u)dt	(1.18)
Jo
subject to
x = f(t,x, u),	z(0) = x, x(T) e ft.	(1-19)
Observe that if (1.3)-(1.4) takes the simple form
x = u, u(t) eV = JRn,	(1.20)
and if 12 = {?;} consists of just one point, then we do not have any con-
straint on the derivative x. Our problem of optimal control thus reduces to
the standard problem in the Calculus of Variations:
min f L(t,x,x)dt,	ж(0) = x, x(T) = у.	(1-21)
*(•) Jo
Roughly speaking, the main difference between the problem (1.18)-(1.19) and
(1.21) is that in (1.21) the derivative x is unrestricted, while in (1.18)-( 1.19)
it is constrained within the closed set F(x) introduced at (1.6).
The basic mathematical theory of optimal control has been concerned with
three main issues:
(i)	Existence of optimal controls. Under a suitable convexity assumption, op-
timal solutions can be constructed following the direct method in the
Calculus of Variations, i.e., as limits of minimizing sequences, relying on
compactness and lower semi-continuity properties. When the convexity
condition is not satisfied, the problem usually does not admit any optimal
solution. In some special cases, however, the existence of optimal control
can still be proved, using a variety of more specialized techniques.
(ii)	Necessary conditions for the optimality of a control. The ultimate goal
of any set of necessary conditions is to isolate a hopefully unique candi-
date for the minimum. The major result in this direction is the celebrated
Pontryagin Maximum Principle , which extends to control systems the
Euler-Lagrange and the Weierstrass necessary conditions for a strong lo-
cal minimum in the Calculus of Variations. These first order conditions
have been supplemented by several high order conditions, which provide
additional information in a number of special cases.
(iii)	Sufficient conditions for optimality. For some special classes of optimal
control problems, one finds a unique control u*(*) which satisfies the Pon-
tryagin’s necessary conditions. In this case, u* provides the unique solution
to the optimization problem.
For general nonlinear systems, however, conditions which guarantee the
optimality of a control u*(-) can only be obtained by a global analysis.
1 Introduction
11
Toward this goal, a standard technique is to embed (1.18)-(1.19) in a
family of problems, obtained by varying the initial conditions. The value
function V, defined as

= min
uEU
subject to
x = f(t, x, u), x(r) = y, x(T) e J?,
can then be characterized as the solution to a first order Hamilton-
Jacobi partial differential equation and computed by dynamic program-
ming methods. In turn, from the knowledge of the function V and its
gradient VXV, one can determine the optimal control и in feedback form.
The strong nonlinearity of the Hamilton-Jacobi equation and the possible
lack of regularity of the value function V account for the main difficulties
toward a rigorous mathematical analysis. In this direction, a major step
forward has been provided by the theory of viscosity solutions.
In addition to the fundamental theory, valid for control systems of the
general form (1.3), a wealth of results are available for some special systems
which can be analyzed in much greater detail. In particular, consider the linear
system with constant coefficients
x = Ax 4- Bu,
(1-22)
where x 6 IRn, и 6 IRW and the matrices А, В have dimension n x n and
n x m, respectively. For a given control t	the corresponding solution
of (1.22) admits the explicit integral representation
rr(t, u) = efAa?(0) + I s^ABu(^s)ds.
Jo
This allows an in-depth study of all the relevant properties of the system.
Another important class consists of semi-linear systems, having the form
m
i = fo(x) + ^fi(x)ui,
1=1
(1.23)
where fa. fa, • • • , fm are smooth vector fields on IRn. In general, there exists
no explicit representation for the trajectories of (1.23) in terms of integrals of
the control. Nevertheless, a rich mathematical theory has been developed for
these systems, applying techniques and ideas from differential geometry and
the theory of Lie algebras.

2
Review of Differential Equations
In the basic model of a control system, as soon as a control function и — u(f)
is assigned, the evolution can be determined by solving the O.D.E.
z = g(t, x) = f(x, u(0)	(2.1)
In this chapter we review various aspects of the theory of differential equations,
with particular focus on issues that arise from applications to control systems.
Let J? be an open set in IR x IRn. Given a function g : J? IRn, by a
(Caratheodory) solution of the O.D.E.
x = g(t,x)	(2.2)
we mean an absolutely continuous function 11—>	defined on some interval
which satisfies (2.2) almost everywhere. Equivalently, we require that
x(t) = x(to) -|- / g(s,x(s))ds
J to
for every t € [£оД1] where the function is defined.
In the classical theory of ordinary differential equations, it is assumed
that the function g is continuous w.r.t. both variables. For applications to the
theory of control, however, it is important to consider also the case where g
is only measurable w.r.t. the variable t. This more general setting is needed
for two main reasons:
In many practical situations, a controller acts on the system using a fi-
nite number of switches that can be turned on and off. In a mathe-
matical model, the control function thus takes the form t »—► u(t) =
with Uj(t) = 1 or Uj(t) = 0 if at time t the J-th switch
is turned on or off, respectively. The function u(-) thus takes values in a
discrete set. All non-constant controls are necessarily discontinuous.
In several optimization problems, the existence of an optimal control can be
established only within the class of all measurable functions u(-). Quite
often, the optimal control is actually discontinuous.
14	2 Review of Differential Equations
When a measurable control function и = u(t) is inserted in the equation
describing a control system, we obtain an O.D.E. of the form (2.1), whose
right hand side is only measurable w.r.t. the time t.
2.1 Fundamental theory
We begin this chapter by proving a basic result on the local existence and
uniqueness of solutions to the initial value problem for the O.D.E. (2.2).
Basic assumptions. Throughout this section we assume that the function
g : J? i—* IRn satisfies the following conditions:
(A)	For every x the function t—>g(t, x) defined on the section
41x = {t : (t, rr) € 12}
is measurable. For every t, the function x^>g(t. x) defined on the section
= {x : (t, x) G f?}
is continuous.
(B)	For every compact К C (1 there exist constants C#, Lk such that
|3(t,x)| <	, \g(t,x)-g(t,y)\< LK\x-y\ for all (t, x), (t y) £ К.
(2-3)
Fig. 2.1. The horizontal and vertical sections J2«, and the approximation of w
by a piecewise constant function.
Theorem 2.1.1. (Existence of solutions). Given a map g : 41 IRn,
consider the Cauchy problem
X = g(t,x),
x(to) = x0,
(2-4)
2.1 Fundamental theory
15
for some (to,^o) € ft-
(i) If g satisfies the assumptions (A), (B), then there exists e > 0 such
that (2.4) has a local solution x(-) defined for t€ [to, to + e].
(ii) Assume, in addition, that the function g is defined on the entire space
IR x IRn and there exist constants C, L such that
\g(t,x)\<C,	\g(t,x) — g(t,y)\ < L\x — y\ for all t,x,y. (2.5)
Then, for every T > to, the initial value problem (2.4) has a global unique
solution x(-) defined for all t G [to, T]. Moreover, the solution depends contin-
uously on the initial data Xq .
Proof. The proof will be achieved in several steps.
1.	We first prove (ii). Assuming that (2.5) holds, we shall construct a forward
solution of (2.4), on any given interval [to, Т]. In view of applying Theorem
A.2.1 of the Appendix, define A = lRn. The initial condition xq € IRn plays
here the role of a parameter. Moreover, let X be the space of all continuous
functions from [to,T] into IRn with the “weighted” norm
lk(-)llt = toI5^T e
which is equivalent to the usual C° norm
lk(-)llco = Wf)l-
Finally, define the map Ф : А x X 1—► X by setting
Ф(т0, w(-))(t) = xq + [ g(s,w(s))ds, t G [t0,T].
J t0
(2.6)
(2.7)
2.	To prove that Ф is well defined, for each function w(-) G X we need to
show that the composite map s 1—► g(s, w(sf) is integrable. This is not entirely
obvious, because g itself is not continuous. We argue as follows: Given the
function w(-), consider the sequence of piecewise constant functions
as in Fig. 2.1, with
= w
Л -u
*0 n----I
\ y /
if
to + -
I/
to 4-----
у
t g
By (A), the maps t—>g(t, w^tf) are all measurable. Moreover, the second
assumption in (2.5) implies
lim |<?(£,w(£)) — g(t, wl/(t))| < lim L|w(t) — wp(f)| = 0.
V—*OO	P—>OO
Hence the function t—>g(t, w(tf) is measurable, being the limit of a sequence
of measurable maps. Since |^(s,w(s))| < C for all s, the integral in (2.7) is
16	2 Review of Differential Equations
well defined and depends continuously on t. Hence, Ф is well defined and takes
values inside X.
3.	The continuous dependence of Ф on Xq is obvious. To study its depen-
dence on w(-), consider any two functions w, w' e X and set ||w — w'||j = 6.
Recalling the definition (2.6), we have
|w(s) — wz(s)| < Se2Ls for all s 6 [a, b\.
Moreover, the assumption of Lipschitz continuity in (2.5) implies
e 2Lt |Ф(яо, w)(t) — Ф(^о, w')(t)| = e 2Lt I g(s,w(sy) — g(s,w'(s))ds
JtQ
	<e 2Lt 1 L\w(s) — w'(s)| ds */ to < e~2Lt I L8e2Ls ds J t() 6 < 2
for all t G [*0, Т]. Therefore,
||^(xo,w)-^(x0,w')llt | Ik-w'llf	(2-8)
4.	We can now apply fixed point Theorem A.2.1, obtaining the existence of a
unique continuous mapping xq—»#(•) such that x = Ф(хв,х), i.e.
x(t) = Xq + / g(s,x(xy)ds for all fG[t0,T].
J to
By definition, #(•) is the required solution to (2.4). This achieves a proof of
(ii).
5.	Finally, we prove the statement (i) concerning local existence, without the
additional assumption (2.5). Choose e > 0 small enough so that the cylinder
K = {(t,x): \t-to\<£, |x - x0| < e}
is entirely contained inside the domain Г2. Then consider a smooth cut-off
function ф : IR x IRn i—> [0,1] such that ф = 1 on A\ while 0 = 0 outside some
larger compact set K', with К С К' С 12. Observe that the function

if
if
(t,x) G ii,
(t,x) a,
(2-9)
2.1 Fundamental theory
17
satisfies (A) and (B), together with the extra assumption (2.5), because it
vanishes outside the compact set K'. By the previous steps, there exists a
solution x(-) to the Cauchy problem
i(t) = 9^(f.x(f)\ x(to) = x0,
(2.Ю)
defined on arbitrarily large interval [io, Т].
We now recall that the cylinder К is a neighborhood of the point (io,xo).
Therefore, for some e > 0 sufficiently small, the point (t,x(t)) remains inside
К as i G [io, to + e]. Since g and coincide on K, the function jr(-) thus
provides a local solution to the original problem (2.4), restricted to the smaller
time interval [io , io + e].
Remark 2.1 The construction of a solution backward in time, on an interval
[to — £, to] is entirely analogous. It can be reduced to the previous case by
reversing time (i.e. setting т = —t) and considering the equation
dx(r)
dr
-s(-T,x(r)).
The next lemma provides a useful tool for estimating the distance between
two solutions of a differential equation. It represents the main ingredient in
several uniqueness proofs, and in this respect it can replace the original version
of Gronwall’s Lemma , which has more mathematical content and also a longer
proof.
Lemma 2.1.2. (Gronwall). Let z(-) be an absolutely continuous nonnegative
function such that
z(io) < 7, z(t) < a(t)z(t) + /3(i) for a.e. t € [io, T],	(2.11)
for some integrable functions and some constant 7 > 0. Then for every
t G [t0, Г] the following holds
>t
z(t) < 7 exp
to
a(s)ds\+ / /3(s)expl / a(cP) da
/	J t0	\JtQ
ds .	(2.12)
Proof Notice that the right hand side of (2.12) is precisely the solution to
the linear Cauchy problem
w(i0) = 7, w(t) = a(i)w(i) 4- Z?(t).
In order to establish the inequality, consider the absolutely continuous func-
tion
V>(i) = exp
/ a(s)ds) z(t) — / /?(s)exp
Jt0 / L JtQ
ds
18
2 Review of Differential Equations
Using (2.11), a direct computation shows 'ijfit) < 0 for almost every t. There-
fore
V>(£) < V>(*o) =	< 7 for all t e [to? Т].	(2-13)
Multiplying (2.13) by exp( a(s)d.s), from the definition of -0 we obtain
(2-11).
We can now establish the uniqueness of the solution to the Cauchy problem
(2.4), whose local existence was proved in Theorem 2.1.1.
Theorem 2.1.3. (Uniqueness). Let g : J? i—> IRn satisfy the assumptions
(A) and (B), as in Theorem 2.1.1. Let Xi(-),X2^) be solutions of (2.4), defined
on the intervals [to, tj, [to, t?] respectively. If T = min{ti,t2}? then x\(t) =
^(t) for all t € [to, Т].
Proof. Let Lk be the Lipschitz constant in the assumption (B), corresponding
to the compact set
К = |((,ari(t)),	: t ё [io,r]|.
Then the absolutely continuous function z(t) = |^i(t) — #2 (01 satisfies
z(t0) = 0, i(t) < |ii(t) — 0:2 (01 < Lxzff) for a.e. t.
Applying Gronwall’s lemma with a = Lk, 0 = 7 = 0, this implies z(t) < 0
for all t e [to, Т].
In general, the solution to the Cauchy problem (2.4) is defined only locally,
for t in a neighborhood of the initial time tg. If a solution cannot be extended
beyond a certain time T, two cases may arise (see Fig. 2.2):
1. As t —> T—, the point (t, x(t)) approaches the boundary df2 of the domain
12 where g is defined.
2. As t—, the solution blows up, i.e. \x(t)| —>oo.
The next theorem shows that these are actually the only two possibilities.
Theorem 2.1.4. (Maximal solutions). Let the basic assumptions (A), (B)
hold. Let T > to be the supremum, of all times т such that (2.4) has a solution
#(•) defined on [to,r]. Then, either T = oo, or else
( hm_ ^(t)| +
(2.14)
2.1 Fundamental theory
19
Fig. 2.2. Two maximally extended solutions.
Proof. Assume T < oo. If (2.14) does not hold, then there exist Л/, s > 0 and
a sequence ty^T— such that, for all v > 1,
|ж(^)| < Af,	2j(ip)), сИ2) >e.
By possibly taking a subsequence, we can assume that x(ty) converges to a
point Xoo, with (T, ^oo) e J?. Choose p > 0 so small that the cylinder
К = {(t,x) : \t - T\ < p, |x - Zoo| < p}
is entirely contained in J?. As in (2.9), construct a function : IRxIRn »—> IRn
such that = g on A, and in addition g^ satisfies the global bounds
|^(*,ж)|<С,	|<z+(t,rr)-5f(t,y)| < L|x-y| for all t,x,y, (2.15)
for some constants C, L > 1. Fix 5 > 0 small enough so that
(2C+l)5<p,
and choose и so large that
XqQ I <	T try <
By (ii) in Theorem 2.1.1, the Cauchy problem
У(«) = 9\t,	y(tv) = x(t„)
has a solution ?/(•) on [ty, T + <5]. We can now define an extension x of x by
setting
f	tQ<t<ty
X^ = \y{t} if tv<t<T + 8.
20	2 Review of Differential Equations
Since |?/(t) | < C, for t e [t^, T 4- J] we have
\y(t) ~^oo| < \y(t) -	+ |^(L/) -Zed < C(t-tp)4-5 < 2Cb + 8<p.
Therefore, for all t 6 [tp, T4-5], the point (t, y(tfi) remains inside the compact
К where g and g^ coincide. The function £(•) thus provides a solution of the
original problem (2.4), defined on the strictly larger interval [to, T 4-5]. This
contradicts the maximality of T, thus proving the theorem.
In the case where g is defined on the entire domain [to, oo] x Rn, to
establish the the global existence of the solution to (2.4) it thus suffices to
prove that x(-) remains bounded on bounded intervals of time. In general,
a-priori estimates on the size of |#(t)| can be obtained by a comparison with
a scalar O.D.E., as we now describe.
Theorem 2.1.5. (A-priori bounds). Let g : 12 i—> Rn satisfy the basic
assumptions (A)-(B). Let 0 : [to, ti] xR i-> R be a scalar function, measurable
in t and continuous in x, such that
4>(t,r) > max \g(t,x)\	for all t,r.	(2.16)
|x|=r
Let r : [£o3i] ► R be an absolutely continuous function such that
r(t) >zp(t,r) for a.e. t e [Wi],	r(t0) > Ы •	(2.17)
If the set К = {(t,x) : to < t < t±, |x| < r(t)} is contained in (2, then the
Cauchy problem (2.4) has a solution defined on the entire interval [to, ti],
which satisfies
|#(t)| < r(t), for all t e [to, *1].	(2.18)
Proof Define the auxiliary function
^*(t,rr) =
\g(t,j:)
\-9(г’ г^й)
if
if
kl < r(t),
И > r(t).
(2-19)
Otherwise stated, g*(t, x) = g(t, 7гг(гг)), where 7r*(a?) denotes the perpendicu-
lar projection of x € Rn on the ball centered at the origin with radius r(t).
From the basic assumptions (A)-(B) it follows that #* is globally bounded
and Lipschitz continuous w.r.t. x on the domain [to, tj x Rn. Therefore, the
Cauchy problem
y(t) =	2/(0) = x0,	(2.20)
has a unique global solution x* : [to, ti]	IRn .
We now observe that the two maps t h-> |.r*(t)| and t •—* r(t) are both
Lipschitz continuous and satisfy \x*(to)| < r(to), together with
4k*(0l < |ff*(tz*(0)| < max |5(t,x)| < ^(t, r(0) < r(t)
at	i	I |i|=r
2.2 Linear systems
21
at almost every time t € [^о,й]. Hence
|a?*(£)| < r(t) for all t e
(2.21)
By definition g*(£,x) = g(tyx) whenever |rr| < r(t). Therefore, the function
#*(•) coincides with the unique solution of the Cauchy problem (2.4). By (2.21)
this proves the theorem.
In many applications, useful estimates can be obtained from the above theo-
rem by a judicious choice of the function r(-).
Corollary 2.1.6. Assume that the function g = g(t,x) satisfies the bound
|^,х)| < C(1 4-|®|)	(2.22)
for some constant C and all x € lRn. Then any solution of (2.4) satisfies the
a priori estimate
|x(t)|	(1 + |x(i0)|) •	(2.23)
Indeed, for t > the estimate (2.23) is obtained by taking ^(t, r) =
C(1 + r). Solving the scalar Cauchy problem
r = C(1 T r),	r(t0) = r0 = hr(t0) I
we find
r(t) = eC(t"fo)r0+ f Cec(-s-to} ds = eC('-to)|z(f0)| + (eC(s-to) - 1) .
This yields (2.23) in the case t > to- For t < to the estimate is obtained in an
entirely similar way, reversing the direction of time.
2.2 Linear systems
In this section we consider differential equations of the form
x = A(fi) x	(2.24)
p=-pA(t)	(2.25)
where t—>A(t) is a measurable map from an interval [a, b] into the set of n x n
matrices. We regard x as a column vector and p as a row vector. The equations
(2.24)-(2.25) are thus a short-hand notation for
22
2 Review of Differential Equations

( an • • • ain
\ ^nl ’ ’ ’ ®nn /
Throughout this book, the norm of a matrix A is defined as
||A|| = max |Arc| = max|pA|.
|x| = l	|p| = l
In the special case where the n x n matrix A(t) = A is independent of
time, the solution to the Cauchy problem
x = Ax,	#(0) = x	(2.26)
can be written in the form
x(t) = etAx.	(2.27)
Here the exponential matrix elA is defined as the limit of the absolutely con-
vergent series
fk дк
‘‘л - E -jr 	<2-28>
k=0
We recall that, by definition, A0 = I is the n x n identity matrix.
Observe that, if В = R~rAR for some invertible matrix 7?, then
= Re^R-1.
Indeed, for every k > 1 one has
(RBR-1^ = RBR-1 • RBR-1 • • • RBR~l = RB^"1.
The actual computation of the exponential matrix etA can thus be carried out
by reducing A to a more convenient canonical form B. and then computing
etD.
Example 2.1. Assume that A is a 6 x 6 matrix, with
det(a-^) = (C-A)«-M)3(C-(a + i/3))(C-(a-f/3)),
so that Л is a simple real eigenvalue, /i is a multiple eigenvalue and a±i(3 are a
pair of complex conjugate eigenvalues. Assume that the geometric multiplicity
of /1 is 1. Then there exists an invertible matrix R that reduces A to the
canonical form
B = R~'AR =
/Л 0 0 0 0 0 \
0/1100 0
0 0 /z 1 0 0
0 0 0 // 0 0
0 0 0 0 a -(3
\0 0 0 0 (3 a /
2.2 Linear systems
23
In this case one has
	(ext	0	0	0	0	°	\
	0		te*“ («2/2)6*“		0	0
tB _	0	0	eMt		0	0
e =	0	0	0		0	0
	0	0	0	0	eat cos/3t —eat sin (3t
		0	0	0	eat sin eat cos j3t )
and etA = Re^R-1.
The next theorem provides the global existence and uniqueness of solut ions
to linear systems, with general time-dependent coefficients.
Theorem 2.2	.1. (Existence of solutions for linear systems). Assume
that ||A(t)|| < L for some constant L and all t E [a, b]. Then, for any to e [a, b]
and every initial condition Xo E !Rn, the Cauchy problem
x = A(t)x, x(to) = Xq,
(2.29)
has a unique solution defined on the entire interval [a, 6]. This solution satisfies
|x(t)|<eL|‘ tol|ar0|.
(2.30)
Proof. Indeed, the local existence and uniqueness are obtained applying The-
orem 2.1.1, (i) with g(t,x) — A(t)x. For t > to, the estimate (2.30) follows
from Theorem 2.1.5, with ^(t, r) = Lr, r(f) = eL^“tol|xo|. Since ||Л|| = ||— A||,
reversing time we obtain (2.30) also in the case t < to-
Of course, an entirely similar result holds for (2.25). The systems (2.24)
and (2.25) are related by a fundamental property:
Theorem 2.2	.2. (Adjoint systems). Let #(•), p(-) be any two solutions of
(2.24), (2.25) respectively, defined on the same interval of time: t E [a,b].
Then their inner product p(t) • x(t) is constant.
Proof. This is verified by the direct computation
^-(p- x) = p- x+p>x = —pAlf) - x + p - A(t}x = 0.
dt
Since the system (2.24) is linear and homogeneous, the set of solutions is
a linear space. In other words, if #(•) and ?/(•) are solutions of (2.24), then
the linear combination Xx + py provides yet another solution, for every choice
of X,p E 1R. To obtain the general solution to a Cauchy problem, it thus
suffices to construct a set of n linearly independent solutions. This motivates
the following construction. Let
24
2 Review of Differential Equations
be the elements of the standard basis in IRn. For a fixed time s and each
j = 1,..., n, call 11—► Vj(t, s) the solution to the Cauchy problem

= e,.
Construct the n x n matrix Af(£,s) whose columns are given by the vectors
vi, V2, •.., vn • Namely
M(t,s) = vi(t,s) v2(t,s)
vn(t,s)
This is called the fundamental matrix solution of (2.24). For a fixed value of
s, the map t	M(t, s) provides a matrix-valued solution to the problem
= A(t)M{t, s),	M (s, s) = I,
(2-31)
where I denotes the n x n identity matrix.
Theorem 2.2	.3. (Properties of the fundamental matrix solution). As-
sume that the matrices A(t) are uniformly bounded, with coefficients depending
measurably ont € [a, b\. Then for every € lRn, the function x(T) — M(T, .s)£
provides the solution to the Cauchy problem.
= A(t)x(t),	x(s)=£.	(2.32)
The fundamental matrix solution M satisfies
M(t, s)M(s,r) = M(t,r) for all	(2.33)
=	(2-34)
Moreover, if h : [a, b]lR.n is integrable, then any function satisfying
x(t) = M(t,r)x(r) + У M(t, s)h(s) ds	(2.35)
is a solution to
x = A(t)x(t) + hft).	(2.36)
2.2 Linear systems
25
Proof. The first statement follows immediately from (2.31). To prove (2.33)
we observe that, for every £ € Rn, r e [a, 6], the functions
^1(0 = Af(t, s)Af(s, r)£,	^2(t) =
are both solutions of the Cauchy problem
J^rr(£) = A(t)x(t), x(s) = M(s, r)£ .
By uniqueness, Xi(t) = xz(t) for every t. Since £ € IRn was arbitrary, this
proves (2.33).
From (2.33) it now follows M(t, s)M(s, t) = I, hence the map s h->
= [Af(s,£)]-1 is differentiable. An elementary computation yields

— [M(M)M(M)] = 0 =
M(s,t) + M(t,s)[A(s)M(M)].
(2.37)
Multiplying (2.37) on the right by [M(s,t)]-1 we obtain (2.34).
The last assertion is verified by straightforward differentiation, writing
(2.35) in the equivalent form
xft) = Л/(£,т)
я:(т) -F [ M(r,s)h(s)ds
By Theorem 2.2.1, each bounded, measurable matrix-valued function A(-)
determines a fundamental matrix solution M(•, •). The next result states that
M depends continuously on A, in the appropriate norms.
Theorem 2.2	.4. (Continuous dependence of the fundamental matrix
solution). The map A(-) i—* Af (•, •) is continuous w.r.t. the distances
M(-)-OllL1= [b\\A(t)-A'(t)\\dt,
J a
\\M(•, •) - M'(-, -)llco = max \\M(t,s) - M\t,s)||.
a<s,t<6
Proof. Fix any s G [«,5] and let v be any unit vector. For t > s, define
v(t) = M(t, s)v, v'(t) = M'(t, s)v, z(t) = v(f) - ?/(£),
and observe that z(s) = 0. Then
z = v — i)' — A(t)v - A'(fyu',
and
^|z(f)| < H(t)« -	+ \A(t)v' - А'(*У|
<||A(t)||-|z| + ||A(t)-A'(t)||-K|.
26	2 Review of Differential Equations
Since we are assuming the bound ||A'(t)|| < L, there holds
|«'WI < exP IH'MII da^ < ець~а\
We now apply Gronwall’s Lemma with
a(t)=||A(t)||5	Ж=еЬ(6-")||А(0-Л/(01|.	7 = 0,
obtaining
|z(i)| < eL^ £ ||A(<t) - A'(a)|l exp ||A(C)II da.
The above estimate shows that
max max \M(t, s)v — M'(t, s)v|	(2.38)
a<s<t<b |v|=l
<e^exp( / ||A(cr)|| da j • / ||A(cr) - >l'(cr)|| da.
a	! J a
Clearly, the right hand side of (2.38) approaches zero as A—+A' in the L1
norm.
The estimates for t < s are obtained in the same way, considering the
systems
dv л/ x	dv' 4/z	. ,	. r ,	.
— = — A(—r)f, -т— — — A (—г) r,	t = —t e [—6, —a].
2.3 Differentiability with respect to initial data
A common problem in the theory of optimal control is to test whether a
given control function ?/*(•) is optimal. This is usually done by comparing the
trajectory t i—> #(f,u*) with other nearby trajectories. The basic ingredient
in this analysis is a detailed description of how the solution of the Cauchy
problem
i(t) = g(t, #(£)), x(t) = £	(2.39)
changes, as the initial data t, £ are varied. Throughout the following, we denote
by £ н-> rr(f,T, £) the solution of (2.39), while Dxg(t,x) is the n x n Jacobian
matrix of first order partial derivatives dgi/dxj at the point (t,x).
Theorem 2.3.1. (Directional derivatives). Let g : J? h-> ]Rn satisfy the
basic assumptions (A)-(B) and be continuously differentiable w.r.t. x. Let
£(•) = x(-, to^o) be the solution of (2.4), defined for t G [fo, h]- For Vq G IRn,
call ?;(•) the solution of the linear Cauchy problem
2.3 Differentiability with respect to initial data
27
Then
i’(t) = Dxg(t,x(t))  v(t), v(t0) = v0-
lim
£->0+
x(t,t0,x0 + ev0)-x(t)
--------------------------V(t) = I),
e
(2.40)
(2.41)
the limit being uniform for t € [£o,*i], |^o| < 1-
Fig. 2.3. The first order variation of a solution, as the initial point жо is changed
to Xo + EVo.
Proof. 1. As in Fig. 2.3, for e sufficiently small define
x£(t) = x(t, to, x0 + ev0),	y£(t) = x(f) + £v(t).	(2.42)
To prove the theorem, we need to show that
lta _0 (2 43)
£-*0+	£
Observe that x£(-) is the fixed point of the map w Ф(ж0 +	w), with
ф(х0 + evq, w(-)\(t) = xo -I- evq + / p(s,w(s))ds,
J to
We recall that Ф is contractive with respect to the equivalent norm ||-||| defined
at (2.6). Thinking of y£(/) as an approximate fixed point, by (A.6) in Theorem
A.2.1 of Section A.2, one has
1	2
- ||xe - ?/e||t < - ||Ф(х0 + ev0, ye) - УеIIt •
It therefore suffices to show that, uniformly for |uq| < 1,
28
2 Review of Differential Equations
lim sup -
z0 + 6v0 + / g(s,y£(s,voy)ds - s/e(t,v0)
JtQ
= 0.
(2.44)
2.	By the equations (2.4) and (2.40), satisfied by £(•) and by v(-) respectively,
we have
жо + / .g(s,£(s)) ds — x(t) = 0,
Ao
ev0 + / Dx^(s,^(s))
J to
• ev(s) ds — Ev(t) — 0 .
The quantity in (2.44) can thus be estimated as
1
— T T
E
1
E
/ g(s, x(s) -I- Ev(sf) ds — x(t) — Ev(t)
J to
#o + evo+ / g(s,x(sf)ds
J to
•'to	)
+ 11 lPxg(s, £(s) + (tev(s)) — Dxp(s, f(s))] • ev(s) da ds
J to A
< I I	i(s) + crev(s)) - Dxg(s,x(s))\\ • |v(s)|d<7ds. (2.45)
J to Jo
Let К C 12 be a compact set containing a neighborhood of the graph of x and
Lk as in assumption (B). Then |v(s)| is bounded by eLl<s|vq|, hence it is clear
that the right hand side of (2.45) approaches zero, uniformly for t G [to, Zj,
|Vo| < 1. In turn, this proves (2.43), hence (2.41).
For each t € [Zo, й], Theorem 2.3.1 states the existence of all directional
derivatives for the map £ •—► rr(Z,Zg,£). In the next theorem, we observe that
these derivatives depend continuously on the point Xo where they are com-
puted, and conclude that the map is differentiable.
Theorem 2.3	.2. (Differentiability w.r.t. the initial point). Let g : Г2 h-*
IR” satisfy the basic assumptions (A)-(B) and be continuously differentiable
w.r.t. x. Let f(-) be a solution to (2.4), defined on [Zg,ti]. Then, for any t €
[Zg,Zi], the map £ i—► x(t,to,£) is continuously differentiable in a neighborhood
of .Tg. Its Jacobian matrix at a given point Xo is
D^t,t0^)^xo = Af(Mo),	(2.46)
where A/(-,-) denotes the fundamental matrix solution to the linear problem
v(t) = Dxg(t,x(t,t0,xQ))
(2-47)
2.3 Differentiability with respect to initial data
29
Proof. The n partial derivatives of the map £ i—► a;(t,to,C) at £ = are
defined as
x(t, fо, .lq “I- c e^) x(t, to, xq)	.
hm —------------------------------	i = 1, • • • , n,
£—0	e
where {ei, • • • , en} denotes the standard basis in IRn. By Theorem 2.3.1 these
limits exist, being equal to Vi(t) = M(L ^o)e?? where M(•, •) is the fundamental
matrix solution to (2.47). To complete the proof, it thus suffices to show that
these partial derivatives depend continuously on the point xq.
Let be a sequence of initial points, with lim^-^o = Xo- Then the cor-
responding trajectories x(-, Zq, Cp) converge to □?(•, to, xo) uniformly on [to, ^1],
and the matrix-values functions t i—► A„(t) = Dxg(t,x(t,to,£tU')>) converge in
L1 to the map t i—► A(t) = Dxg(t, x(t, to, xo)). By Theorem 2.2.4, the corre-
sponding fundamental matrix solutions A/p(-, •) converge uniformly to Af(-, •).
In particular,
lim M„(t, to)ei = M(t, to)ei,
y—*oo
proving the continuity of the partial derivatives.
The last result in this section is concerned with the differentiability of the
solution x(-,r, £) w.r.t. the initial time t, under the additional assumption
that g is continuous also w.r.t. time.
Theorem 2.3	.3. (Differentiability w.r.t. the initial time). In addition
to the basic assumptions (A)-(B), let the function g be continuous in both
variables t,x and continuously differentiable w.r.t. x. Let a:(-,Zo,^o) be a so-
lution of (2.4), defined on some interval Then, for every t e [fo,^i]
the map r 1—► x(t,r, xq) is continuously differentiable in a neighborhood of to-
More precisely, one has
DTx(t, r, ^o)(T=to = “Af (t, to)g[to, xo),	(2.48)
where •) is the fundamental matrix solution for (2.47).
Proof. 1. Call £(t) = x(to,r,xo) the value at time t = to of the solution to
x = g(t,x), ж(т) = ж0-
Since x(t,r, Xq) = x(t,to,^r)), from Theorem 2.3.2 it follows
DTx(t,r,x0) = D^x(t,t0,x0)-^- = M(t,t0)^L	(2.49)
2. Using the assumption that g is continuous in both variables, we now com-
pute
C(t-) - ж0
т - to
1 rto
—- -----— / g(s,x(s.T,xoy)ds = -p(t0,x0).
T — to JT
Inserting this value in (2.49) one obtains (2.48).
л— - liD?
dr r^tQ
30	2 Review of Differential Equations
2.4 A transversality theorem
Let g : [to, x lRn t—> IRn be a continuously differentiable vector field, and let
Л4 C IR',+1 be a n-dimensional manifold. More precisely, assume that there
exists a Cl mapping ф : IRn+1 i—> 1R such that M can be represented as the
zero level set of ф :
M = {(t,x); ф(},х) = 0},	(2.50)
and such that the gradient of ф does not vanish on any point of M :
Чхф)(1,х) = (<^,0X1, • • • ,фХп)(1,х) / (0,0, • • • ,0) for all (f,x) € M.
(2.51)
Let ./,*(•) be a solution to the differential equation
±(t) = g(t,x(tf).	(2.52)
If (т, x(t)) e M. we say that #(•) intersects M transversally at the point
(r, x(r)) if
x(r)) + Vx(/>(r, x(r)) • ,g(r, t(t)) / 0 .	(2.53)
This means that the vector (1, д(т, .t(t)) is not tangent to M at the point
(t, a:(r)). The next theorem states that “almost all” trajectories of (2.52) have
only transversal intersections with Л4.
Theorem 2.4.1. (Generic transversality of trajectories). Let g = g(t,x)
be continuously differentiable w.r.t. both t and x. Assume that, for every Xo
in an open ball В C IR71, the solution t i—* x(t,to,xo) of (2.52) with initial
condition x(fo) = xo is defined on the whole interval [to^i]- Let Л4 C JRn-H1
be an n-dimensional embedded manifold, as in (2.50)-(2.51). Call N the set
of all points Xo E В such that (t,t(t, io^o)) £ АЛ for some r 6 [to? but
the intersection is not transversal. Then meas(ff) = 0.
Proof. 1. The manifold Л4 admits a countable open covering {Л4}} such that,
for every i, there exists a chart : Ai Mi, with Ai C IRn a bounded open
set and at1 diffeomorphism. Since the countable union of negligible sets
has measure zero, it suffices to prove the theorem assuming that M = <p(A),
where A is a bounded open set in lRn and <p : A i—> IRn+1 is a Cl embedding,
i.e.,: Rank(D^(T/)) = n for all у e A.
2.	For each у € A, if 92(1/) = (t, x(t, to, #o)) for some r, xq, set Ф(у) = j;q.
Otherwise stated, if </?(?/) = (t(?/), £(?;)), then
tf'O/) = x(to, r(y), C(,v))	(2.54)
whenever the right hand side of (2.54) is defined and lies in B. By Theo-
rems 2.3.2 and 2.3.3, x(fo,T, £) is continuously differentiable w.r.t. t, £. There-
fore, Ф is a C1 function defined on some open subset A' C A, taking values
inside B.
2.4 A transversality theorem
31
Fig. 2.4. The only two non-transversal trajectories are the ones originating from
x i and X2 •
3.	Let (t/i,--- ,yn) be coordinates on A. If = (r,£), from (2.46), (2.48)
it follows that the partial derivatives of Ф w.r.t. t/i, • • • ,yn are the vectors
dyi	\dyt	dyij
where Af(-, •) denotes the fundamental matrix solution for the linear system
v(t) = Dxg(t,i(t,T,O)  v(t).
The Jacobian determinant det(D^(?/)) vanishes if and only if the n vectors
(2.55)
are linearly dependent. Observing that the first n vectors in (2.55) form a basis
for the tangent space to M at the point y)(y), we conclude that det(£>^(?/)) =
0 if and only if the vector (l,^(r, £)), with £ = аг(т, to,^o), is tangent to M
at (r, £), i.e. if and only if the crossing at (r, £) = <p(y) is not transversal.
4.	By the previous analysis,
Af = P| Л4 where Л4 = |^(т/); у G Af, det(D#(y)) < e} .
E>0
Computing the area of J\T£ using the ^/-coordinates, one obtains
meas(A4) < /	|det(£>’Zz(?/))| dy < emeas(A).
J{y€A', |det(D^(?/))|<£}
Letting £	0 we obtain meas(AQ = 0, proving the theorem.
32
2 Review of Differential Equations
Example 2.2. Every solution of the differential equation x = l^l1/2 with
ж(0) < 0 intersects the manifold M. = {(tx); x = 0} C IR2 tangentially, at
some time t > 0. Of course, this does not contradict the previous theorem
since the function g(t,x) = |x|1?Z2 is not differentiable on M .
Problems
2.1.	Consider the Cauchy problem
z(0) = Xq ,
with a• > 1. Show that, for every xq > 0, the solution becomes unbounded
in finite time. Find the time T, depending on q and Xq, such that
lim x(t) = oo .
2.2.	Consider the Cauchy problem:
x = 2\/|ж[,
z(0) = 0.
Show that for every a < 0 < b there exists a unique solution xab, defined
for every t e IR, such that xab(t) < 0 for t < a. xab(t) > 0 for t > b and
= 0 for b < t < a.
2.3.	Let : J? нч. IR7' satisfy the assumption (A). Prove that (B) is satisfied if
the following holds. The function g is continuously differentiable w.r.t. x
and, for every compact К C 12 there exist constants Cj<, Lk such that
\g(t, x)| < CK, ||£>^(t,x)|| < LK for all (t,x) e K.
2.4.	Show that in Theorem 2.1.1 the global existence of solutions still holds if
the assumption (2.5) is replaced by the following weaker condition: There
exists an integrable function ф : [io,T] i—► R± such that

for all (t,y) e IR x IRn.
Hint: use the equivalent norm
l|w(-)||t = max e v’(s)ds  |w(0|.
£€[a,b]
2.4 A transversality theorem
33
2.5.	Show that the conclusion of Theorem 2.1.5 remains valid if the definition
(2.16) of is replaced by
/ x	\
V>(£, r) = max ( — , g(f,x) > for all t,r .
|x|=r \|x|	/
Hint: the derivative of the map 11—► |x(t)| is computed by the inner product
2.6.	Given the differential equation
x = x ,
let x(-) be the solution with initial data x(0) = 1. Moreover, for £ > 0,
call xe(-) the solution with perturbed initial data xe(0) = 1 + e. For t < 1,
compute the vector
„(,) _ Um .
4 7	€->0	8
Check that it satisfies the corresponding linearized equation (2.40) with
v(0) = 1.
2.7.	Consider the linear system of differential equations
Xi = x2, x2 = x3, ±з ——Xi-
Compute the solution for a given initial data (xi, x2, x3)(0) = (xi, x2, x3).
2.8.	Write the second order equation
x + x + x = /(t)
as a first order system. Write a formula for the solution, with initial data
x(0) = x(0) = 0.
2.9.	Consider a linear pendulum with external force:
0-0 =	0(0) = 0o, 0(0) = 0.
(a)	Compute the solution for u(t) = sign(0(£)) and 0q = тг/2.
(b)	Consider the solution 0e for the same forcing term, corresponding to
initial data 0£(O) = тг/2 + е. Determine the tangent to the curve £ —> 0е(тг)
at £ = 0.
(c)	Consider the forcing terms ue such that u£(t) = 0 for 0 < t < £ and
ue(t) = sign(0(t)) otherwise. Let ye be the corresponding solutions for
Oq = 7t/2. Compute the tangent to the curve £ —* y£ (тг) at £ = 0.
Hint: compute the times t = t(e) where 0(t) = 0.

3
Control Systems
In this chapter we begin a study of the control system:
x = f(t,x,u),	iz(-) e Z7,	(3.1)
where the set of admissible controls is defined as
U — { w(-) measurable, u(t) G U for all t}	(3.2)
We shall assume the following basic hypothesis:
(H) The set U C Hlm of control values is compact, f2 is an open subset of
1R x IRn, the function f : Г2 x U i—> IRn is continuous in all variables and
continuously differentiable w.r.t. x.
We say that an absolutely continuous function #(•) defined on some interval
[a, b] is a solution of (3.1) if its graph {(t, x^t)); a < t < b} is entirely contained
in <2, and if there exists a measurable control u, taking values inside U, such
that ±(t) = f(t, u(t)) for almost every t G [а, Ь].
The main goal of this chapter is to provide information about the dynamics
of the system (3.1). We will show how to construct solutions to (3.1), and study
how trajectories depend on the choice of the control function u(-).
The first section compares a control system with a differential inclusion,
showing that the two mathematical models are essentially equivalent. In Sec-
tion 3.2 we apply the basic results on O.D.E’s with measurable right hand side,
and derive the existence and uniqueness of the solution, for a given control
function t u(t).
As the control ?/(•) varies, a whole set of possible trajectories is obtained.
Some basic properties of this family of trajectories are described in the follow-
ing Sections 3.3-3.4. In particular, it is important to understand which points
can be reached at a given time T, by suitably choosing the control function.
Preliminary results in this direction are given in Section 3.5.
The case of linear systems with constant coefficients is studied in detail
in Section 3.6. This is a classical topic in engineering literature, with exten-
36
3 Control Systems
sive applications. Thanks to an explicit formula representing the trajectories,
precise results can here be given.
In turn, every nonlinear system can be locally approximated by a linear
one, in a neighborhood of a given point. By a linearization method, from a
global controllability result valid for linear systems, in Section 3.7 we thus
obtain a local result valid for general nonlinear systems. For a special class of
systems, where the control variable enters linearly in the equations, further
controllability results are given in section 3.8, based on the analysis of Lie
brackets.
According to the analysis in Section 3.3, the set of trajectories of a control
system is closed if, at each point x. the set of possible velocities F(x) C IRn
is closed and convex. When this key property fails, one can always construct
an auxiliary system, where the velocity sets F(x) are the convex hulls of the
original ones. These “chattering systems” are introduced in Section 3.9. They
are particularly useful in connection with optimization problems, because one
can prove that an optimal control problem always admits a generalized solu-
tion in the form of a “chattering control”. In the special case of linear systems,
for any chattering control one can find a genuine control of the original sys-
tem which steers the system exactly to the same terminal point. For example,
points reached by controls u(t) G [—1,1] can also be reached using controls
which take values only in the two end-points: u(t) = 1 or u(t) = — 1. This is
the content of the famous bang-bang theorem, given in Section 3.10.
3.1 An equivalent differential inclusion
In connection with (3.1), define the multifunction
F(t.x) = {f(t,x,a;) : a; e U}
and consider the differential inclusion
x e F(t,x).
(3.3)
(3-4)
Observe that, for each (L^), the set of admissible velocities x in (3.1) is given
in parametrized form, as the image of the fixed set U c IRm. On the other
hand, when we study the differential inclusion (3.4), we do not assume any
parametrization of the set F(t,x) C IRn, see Figure 3.1. The next result shows
that these two approaches are essentially equivalent.
Theorem 3.1.1. (Filippov). An absolutely continuous function x : [a, b]
Rn is a trajectory of (3.1) if and only if it satisfies (3.4) almost everywhere.
Proof. 1. The fact that every solution of (3.1) is a solution of (3.4) is an
immediate consequence of the definitions.
3.2 Fundamental properties of trajectories 37
Fig. 3.1. Parametrized and non parametrized set of velocities.
2. Viceversa, let x(-) be a solution of (3.4). Fix an arbitrary element w in the
control set U. and define the multifunction
W(t\ = f {w e U:	w) = ±(i)} if x(t) € F(t,x(t)),
[ {w}	otherwise.
Notice that the second alternative holds if either the function rr(-) is not
differentiable at the time t. or else if ±(i) exists but does not lie in the set
F(£, By assumption, this happens only on a set of times of measure zero.
We now define the control и by choosing u(t) as the first element of the
set W(Z) w.r.t. the lexicographical ordering. Such an element exists because
each set W(£) is compact.
3. By construction it follows that x(t) = /(L z(f), ?/(£)) for almost every t e
[a, b\. To prove that the control function u(-) is measurable, we use Lusin’s
theorem and construct a sequence of disjoint compact sets J?,... with
Ji C [a,b],
meas [a, b] \ |^J Ji I = 0,
\ /
and such that the derivative £(•) is well defined and continuous, restricted to
each Ji. Therefore, restricted to Л, the bounded multifunction W has closed
graph. By Theorem A.7.3 in the Appendices, for t e Ji the lexicographic selec-
tion t u(t) e IV(i) is measurable. Since the sets Ji cover almost the entire
interval [a, b], we conclude that the selection и : [a, b] h-> U is measurable.
3.2 Fundamental properties of trajectories
Let /, U satisfy the basic hypothesis (H). Let an initial value x be given. For
any measurable control и : [0, T] U, the Cauchy problem
38	3 Control Systems
±(t) = /(t,a?(t), u(t)), rr(O) = ж,	(3.5)
is equivalent to
±(t) = g(£,a:(t)),	#(0) = x,	(3.6)
where g(t,x) = f(t,x,u(t)) satisfies the basic assumptions (A) and (B) of
Chapter 2. By the results proved in Chapter 2, if (0,x) E Г2, there exists
e > 0 such that the Cauchy problem (3.6) has a unique local solution, defined
on the small time interval [0,s]. To study how the solution ;r(-,u) varies with
the control u. we first consider the globally bounded case, assuming
(H*) The set U C IRm of control values is compact. The function f =
f(t,x,u) is defined and continuous on the entire space IR x IR7' x U.
continuously differentiable w.r.t. ж, and satisfies
|/(Ж,«)| < C, \\Dxf(t,x,u)\\ < L.	(3.7)
for some constants C, L and all t,x,u.
Theorem 3.2.1. (Global existence and continuous dependence). Let
the assumption (H*) hold. Then, for every T > 0, и G U, the Cauchy problem
(3.5) has a unique solution x(-,u) defined for all t E [0,T]. The “input output”
map u(-) (—► rr(-, u) is continuous from L1 ([0, T]; IR771) into C° ([0, T]; IRn).
Proof For each control u, the trajectory #(-, iz) is the fixed point of the trans-
formation w i—> Ф(и, w) defined by
Ф(и, w)(t) = x + I f(s,w(s),u(s)) ds.	(3.8)
./o
The theorem is proved showing that Ф : Л x X X satisfies the assumptions
of the Contraction Mapping Theorem A.2.1 in the Appendix. Here we use the
spaces X = C°([0, T]; R”) and Л = Ll([0,T]; Rm), so that the control func-
tion u(-) plays the role of a parameter. On the space X we use the equivalent
norm
Mt = m<K e“2Lt|w(f)|.	(3.9)
1. By the hypothesis (H*), the map Ф is well defined. If (zzp)p>i is a sequence
of admissible controls approaching a in the L1 norm, for any subsequence и„'
we can extract a further subsequence u„" such that и„"(Р) —> u(t) for almost
every t. By the Lebesgue dominated convergence theorem, it follows
lim /	|/(s, w(s), u„(s)) — f(s, zz(s))l ds = 0.
p-oo	I	I
Since the subsequence was arbitrary, we conclude that the above limit
holds for the entire sequence uu. Therefore, for each continuous function w,
the image Ф(ир, ш) converges to $(zz, w) uniformly on [0, Т]. This proves the
continuity of Ф w.r.t. u.
3.2 Fundamental properties of trajectories
39
2. For any fixed u. the second inequality in (3.7) implies
|/(i,x, u) - f(t,y,u)\< L\x - y|.
To prove that Ф is a strict contraction w.r.t. the second variable, assume
||w — w'||| = 6. Recalling the definition (3.9), one has
|w(s) — w'(s)| < Se2Ls for all s € [0,T].
Therefore, for every t > 0 we obtain
e 2Lt |Ф(и,	- Ф(и, w')(t)| =
= e~2Lt
f(s, w(s), u(sf) — f(s, w'(s),u(sy) ds
L\w(s) — w'(s)\ds
L6e2La ds < -.
2
This implies
||<?(u,w) -<Z>(u,w')llt <	— wZ^+ *
(3.10)
We can now apply Theorem A.2.1 and obtain the existence of a unique fixed
point x = Ф(и,х), for every given control function u. By the definition of Ф,
this provides a solution to the Cauchy problem (3.5).
In general, if f is only continuous w.r.t. u, under the assumptions of Theo-
rem 3.2.1, the map u(-) •—> a?(-,u) may not be Lipschitz continuous from L1
into C°. This can be easily seen from the example
i(t) = f(t,x,u) = u1/3,	#(()) = 0, u(t) e [0,1].
The input-output map, however, turns out to be always Lipschitz continuous
w.r.t. the distance
d(u, v) = meas {t E [0, T]; u(t) v(t)}	(3-11)
on the set of admissible controls
Id = {и : [0, T] i—> U; и measurable} .
(3-12)
Proposition 3.2.2. Let T > 0 be given, and assume that f, U satisfy the
basic hypothesis (H*). Then the input-output map u(-) •—> x(-,u) is Lipschitz
continuous from the complete metric space
1Л = {и : [0, T] »—> U; и measurable} ,
with the distance d(-,->) defined at (3.11), into C°([0, T]; IRn).
40	3 Control Systems
Proof. Identifying controls which differ on a set of measure zero, it is clear
that U is a complete metric space. Observing that
||u — v||li < d(u,v) • max |o> —o/|,
from the proof of Theorem 3.2.1 it follows that the transformation Ф defined
at (3.8) maps U x C° continuously into C°, and that (3.10) holds. Therefore,
all assumptions of Theorem A.2.1 hold, where the metric space (U,d) plays
the role of A and where C° with the equivalent norm || • || f plays the role of
X. Define the constant
M = max {|/(t,#,u?)|; t G [0,T], |a?| < |^| + CT. cuGU}.
Using (A.6) with у = :r(-,u), Л = v, we obtain
||.7,-(-. u)-®(-,v)||f < 2||ar(*,«)-#(v,x(-,u))||t
= 2 max e-2Lt / f(s, x(s, u),u(s)) ds — / /(s,x(s, u), v(s)) ds
t€[0,T] Jo	Jo
<2 / |/(s,x(s,u),u(s)) — f(s, x(s,u), v(s))| ds
Jo
< 2T-2Md(u,v}.
Hence, for any u. v G U one has
max |x(t,u) — j:(t,v)| < ^MTe2LT • d(u, v),
te[o,r]
proving the Proposition.
If in place of (H*), the control system satisfies the weaker hypothesis (H),
for a given control и : [0, T] U the solution x(-,u) of (3.5) may not be
defined on the entire interval [0, Т]. Indeed, as in figure 2.2, the norm of
the solution \x(t. u)\ may approach -hoc, or else (t,;r(t,u)) may approach the
boundary of the domain C, before time T.
From Theorem 2.1.5 we immediately obtain an estimate on the solutions
to the Cauchy problem (3.5).
Theorem 3.2.3. (A-priori bounds on trajectories). Let f : J?xU »-► IRn
satisfy the basic assumption (H). Let ф : [to, ti] x IR •—> R be a scalar function,
measurable in t and continuous in x, such that
max	|/(t, x, a?)|	for all t,r.	(3.13)
|x|=r, iveu
Let 11—> r(t) be an absolutely continuous function such that
r(t)>-0(t,r) for a.e. tc[to,U], r(to) > |#| •	(3-14)
3.2 Fundamental properties of trajectories
41
If the set К = {(t,x) : 0 < t < T, |x| < r(£)} is contained in L2, then
for every admissible control и : [0, T] i—> U the Cauchy problem (3.5) has a
solution #(•) defined on the entire interval [0, T], which satisfies
|^(£)|<r(£), for all t 6 [0, Т].	(3.15)
Proof. Given any control и : [0, T] i—► U, consider the function g(t,x) =
f(t, x, u(tY). The result then follows from Theorem 2.1.5, applied to the
Cauchy problem (3.6).
As in Corollary 2.1.6, as a special case we obtain
Corollary 3.2.4. In addition to the hypothesis (H), assume that f : [0, -Foo) x
]Rn x U satisfies
\f(t, x, u)| < C(1 4- |ж|) for all t,x,u.
Then, for every admissible control ti(-), the solution to (3.1) with x(0) = x
satisfies
|яг(£,и)| < ect|.t| + (eCt - 1) .	(3.16)
For future applications, we now prove a further related result: if the solu-
tion t i—> x(t,u) corresponding to a given control w(-) is well defined on the
whole interval [0, T], then the same holds for all controls v(-) sufficiently close
to и in the L1 norm.
Proposition 3.2.5. Let the basic hypothesis (H) hold. Let the solution x(,u)
of (3.5) corresponding to the control и be defined on the entire interval [О, Т].
Then there exists p > 0 such that
(i) For every vEU with ||zz - v||u < p, the trajectory #(-, r) is defined on the
entire interval [0, T]. The map v(-) »—> x(,v) is continuous from Ll into
C°.
(ii) For every control in the set Up = {t'E U\ d(u,v) < p}, the trajectory
x(-,v) is defined on the entire interval [0,Т]. The map ?;(•) f-> x(,v) is
Lipschitz continuous from Up into C°, w.r.t. the distance (3.11).
Proof. Construct a smooth cut-off function ф : IR x IRn н-> IR such that ф = 1
on a neighborhood Af of the graph of the trajectory {(t, x(t, u)); t e [0,T]},
while ф = 0 outside a compact set К C 12. Then the function
ft (4	«)/(*, a:,u) if (*,*) € 12,	(r.
satisfies the global bounds (H*). Hence the previous results apply to /f. By
the continuous dependence proved in Theorem 3.2.1, for all controls v(-) with
||v ~ ^IIl1 sufficiently small, the solution 1i—► x\t,v) of
x = f\t, x, v),	z-(0) = x	(3.18)
42
3 Control Systems
is defined on [0, T] and remains inside J\f. The same holds whenever the dis-
tance d(u, v) defined at (3.11) is sufficiently small. Since f = onff, x\-,v)
coincides with the solution x(-,v) to the original problem (3.5). From The-
orem 3.2.1 we thus deduce the statement (i), while (ii) is a consequence of
Proposition 3.2.2.
The continuous dependence of trajectories on the control function и is a
basic result. However, in the analysis of optimal control problems, stronger
regularity properties are needed. In the next theorem we consider a reference
control и and a one-parameter family of perturbed control functions, of the
form u£ = и + eAu. If f is differentiable also with respect to the control
values, we show that the corresponding family of trajectories x(•,?/ + eAu) is
differentiable w.r.t. the parameter e.
Theorem 3.2.6. (Differentiability w.r.t. the control). In addition to the
basic hypothesis (H), assume that f is defined on V, with V open neighbor-
hood ofXJ, and is continuously differentiable w.r.t. u. Let u(-) EU be a control
whose corresponding solution #(-, u) of (3.5) is defined on [0, Т]. Then, for ev-
ery bounded measurable Au(-) and every t E [0, T], the map e h-► x(t, u + eAu)
is differentiable. Its derivative at e = 0 is
d
—-x(t, и T еЛ?/)|е=0 = / M(t, s)Duf (s, x(s, u), u(s)) • Au(s) ds. (3.19)
Jo
Here Duf denotes the n x m matrix of partial derivatives dfi/duj, and M is
the matrix fundamental solution for the linearized problem
v(t) = Dxf(t,x(t, u),u(t)) • v(t).
(3.20)
Proof. The result will first be proved under the additional assumptions (H*),
then in the general case.
1.	Call z(t) the right hand side of (3.19). Recalling Theorem 2.2.3, z is a
solution to
z(t) = A(t)z(t) T Duf(t, x(t, u), u(t)) ♦ Au(t), z(0) = 0,	(3.21)
with A(t) = Dxf(t,x(t,u),u(t)). Next, define
x£(t) = x(t, и + eAu), ?/c(t) = x(t,u) + Ez(t). (3.22)
To prove the theorem, we need to show that
lim =0 (323)
£—о	г
2.	Observe that же(-) is the fixed point of the map w i—> Ф(и+еДи, w) defined
at (3.8), which is contractive with respect to the equivalent norm || • ||| defined
3.2 Fundamental properties of trajectories
43
at (3.9). Thinking of as an approximate fixed point, using the estimate
(A.6) in the Appendix with к = we obtain
1	2
-|ke-3/e||t < -||Ф(ы + еД«, 3/e) — 3/ellf-
To prove (3.23) it therefore suffices to show that
lim I sup -
£_>0 UG|O,T] £
= 0.
(3-24)
3.	Recalling (3.21) and the definition (3.22) of yc, we obtain
1
E
4- ez(s), u(s) 4- eAu(s)^ ds — x(t) - ez(t)
4- j Duf x(s,u), u(s)j - eAu(s) ds — x(t) ~ £z(t)
4- / / l/9r/(.s, x(s. u) + crs2(s), u(s) + aeAu(s))
Jo Jo L
—Dxf(s, x(s,u), ?i(s))j • ez(s) dads
+ J [z)u/(s, x(s, u) + aez^sy u(s) 4- aeAu^s))
—Duf(s, x(s,u), u(s
eAu(s) dads
У У ||dx/(s, x(s,u) -I- aez(s), u(s) 4- aeAu(s))
—Dxf(s. x(s,u), u(s)) | • |z(s)| dads
4- У У ||£>u/(s, a;(s, u) 4- aez(sy u(s) 4- aeAu^s))
—Duf(s, x(s,u), tz(s)) • IAu(s)| dads .
(3.25)
By the Lebesgue Dominated Convergence Theorem, the right hand side of
(3.25) converges to zero, proving (3.24) and hence (3.23). This completes the
proof under the additional assumption (H*).
4. To cover the general case, it suffices to consider an auxiliary function
defined as in (3.17), which satisfies (H*) and coincides with f for all (t,x) in a
neighborhood of the graph {(£,a?(t, u)); t e [0,T]}. The result then holds for
the system (3.18), hence for the original system (3.5) as well.
44
3 Control Systems
3.3 Closure
In this section we study one of the key qualitative properties of the set of
trajectories, namely its closure. Consider a sequence of admissible controls
uy and assume that the corresponding solutions .?(•, of (3.5) converge
to #(•) uniformly on [0,T]. Our main concern is whether this limit trajectory
is a solution of the original control system. This is the case if we can find
a control u(-) such that x(t) = f(t, x(t), u(t)) for a.e. time t. In general,
this may not be the case. Indeed, even if the trajectories xy(-) converge, the
controls Uy(/) might have a highly oscillatory behavior and not converge in
L1.
Example 3.1	. Consider the system on IR:
x(t) = u(t), z(0)=0, u(t) e {-1,1} a.e.	(3.26)
For t e IR, define
1 if sin(i/t) > 0,
—1 if sin(i/t) < 0.
As shown in figure 3.2, the sequence of trajectories t ь-> x(t, uy) converges to
zero uniformly for all t e IR. However, .t(£) = 0 is not a solution of (3.26).
Fig. 3.2. A sequence of highly oscillatory controls and their trajectories.
The closure of the set of trajectories is best studied within the framework
of differential inclusions.
Theorem 3.3.1. (Closure of the set of trajectories). Assume that the
multifunction (t,x) »—> F(t,j?) is Hausdorff continuous on IR x IRn with
compact convex values. Then the set of trajectories of (3.4) is closed in
C°([0,T];Rn).
3.3 Closure
45
Proof. 1. Let xp(-) be a sequence of trajectories of (3.4) tending to #(•) uni-
formly on [0,T]. Since the sets F(t, x) are uniformly bounded as Lx range in
a compact domain, the xI/(-) are uniformly Lipschitz continuous. Therefore,
the function x(-) is Lipschitz continuous as well, hence differentiable a.e. on
[0,T]. To prove the theorem, we thus need to show that
x(t) G F(t,x(t)).
(3.28)
at each time r where the time derivative x(t) exists.
2. Assume, on the contrary, that x(t) exists but the inclusion (3.28) fails. Since
the set of velocities F(t, x(r)) is compact and convex, by Lemma A.8.5 in the
Appendix the two sides of (3.28) can be strictly separated by a hyperplane.
As shown in figure 3.3, there exists s > 0 and a unit row-vector p G lRn such
that
p у <p • i(r) — 3s for all у G F(t, x(r)).
By continuity, there exists 6 > 0 such that, for \t — t| < 6 and |x' — x(t)| < J,
one still has
p • у < p • x(t) — 2s for all у G F(t, xf).
(3.29)
Recalling that the map t *—> x(t) is differentiable at t = t, we can choose
r' G ]t, t + 5] such that
X^T \— ±(r) < e,	|x(t) - х(т) I < 8 for all t G [r, r'].
т — т
By uniform convergence, we now have
On the other hand, for all v sufficiently large the bound (3.29) implies
xy(r') -хДт)
T' — T
! p • Xy(t) dt <p • x(t) — 2s .
T— T
This contradiction proves (3.28)
Using Theorem 3.1.1, the previous result can be applied to the control
system (3.1).
Corollary 3.3.2. Let the basic assumptions (H) hold. Let xl/(«) be a sequence
of solutions to (3.1) converging to x(-) uniformly on [О, Т]. If the graph
{(£,x(t)); t G [0, T]} is entirely contained in L2 and all sets of velocities
F(t,x) = {f(t,x,u); и G U} are convex, then x(-) is also a trajectory of the
control system (3.1).
46
3 Control Systems
F(T, x(t))
Fig. 3.3. Left: the speed x(t) is strictly separated from the set F(t, x(r)). Right: if
the sequence of trajectories xv^) converges to rr(-) at time t = r, it cannot converge
also at time r'.
Remark 3.1. In the above results, the key assumption is the convexity of
the sets of velocities F(t,x) C IRn, not the convexity of the set of controls
U C IRm.
If U is convex and the function f is affine w.r.t. the control variable u,
then every set F(£, x} is convex. The most general function f of this type can
be written in the form
f(t,x, u) — g(t, x) + B(t, x)u
where B(t,x) is an n x m matrix, for each given t,x.
On the other hand, if U is convex but f is non-linear w.r.t. the variable
u, then the sets F(t,x) = {f(t,x,u); и € U} may not be convex, in general.
Example 3.2	. Consider the control system on Ш2
(±i,±2) = (zz, 1 - u2)
и e U = [-1. 1].
Here U is convex. However, consider the sequence of rapidly oscillating con-
trols u^t) as in (3.27). Starting from the origin at time t = 0, the corre-
sponding trajectories t •—> x^t) = х(1,и„) converge to the null trajectory
= (0,0). uniformly w.r.t. t. However, this is not a trajectory of
the system, for any control u(-). Here the non-convexity of the set of velocities
F{t,x) = {(s/1,2/2); 2/2 = 1 — 3/1 , У1 € [-1,1]}
yields a set of trajectories which is not closed.
3.4 Density
47
3.4 Density
In this section we study what happens if the set U of admissible controls is re-
placed by a smaller one. For example, instead of using all measurable controls
и : [0, T] •—> [—1,1], suppose we can only use piecewise constant controls, or,
say, controls taking only the two values +1, —1. Do these limitations substan-
tially reduce our ability to control the system? Two results in this direction
are presented below.
Theorem 3.4	.1. (Density of trajectories with piecewise constant con-
trols). In connection with the Cauchy problem (3.5), let f, U satisfy the basic
hypothesis (H). Then the family of trajectories corresponding to piecewise con-
stant controls is dense in the set of all solutions (with measurable controls).
Proof. Given a measurable control и 6 U, assume that the corresponding
solution x(-,u) of (3.5) is defined on [0, Т]. Construct a sequence of piecewise
constant controls uy G Id converging to и in L1. If f satisfies the global bounds
(H*), by Theorem 3.2.1 the corresponding trajectories x(-,uy) converge to
x(-,u) uniformly on [0, Т]. To prove that the uniform convergence holds also
in the general case, it suffices to consider an auxiliary function f\ defined as in
(3.17), which satisfies (H*) and coincides with f for (t, x) in a neighborhood
of the graph {(t, x(t, w)); t 6 [0, T]}. The result then holds for the system
(3.18), hence for the original system (3.5) as well.
The next result describes under what conditions one can replace the set U
of control values by a smaller set U' C U and still be able to approximate all
the trajectories of the original system. By cd(S) we denote the closed convex
hull of a set S C Rn, i.e. the intersection of all closed convex sets which
contain S.
Theorem 3.4	.2. (Approximation using a smaller set of controls). Let
f, U satisfy the assumptions (H). Consider a subset U' C U such that
cd{f(t,x,u)-, и G U'} D {f(t.x,u); и G U} forallt,x. (3.30)
The every trajectory of
i = f(t,x,u}, x(0) = x, u(t)GU.	(3.31)
can be approximated by a trajectory of
x = f(t,x,u), rr(O) = x,	u(t) G U',	(3.32)
uniformly on bounded intervals.
48
3 Control Systems
Fig. 3.4. Constructing a trajectory which remains within the tube Г£ around x
Proof. 1. Let t x(t) be a trajectory of (3.31), corresponding to the admissi-
ble control t и-► u(t), say, defined for t e [О, Т]. We have to show that x can be
approximated by trajectories of (3.32), uniformly on [0,Т]. By Theorem 3.4.1,
it is not restrictive to assume that u is piecewise constant, right continuous.
2.	Fix a radius p > 0 arbitrarily small, and consider the tube around the
reference trajectory x
By the assumption (H), there exists a Lipschitz constant L such that
u) — f{t,x\u)\ < L\xfx'\	for all (t, x), (£, x') ё Г, uCU.
Define the positive, increasing function
V’(t) = eLt - | .
О
Choose e > 0 sufficiently small so that £^(T) < p. Note that this implies that
the tube
Ге = |(t,x); t € [0,T], |rr - x(Z)| < s^(t)} .	(3.33)
is contained inside Г.
3.	The theorem will be proved by constructing a piecewise constant control
и : [0, T] U' whose trajectory remains inside the tube Ге. namely
\x(t, u) - x(t)\ < s^(t)	for all te[0,T].	(3.34)
The main constructive procedure is shown in figure 3.4, At time to = 0 we
choose an arbitrary control value uq € U'. We define u(P) = Uq until the first
3.4 Density
49
time ti when the trajectory x(-) hits the boundary of Д, We then choose a
new control value Ui € U' such that, at time t = ti, the velocity ±(ti) =
f(ti, rr(Zi), щ) points strictly inside the tube Г£. We take u(t) = u\ until
the next time, say t2 > £1, when the trajectory hits again the boundary of
Г£, etc... In the following, we describe the inductive step more in detail, and
show that in a fine number of steps one can cover the whole interval [0,T].
4.	Assume that the piecewise constant control и has already been constructed
on the time interval [0, ti] and the corresponding solution satisfies
\x(t) — x(t)| <	for all t € [0, ti].
Define the unit vector pi =	and choose Ui e U' such that
щ), pi) > (f(ji, x(ti), u(ti)), pi) -	.	(3.35)
In other words, among all possible velocities, we choose one which has almost
the largest possible inner product with the unit vector pi, in the direction of
x(ti) — x(ti). Notice that an element щ G U' satisfying the inequality (3.35)
certainly exists, because of the key assumption (3.30).
5.	We now extend our solution rr(-) beyond time ti using the constant control
u(t) = Our main concern is the distance between x(t) and the reference
solution x(t). Using (3.35) and the Lipschitz continuity of f w.r.t. x, at the
time t = ti the derivative of the distance can be estimated as follows:
ft{ l*(0 -*(01	,+
= (/(*»,	«(*<)) - /(£»,	, pi) - eLeLti
<	x(ti), - f(ti, x(ti), Ui) , Pi)
+ i(ti), щ) — /(ij, x(ti), -u»)| - eLeLti
<	+ Le (eLti - I) - eLeLti
____eL
-___3 ’
Therefore, on some open interval ]ti, ti + the solution remains strictly
inside the tube Г£. We can thus define u(t) = щ on the interval [ti, tj+ih
where is the first time > ti when the solution satisfies
|x(t)	= e&(t).
At time t = we choose a new control value , and so on.
6.	To complete the proof, we have to make sure that, in a finite number of
inductive steps, we can cover the whole time interval [0, Т]. By assumption,
the control й is piecewise constant, say u(t) = uj € U for t G Ij = [tj, Tj+i[,
j = 1,... ,N. It thus suffices to show that, in a finite number of steps, we can
50	3 Control Systems
cover each of the intervals Ij.
For every boundary point (t, y) G дГ£, with \y — £(т)| = £^(t), t G Ij, there
exists a control w G U' such that the solution to the Cauchy problem
x = f(t,x,w),	x(r) - у,
satisfies
|x-(t) — x(t)\ <	for all fe]r, t + J[,	(3.36)
for some 6 — \T.y) > 0 depending on r, y. By continuity, the same control
can be used for all points (rf,yf) 6 дГ£ sufficiently close to (r,y). The
corresponding length in (3.36) depends continuously on rf,yf. We can
now cover the compact set
|(т,у); т 6 [г,, t>+1] , |y - i(r)|
with finitely many open sets Jt where the function 5 = \r,y) remains uni-
formly positive. Therefore, in the previous construction, we can always achieve
> ti T 6
for some fixed 5 > 0. The inductive procedure thus terminates after a finite
number of steps.
An important case where the key assumption (3.30) holds is the following.
Assume that the control system is linear w.r.t. u:
x = h(t, x) + A(t, x) - u,
each A(t,x) being an n x m matrix. Assume U' C U and call F,F' the
corresponding multifunctions, defined as in (3.3). Then, by linearity,
coU'DU => cdF'(t,x) D F(t,x) for all t,x. (3.37)
The implication (3.37), however, is usually false when f is nonlinear.
Example 3.3	. Let f.g be smooth vector fields on lRn. Then the set of tra-
jectories of
x = f(x) + g(x)u(t) a:(0) = 0, u(t) G { — 1,1} a.e.
is dense in the set of trajectories
x = f(x) + g(x)u(t) rr(O) = 0, u(t) G [— 1,1] a.e.,
because co{ —1,1} = [—1, 1] and the velocity x is a linear function of u.
Example 3.4	. On IR2, consider the systems
3.5 Reachable sets
51
(ii,±2) = (u, 1 - it2),	(^1,ж2)(0) = (0,0), u(t) € U = [-1,1], (3.38)
(±i,i2) = (tz,l -zz2), (a?i,x2)(0) = (0,0), u(t) e U' = {-1,1}. (3.39)
Then coU' = U, but the set of solutions to (3.39) is not dense in the set of
solution to (3.38). Indeed, taking w(f) = 0 one obtains a solution to (3.38):
(xi,x2)(t) = (0,t).
This solution cannot be approximated by trajectories of (3.39), because u(t) G
{-1,1} implies ±2(t) = 0. In this case, the key assumption (3.30) fails.
3.5 Reachable sets
In this section we consider a control system whose dynamics is independent
of time:
± = /(ж, ?z),	a:(0)=^,	w(-)eZ7,	(3.40)
where the set U of admissible controls is given by (3.2). The reachable set
R(r, x) at time t, starting from x, is then defined as
R(r, x) = < x(t) ; rr(-) is a solution of (3.40) with a?(0) = x
Fig. 3.5. The reachable set at time т > 0, starting from the point x.
More generally, given a set К C IRn of initial states, we define R(t, K) as
the reachable set at time t, starting from points in K:
R(r,K) = |z(t) ; x(-) is a solution of (3.1) with x(0) € k\ .
The next theorem establishes the closure of the reachable sets, under a
suitable convexity assumption. In a later chapter, this closure property will
be of great importance, providing the existence of optimal controls.
52
3 Control Systems
Theorem 3.5.1. (Closure of the reachable set). Let f, U satisfy the basic
assumption (H). Assume that the graphs of all solutions of (3.40) starting from
any point x 6 К are contained in some compact set К' c L2, for t G [О, Т].
If all sets of velocities F(x) = {f(x,w); tv € U} are convex, then, for every
т G [0,T], the reachable set R(t,K) is compact.
Proof. Let be a sequence of points in R(r, K) tending to By defini-
tion, for each v we have £y = xu(r) for some solution to (3.40) with хДО) G K.
By assumption, the graphs of these solutions remain inside the compact set
K'. Therefore the derivatives xy are uniformly bounded and the solutions
xy(-) are uniformly Lipschitz continuous. According to Theorem A.4.1 we can
extract a subsequence which converges to a limit function x(J uniformly on
[0,т]. Clearly x(0) = lim^—oo хДО) G К because К is closed. Moreover, be-
cause of the convexity of the sets F(x), by Corollary 3.3.2, there is a control
function zz(-) G IL such that x(t) = /(x(t),u(f)) for a.e. t G [0, т]. We now
have
£ = lim £y = lim ху(т) = x(r) G R(t, K),
u—►ОС	V—»oo
showing that the reachable set R(r,K) is closed.
Example 3.5	. On IR2 consider the system (see figure 3.6)
(±i, ±2) = (^, ^i), Cei,#2)(0) = (0,0), u(t) G U = {-1,1}.	(3.41)
On a fixed interval [0, T], consider the sequence of rapidly switching controls
defined at (3.27). The corresponding trajectories satisfy
xi(£,u„) = / uy(s)ds —> 0 uniformly on [0,T],
Jo
X2(t>uiS) = / [^i(s,Wx/)] ds —> 0 uniformly on [0, Т].
Jo
However, the limit trajectory x(t) = (0,0) is not an admissible solution of the
system (3.41). Indeed, if t ► (xi (t), x2(t)) is a solution, then X\(t) G { — 1,1}
implies X\(t) / 0 at almost every time t. Hence
x2(T) = f x2(t)dt>0.
Jo
We conclude that the reachable set /?(т) starting form the origin is not closed,
for every т > 0.
In general, it is impossible to give an explicit formula for the reachable
sets. In certain cases, some information can be obtained by comparing the
reachable set with the sub-level sets of a given function ф : Hn h-> IR.
3.5 Reachable sets
53
Fig. 3.6. Each set of velocities F(x) C IR.2 for the system (3.41) contains exactly two
vectors and is not convex. A highly oscillatory control produces a trajectory xu
that remains close to the origin, but always has a strictly positive second component.
The reachable set R(t) is not closed because it does not contain its lower boundary.
Theorem 3.5.2. (Outer estimates on the reachable sets). Consider the
control system (3.J0), satisfying the basic assumptions (H). Let ф : IR7'i—> IR
be a C1 function such that
\7ф(х) • /(a?, u) < 1 whenever </>(rr) € [0, T], и G U .
If the set К of initial states satisfies
К C {z ; ф(х) < 0} ,	(3.42)
then for every т G [0, T] the reachable set R(t, K) is contained in the corre-
sponding sub-level set of ф, namely (see figure 3.7)
R(r, К) С {x; ф(х) < г} .	(3.43)
Fig. 3.7. The reachable set R(r, K) is contained in the level set {ф(х) < r}.
Proof Let x(-) be any solution of the control system, starting from a point
xtK. Using the chain rule, we obtain
^</>(x(t)) = V0(x(i)) • i(t) = V0(x(t)) • /(x(i), u(f)) < 1.
at
54
3 Control Systems
Therefore
</>(х(т)) < 0(ж(О)) + [
Jo
) dt<r.
This proves (3.43).
Theorem 3.5.3. (Inner estimates on the reachable sets). Consider the
control system (З.^О), satisfying the basic assumptions (H). In addition, as-
sume all sets of velocities F(x) = {f(x, ca); co G U} are convex. Let
ф : IRn 1—> IR be a C1 function whose level sets are bounded and such that
max Vd(a;) • f(x,cS) > 1 whenever t G [0,T], ф(х) E [0,T].
If the set К of initial states satisfies
К 3{x; ф(х) < 0} ,	(3.44)
then for every т G [0, T] the reachable set R(r,K) contains the corresponding
sub-level set of ф, namely
R(r, К) Э {37; 0(x) < t} .	(3.45)
Fig. 3.8. Construction of a trajectory reaching the point y.
Proof. 1. Consider any point у G JRn such that ф(у) < т. To prove the
theorem, we need to construct a trajectory #(•) of the control system such
that a:(0) G К and x(r) = y. Because of the assumption (3.44), it suffices to
construct a trajectory t x(t) such that
z(t) = y,
ф(х(1)) < t for all t G [0,t] .	(3.46)
2. As an intermediate step, given any £ > 0, we will construct a solution such
that
я(т) = у,
0(z(t)) < t + e(r - t) for all t G [0, r] .	(3.47)
3.5 Reachable sets
55
This will be achieved defining a piecewise constant control function, starting
at t = т and working backwards in time. Consider the tube-like domain (see
figure 3.8)
Г€ = | (£, ж); ф(х) < t + е(т - t) |.
Set = t. By assumption, there exists a control value uq 6 U such that
(3.48)
By (3.48) and the continuity of V^>, the solution to the backward Cauchy
problem
x = f(x,u0),	я(т) = У
satisfies
V0(s(t)) • /(rr(t), u0) >1-6
on some open interval ]t — 5, t[. Therefore
</>(a;(f)) < t + e(r — t)	t € ]t - 5, t[ .	(3.49)
We then set u(t) = uq on an interval [ti, to], where ti is the first time where
the solution hits the boundary of the tube Te, i.e.
ti — inf < r; </>(x(ty) < t + 2s(t — t) for all t e [t',r] |.
We then choose a control value щ such that
V</>(x(ti)) • /(x(ti),ui) > 1.
This guarantees that the backward solution to x = /(ж, satisfies (3.49) on
some open interval ]ti — 5, t± [. The solution can thus be prolonged backwards
up to the next time t2 < ti where it hits the boundary of the tube Ге, etc...
By a uniform continuity argument, the entire interval [0, t] can be covered
in a finite number of steps, so that 0 = t?\r < ♦ • • < £2 <	< to = t, for some
integer N. This yields a solution of the control system which satisfies (3.47).
3. Since £ > 0 in (3.47) was arbitrary, we can now consider a sequence £m —► 0
and construct solutions rrm(-) so that
Хт(т) = у ,	фт(0) < t + £m(r - t) for all Ш>1, t € [0, t] .
(3.50)
By the boundedness of the sub-level sets of ф, all these solutions are uni-
formly bounded, hence their speeds |±m| remain uniformly bounded. By the
compactness Theorem A.4.1 in the Appendix, we can find a subsequence which
converges to some function t •—> x(t) uniformly on [0, г]. According to Theo-
rem 3.3.1, this limit trajectory x(-) is itself a solution of the control system
(3.40). By (3.50) and the continuity of ф, it is clear that (3.46) holds. This
completes the proof.
56	3 Control Systems
3.6 Linear systems
In this section we analyze in greater detail the case of a linear system :
x — A(t)x + B(t)u, x(0) — x, u(t) € U.	(3.51)
Here x e IRn, U C IRm, while A(t) and B(t) are respectively n x n and n x m
matrices, continuously depending on t.
Calling M(t,s) the matrix fundamental solution for the homogeneous
problem
x = A(t)x,	(3.52)
the solution of the Cauchy problem (3.51) can be written as
x(t) = M(t, 0) x + I M(t,s) B(s)u(s) ds.	(3.53)
Jo
We begin by examining the case where x = 0 and the set U is the entire space
IRm. The reachable set at a time t > 0 is then described by
f ft	>|
R(t) = \ M(t,s) B(s)u(s)ds; и e 1?([0,*]; IRW) к (3.54)
I Jo	J
Lemma 3.6.1. For each t > 0, The reachable set (3.54) for the linear system
(3.51), starting at x = 0, is a vector subspace of№\
Proof. Consider any two points Xi, x? € 1R(£). Assume that these can be
reached at time t using the controls ui(-), U2(-)- Then, for any Ai,A2 € IR
the point x = Ai^i T А2Ж2 can be reached using the control u(t) = Apui(t) +
A?^^)- Indeed, by linearity
M(t, s) B(s) (Aitii(s) + A2u2(s)) ds
= Ai M(t, s) B(s) U1(s) ds + A2 M(t, s) B(s) u2(s) ds
= A;Xi T A2#2 •
Complete information on the reachable sets R(t) can be obtained for the
linear system
x = Ax + Bu	u(t) e IR/n	(3.55)
where the matrices A, В are constant in time. In this case, the controllability
matrix is defined as the n x nm matrix
C(A,B) = (B, AB,..., An~} B).
Theorem 3.6.2. (Reachable subspace for a linear system). For every
t > 0, the reachable set for the linear system (3.55) starting at the origin is
precisely the subspace spanned by the columns of the n x nm controllability
matrix C(A, B).
3.6 Linear systems
57
Proof. 1. In the case of constant coefficients, the matrix fundamental solution
takes the form M(£,s) =	wjiere
ifc Лк
ЛА =	1 Л
k\
k=0
Therefore
7?(t) = {f'e^^Bu^ds-,	L^fO^R”1)
(3.56)
2.	We recall that и € IRm is a column vector, while В is a n x m matrix. Let
bi,..., bm be the column vectors of B, so that
В = I by
(3.57)
We need to show that
7?(t) = span/ Akbj ;
к = 0,..., n — 1, j =
Since we already know that R(t) is a subspace of IR71, to prove the theorem
it suffices to show that, for every row-vector p e IRri,
p • x = 0
for all x e R(i)
(3.58)
if and only if
p  Akbj = 0
for all к = 0,1,... ,n — 1, j =	(3.59)
3.	Assume that (3.59) holds. By the Cayley-Hamilton theorem, the matrix
A is a root of its characteristic polynomial. Hence there exist real numbers
co, Ci, ... , cn_i such that
n—1
Ап = ^аА*.
i=0
By induction on k, it follows that every matrix Ak can be written as a linear
combination of the n matrices A0 = Z, A, A2,..., An-1. From (3.59) it thus
follows
p • Akbj =0 for all к > 0 , j = 1,..., m.
In turn this implies
p . etA bj = 0
for all t > 0, j = 1,..., m,
58
3 Control Systems
and hence
/»t	pt m
P‘x(t,u) = p l e^~s^ABu(s) ds = /	• e^~s^A bj Uj(s) ds = 0
Jo	Jo J=1
for every control function и = (ui,..., um). Therefore (3.58) holds.
4.	Next, assume that (3.58) holds. Fix any j € {!,...,m} and choose the
vector-valued control function u(t) in (3.57) so that
zzj(Z) = l,	= 0 for i^j.
The assumption (3.58) now implies
p-x(t) = 0.
Differentiating several times this identity w.r.t. t we obtain
0 =	= P,i(0=p-(^4a;(0 + bj)>
d^
0 = -^\p-x(t)] = p-x(t) = p  Ai(t) = p  A(Ax(t) + bj),
dk	dk
° =	= p'dtkx^=p'A ^Ax^ + b^‘
Notice that here к > 0 is an arbitrary integer. At time t = 0 we have x = 0
and the above identities take the simpler form
0 = p • 5j ,	0 = pAbj, ... , 0 = p-Afc-1fy, ...
Therefore (3.59) holds.
We say that the linear system (3.55) is completely controllable if, for every
t > 0, the reachable set R(t) starting from the origin coincides with the entire
space IRn. From the above theorem we immediately obtain
Corollary 3.6.3. The linear system (3.55) is completely controllable if and
only if
Rank(B, AB, A2B, ... , Лп-1 в) = n.	(3.60)
Indeed, the dimension of reachable subspace R(t) equals the number of lin-
early independent column vectors in the controllability matrix C(A, B). Hence
Rft) = Rn if and only if the rank of this matrix is n.
3.7 Local controllability of nonlinear systems
59
Remark 3.2. If the linear system (3.55) is completely controllable, then from
any initial point xq 6 IRn we can reach every other point x\ at a given time
t > 0. Indeed, let u(-) be a control function that steers the system from the
origin to the point x\ — etAXQ. This means
#(£) = [ a)ABu(s)ds =	•
Jo
Using this same control function u(-) in connection with the initial data z(0) =
xq, at time t the system is steered to the point x±.
Example 3.6 Consider the linear system
x = Ax -Ь Bu. #(0) = 0,
x e IR3, и € IR,
where
/-1 0-l\	/ 0\
A = I 0 1-1 I , В = I 1.
\ 0-1 1/	\-i/
The controllability matrix is given by:
/ ° i 1\
C(A,B) =	1 2 4 .
\-l -2 -4/
(3.61)
The rank of C(A, B) is two, thus the system is not controllable. Let us find a
row vector p = (рьР2,Рз) which is orthogonal to B, AB and A2B\
pB = p AB = pA2B = 0.
This yields the system of three linear homogeneous equations
P2 - Рз = Pi + 2p2 - 2p3 = pi + 4p2 - 4p3 = 0 .
A non-trivial solution is p = (0,1,1). FYom the proof of Theorem 3.6.2, it
follows that the reachable set at any time t > 0 coincides with the subspace
orthogonal to p, namely
R(t) = {(^i,^2,rr3); лг2+^з = 0}«
3.7 Local controllability of nonlinear systems
We now consider a general nonlinear system, with dynamics independent of
time:
x = /(x,u)
u(t) e U С Г.
(3.62)
60
3 Control Systems
Given a point x E IR71, we say that the system is (small time) locally con-
trollable at x if, for every t > 0, the set R(t,x) contains a neighborhood of x.
Roughly speaking, this means that the system can be steered from x to all
nearby points, within a small interval of time.
From the global controllability theorem for linear systems, by a lineariza-
tion argument one can deduce a result on local controllability, valid for general
nonlinear systems.
Theorem 3.7.1. (Small time local controllability). Consider the control
system (3.40) and, assume that set U of admissible control values contains a
neighborhood of the origin 0 E IRm. At a given point x E IRn, assume that
(a)	/(£,0) = 0,
(b)	Defining the matrices of partial derivatives of f w.r.t. x and и computed
at the equilibrium point (a?, 0)
A = Dxf(x, 0), В = Du(x, 0),	(3.63)
the linearized system
x = A • (x — x) 4- В и	(3.64)
is completely controllable, i.e. A, В satisfy (3.60).
Then the system (З.4О) is locally controllable at the point x.
Proof. 1. Fix any т > 0. Since the system (3.64) is controllable, there exists
n control functions ..., u^ : [0, t] 1—> IRm such that the corresponding
solutions
X{(t) = A • (xi(t) — z) + Bu^\t),	^i(0) = x,
reach n points, say
У\ =^i(t), ... , yn — xn(r) .
By an approximation argument, as in Theorem 3.4.1, we can assume that the
controls are piecewise constant, hence uniformly bounded on the time
interval [0,т].
2. Since U contains a neighborhood of the origin, for every choice of 0 =
(01,..., 0n) € IRn with |0| sufficiently small, the control
n
i=l
is admissible, taking values inside the set U.
3. Choosing 0 = 0 E IRm, the solution of the Cauchy problem
3.8 Lie brackets and controllability
61
x = f(x,u),	rr(O) = x
(3.65)
corresponding to to the control u(t) = ua(t) = 0 is the constant function
x(t) = x. Next, call t »—> x(t,ue) the solution of (3.65) corresponding to the
control ug. According to Theorem 3.2.6, the partial derivatives of the map
в х(т, uq) at 0 = 0 are computed by
dx(r, ue)
dOi
Г e(T~s}ADuf(x(s),0)  u®(s) ds
Jo
The previous construction implies
\ dGi
дх(т, ue)
dOn
= Rank уi
Therefore, by the Implicit Function Theorem, as 0 varies in a neighborhood
of 0 G IRm, the image of the map G > х(т, iz#) covers a whole neighborhood
of the point x = x(r,uo).
Example 3.7 Consider the motion of a forced pendulum, described by the
equation
x(t) + sina?(t) = u(t)	u(t) e [—1,1].	(3.66)
Taking Xi = x, X2 = x, we can rewrite (3.66) as a first order system, namely
±1
±2
% 2
— sinxi + и
When x — 0 e IR2. The matrices of partial derivatives in (3.63) are computed
The controllability matrix here is
(B, AB) =
This matrix has full rank, hence the system is locally controllable at the origin.
3.8 Lie brackets and controllability
To derive further controllability properties for non-linear systems, some basic
tools from differential geometry are needed.
62
3 Control Systems
Given two smooth vector fields f and g on ]Rn, their Lie bracket is defined
as
[/, s) = Dxg • f - Dxf  g.
In other words, [/, g] is the directional derivative of g in the direction of f
minus the directional derivative of f in the direction of g. For various char-
acterizations and properties of Lie brackets, we refer to section A. 10 in the
Appendix.
The set of vector fields over IRn, with the Lie Bracket operation, is a Lie
Algebra , i.e. a vector space endowed with a bracket operation [•, •] such that
the following holds:
(LAI) The bracket operation is bilinear: for every ai, «2 € IR. and vector
fields Д, /2 and g, one has
[ai/i + a2f2,g] = «1 [fi,g] +«2 [/2,3],
Ь> oil fi + CK2/2] = «1 [9, /1] + «2 b> /2]-
(LA2) The bracket operation is antisymmetric: for every vector fields f and
g, one has
LAs] = -b,/l-
(LA3) The Jacobi identity holds: for every vector fields /, g and h, one has
[/• Ь- Л]] + [fl, [h, /]] + [/г. [f, 5]] = 0.
Next, consider a nonlinear control system on IRn, having the special form
i - ^Juifi(x),	u(t) = (ui(t),...,um(t)) € U, (3.67)
i=l
assuming that the vector fields f , are C°°. We define the Lie Algebra generated
by (3.67) as the smallest subspace £ C C°°, closed for the bracket operation,
which contains all vector fields ft, with i — 1,..., m. For a given point x 6 IRn,
we also consider the vector space
£(x) = {/(я) : / e £} c IR" .
We can now state the following:
Theorem 3.8.1. (Local controllability for nonlinear systems). Con-
sider the control system (3.67), and assume that the control set U C lRm
contains a neighborhood of the origin. Fix any initial point x. If
£(x) = IRn	(3.68)
for all x sufficiently close to x, then for every t > 0 the reachable set R(t,x)
is a neighborhood of x.
3.8 Lie brackets and controllability
63
Proof. As a preliminary, notice that, by choosing the control и = 0, the state of
the system remains constant. This clearly implies R(t,x) C R(t,x) whenever
0 < t < t. Moreover, by possibly choosing a smaller set of controls, we can
assume that the set U is symmetric w.r.t. the origin, i.e. и G U if and only if
—u e U. The proof will be given in several steps.
1.	Let 8 > 0 be given. For each к = 1,..., n we construct a diffeomorphism
фк from an open set Vk C IRA to a /с-dimensional manifold Mk C R(ke,x). In
particular, this will show that the reachable set R(ne, x) contains an open set
Mn = фп(Уп)-> and hence has non-empty interior. We proceed by induction
on k.
2.	Let к = 1. By (3.68) there exists an admissible control u G U such that
52^-1 Uifi(x) 7^ 0. Let t i—> 7(Z) be the trajectory corresponding to the con-
stant control u(t) = u, with 7(0) = x. Choosing Ji G]0, s[ sufficiently small,
the restriction of 7 to the open interval ] —	<5i[ is a diffeomorphism. We
then set Vi =] —	#i[, 0i = 7, and Л4Х = 7(] — Ji, <5i[)-
3.	By induction, assume that for some к < n we have already constructed
a diffeomorphism фк from an opens set Vk to some /с-dimensional manifold
Mk = Фк(Ук) C R(ke,x).
Assume that, for every x G Mk and every и G IRm, the velocity vector
52^=1 U*-A(x) i4 s always tangent to Mk . By (A.62) it follows that also the
brackets [fa, fj](x) are tangent to Mk and, by induction, C(x) is contained in
the tangent space to Mk. Since к < n, this yields a contradiction with (3.68).
By the previous argument, there exists a point x G Mk and a control
u G IRm such that ^2™x Uifi(x) is not tangent to Mk. Since the control
set U contains a neighborhood of the origin, we can here choose u G U.
By continuity, the vector 52™ X Uifify) is still not tangent to Mk , for every
у G Mk sufficiently close to x. To fix the ideas, let x = Фк(т]ъ • • •, flk)-
For convenience, we shall use the exponential notation в •—► (exp0/)(x) to
denote the solution of the Cauchy problem
dw	rz .	/лЧ
— = /(w),	w(0) = x.
With this notation, we now define
(m	\
(771
i=l	/
Here (т/i,..., ?7fc_|_i) range in an open set Vfc+X, with 77^+1 £] — suffi-
ciently close to the origin and (771,..., rjk) ranging in a small neighborhood of
(Чь • • • >%)• This achieves the inductive step.
4. When к = n, our inductive argument shows that the reachable set
Я(тге, x) C IRn has non-empty interior, since it contains the open set Mn =
64	3 Control Systems
Фп(Уп)- Choose any interior point я* E A4n and let w*(-) be a control steering
the system from the origin to x* at time T = tie. Since the system (3.67) is
linear homogeneous w.r.t. the control и and we are assuming that U — —U,
the reversed control
uf(t) = -u‘(2T-t) e и
steers the system from x* back to x, during the time interval t E [T 2Т].
5. For each у e Л4п, t G [T, 2T], call t i—> x(t,y) the solution to the Cauchy
problem
m
±(t) =	'4 (О Л(®(0) >	= у 
i=\
As у ranges in the open set A4n, the terminal points x(2T, y) cover a whole
neighborhood of x. Observing that
x(2T,y) E 7?(T, Л4П) C R(2T,x),
since T = tie can be arbitrarily small, we conclude that the system is locally
controllable.
Under very similar assumptions, one can also establish a global result.
Theorem 3.8.2. (Global controllability for nonlinear systems). Con-
sider the control system (3.67), and assume that the control set U C HU"
contains a neighborhood of the origin. Let (3.68) hold for every x E IRn. Then
for every initial point x, one has
R(x) = (J R(t,x) = !Rn.	(3.69)
r>0
Proof. To establish the result, it suffices to show that the set R(x) is at the
same time open and closed. As before, we can assume that the control set
U C JRm is symmetric, i.e. U = — U.
1.	Let x* E R(x). Hence x* E R(r,x) for some r > 0. According to Theorem
3.8.1, J?(e, x*) is an open neighborhood of U for every e > 0. Hence R(x) D
R(r + £,#*) contains x* in its interior.
2.	Now consider any point x* in the closure R(x). Again by Theorem 3.8.1,
for a fixed e > 0, the reachable set R(e, aU) intersects R(x). In other words,
there exists r > 0 and a point
у E R(r, x) П R(e, #*).
Let й : [0, t] »—> U be a control steering the system from x to y. and let
u* : [0, e] i—> U be a control steering the system from яг* to y. Define the new
control и : [0, т + s] i—> U by setting
3.9 Chattering controls
65
1l(t} = I if *€ [°’ТЬ
[ -и*(т + е —t) if te]r, т + е].
One now checks that this control tz(-) steers the system from x to a:*. Hence
x* e R(r 4- e, x) C R(x),
proving that R(x) is closed.
3.	We have proved that the non-empty set R(x) C IRn is at the same time
open and closed. Since lRn is connected, this implies R(x) = lRn.
Remark 3.3. The two previous controllability results refer to systems of
the form (3.67). In this case, if the set U of admissible control values is a
symmetric neighborhood of the origin, then the sets of admissible velocities
m
f(ж) = {52e u|
k i=l
are also symmetric. Namely, у G F(x) if and only if — у G F(x). A considerably
more difficult problem is the controllability of systems with drift, having the
form
m
x = 7o(z) +	,
«=1
u(«) = (ui(t),...,um(t)) e u.
For results in this direction we refer to [87], [52], [57], or to the monograph
[56].
3.9	Chattering controls
In cases where the sets of admissible velocities F(t,x) = {f(t,x,u); и G U}
are not convex, the reachable sets R(t) may not be closed. We next describe a
natural construction that associates with the system (3.1) an auxiliary system
±	e U# for a.e. t, (3.70)
in such a way that the trajectories of (3.70) are precisely the solutions to the
differential inclusion
x — F^(t,x) = coFityX).
Since F(t, x) C lRn, by Caratheodory’s Theorem A.8.1 in the Appendix, every
point in coF(t, x} can be obtained as a convex combination of at most n + 1
elements:
66	3 Control Systems
cdF(t, x) = <	(#□,... ,0n) C An, Mi € U for all i > .
I i=0	J
(3-71)
Here
An = J 6» - (0O,...,0„); ^0г = 1, 0, > 0 for all г I (3.72)
I	i=0	J
is the standard simplex in IRn+1.
Motivated by (3.71), we define the compact set
U* = U x ... x U x 4 c ]R(n+1)w+(n+1)	(3.73)
and consider the control system (3.70) with
n
Г(?,х,иГ) = fft.x, (uo,...,un,(0o, ...^n))) = 52^/(6,x,Ui).	(3.74)
i=0
Generalized controls of the form u$ = (uq, ... ,un.O) taking values in
are called chattering controls. In practical applications, they can be ap-
proximated by rapidly switching the control value u(t) among the values
uo(t),..., un(t). Here the length of time during which и = щ should be pro-
portional to Gift). The above construction provides a representation of the
closure R(t) of the reachable set for (3.5) as the reachable set R^(t) for the
’’chattering” system
x = f$(t,x,u?y г?(£) e ий, a?(0) = x.	(3.75)
Theorem 3.9.1. Let f.U satisfy the basic assumption (H). Assume that the
graphs of all solutions to (3.5) on [0,T] are contained in some compact set
ГУ с P. Then, for every t С [0, T], the closure R(t,x) of the reachable set
for the system (3.5) coincides with the reachable set R$(t,x) for the system
(3.75).
Proof. 1. From definitions (3.73), (3.74), by Caratheodory’s theorem it follows
{f$(t,x,afly, if € Ua} = Рй(£,гг) = cdF(t, x) = co {f(t,x,u); и € U} .
(3.76)
2.	By Corollary A.8.2, the sets F$(t,x) are compact, convex. Therefore, The-
orem 3.3.1 yields the closure of the set of trajectories for the differential in-
clusion x e F$(t, x). In particular, the reachable set R^(t, x) is closed.
3.	By (3.76), we can apply Theorem 3.4.2 and deduce that the set of solutions
of x 6 F(t,x) is dense on the set of solutions of x e 0(t,x). In particular,
the closure R(t, x) of the reachable set for the system (3.5) contains R$(t,x).
Together with the previous step, this yields the equality R(t,x) = R$(t,x).
3.10 The Bang-Bang theorem
67
3.10	The Bang-Bang theorem
If the sets of velocities F(t,x) = {f(t,x,u); и G U} are not convex, then
we have seen that the reachable sets may not closed. A noteworthy exception
occurs in the case of systems with linear dynamics:
x = A(t)x(t) + h(t,u(t)), u(t) e U, x(0)=±.	(3.77)
Indeed, in this case the application of Lyapunov’s theorem implies that every
point reachable with a chattering control can also be reached by an admissible
control t »—> u(t) € U of the original system.
Th	eorem 3.10.1. (Reachable sets for linear systems). Consider the sys-
tem (3.77). Assume that U C lRm is compact, A(t) is an n x n matrix de-
pending continuously on t, and h : [0, T] x U i—> IRn is continuous. Then for
every r € [0,T], the reachable set R(r,x) is a compact, convex subset ofJRn.
Proof. 1. By the continuity of A and h and the compactness of the set U,
as t e [0,t], all trajectories of the system remain uniformly bounded. In
particular, the closure of the reachable set R(r,x) is compact.
2.	We claim that the reachable set R(r, x) for the system (3.77) coincides with
the reachable set R\r, x) for the chattering system
x = A(t) x(t) -T У2 0t(^)^(L w^W),	a?(0)=x,	(3.78)
2=1
with u^z\t) G U, 0(t) G An for every t G [0, т].
Indeed, fix any point £ G R^(r,x). Let Af(-,-) be the matrix fundamental
solution for the linear homogeneous system
v = A(t) v.
If £ = for some solution of the chattering system (3.78), we have
J1
£ = М(т, 0)£ + / У^М(т, s)h(s, u^(sf)ds.
2=1
for some control functions : [0,r] •—> U and coefficients в = (во,... ,вп) G
An.
We now use the result 14. in the Appendix A.5 (Lyapunov Theorem), with
f^(s) = M(r, s)hfs, «W(s)). This provides the existence of nd-1 disjoint sets
Jo,... ,Jn C [0, r] such that Jo U • • • U Jn = [0, t] and
[ y^ei(s)M(r,s)h(s,u(z\s))ds = I M(r,s)h(s,u^\s))ds. (3.79)
2=0	2=1
68
3 Control Systems
If ?/* : [0, t] i—► U is the control defined by = u^(t) for t G Ji, from
(3.79) it follows
£ = M(r,tyx + /
Jo
Af(r, s)/t(s, u*(s)) ds.
Hence £ is reached at time r by the trajectory of the original system (3.77)
corresponding to the admissible control u*(-). This proves that 7?#(т, £) C
Л(т, x). Since the converse inclusion is obvious, the two sets coincide.
3.	By the previous step, it now suffices to show that the reachable set 1$(т, x)
for the chattering system (3.78) is compact and convex. Because of Theorem
3.9.1, the boundedness and the closure of Яй(т, x) are clear. To prove its
convexity, observe that t x(t) is a trajectory of the chattering system
(3.78) if and only if
x(t) G co < A(t)x(t) + h(t, и); и G U > for a.e.t.
Equivalently,
x(t) — A(t) x(t) G G(t) = co{h(t, cu); w € U} .
If now .Ti(-) and ^2(*) are two trajectories of the chattering system, for
any Л G [0.1] their convex combination x(t) = Aa?i(f) + (1 — A)j?2(£) provides
yet another trajectory. Indeed
±(t) - A(f)x(t) = A(iq(t) - A(/)rr1(t)) + (1 - A)(i2(t) - A(^2«) G G(t)
because each set G(t) is convex. In particular, x(r) = Aa?i(r) T (1 — A)a?2(r) G
Я#(т, ±), proving the convexity of the reachable set for the chattering system.
As a special case, consider a linear system where the admissible controls
take values in a convex polytope with vertices uq,... ,aqv £ Rrn 
x = A(t) x + B(t) и	u(t) G U# = co{oq, ... , cj/v} .	(3.80)
In addition, consider the system
x = A(t) x + B(t) и u(i) G U = {uq, ... , oqv} .	(3.81)
where the controls are allowed to take values only on the vertices of the poly-
tope. In this case, the admissible control functions u(-) are called bang-bang
controls. Indeed, they must be piecewise constant, jumping between the ex-
treme points of LP.
By the previous result, the reachable sets for the two systems are exactly
the same.
3.10 The Bang-Bang theorem
69
Corollary 3.10.2. (The bang-bang theorem). Assume that the nxn ma-
trix Aft) and the n x m matrix В ft) depend continuously on time. Then, for
every initial point ж(0) = x and any т > 0, the reachable sets R^(r,x) and
R(r,x) for the systems (3.80) and (3.81) are compact, convex and coincide.
Indeed, in this case the function h(t, u) = Bft)u is linear w.r.t. u. Hence, for
any t, x, the convex hull of the set of velocities Fft, x) = {A(t)x F В ft)aii; i =
1,..., TV} for the system (3.81) is precisely the set of velocities
F^(t, x) = {л(£)я + B(t)cd ; ш G co{o?i,...,}
for the system (3.80).
Problems
3.1.	A forced linear pendulum is described by the system
x + x = и uft) e IR ,
(3.82)
Write (3.82) as a first order system, taking Xi = x, x% = x. Show that
this system is completely controllable.
Next, assume that the admissible controls are only those taking values
inside the set U = [—1,1]. Show that the reachable set at time т > 0,
starting from the origin, satisfies
7?(t) C {(^i, ж2); x2 + x% < r2} .
Hint: apply Theorem 3.5.2 with ф(х) = y/x2 + x^.
3.2.	As in Chapter 1, consider the system
f Xi = и cos 3 ,
< X2 = U sin $,
3 -au.
(3.83)
modelling the steering of a car in a parking lot. Assume that the two
components of the control satisfy the constraints
uft) € [-1,1], a(t) e [-1,1].
and let an initial position (х1(0),а?2(0), #(0)) = (#i,#2,0) be given.
Show that the system is locally controllable at the point (zi,^#)- Con-
struct explicitly some control functions t uft) and 11—► aft) which steer
the system from (^1,	3) to the origin (0,0,0) at some later time т > 0.
By a suitable translation and rotation of coordinates, show that the car
can be steered from any initial position to any other terminal position.
70
3 Control Systems
3.3.	Prove that in Theorem 3.2.3 the inequality (3.13) can be replaced by the
weaker assumption
<^(Z,r) >
sup
|x |=r, u?eu
/(t, x, cu)
3.4.	Consider the control system
fii = (x? + l)u	p1(0) = 0	|,.^|<1яр
(±2 = (1+^1) 1	|x2(0) = 0	I ( )l -
Show that the reachable set is compact for t < тг/2, but neither closed
nor bounded when t > тг/2.
Hint: when t > тг/2, choose (3 > 0 such that arctan/? > % — t. Compute
the infimum
inf{ar2; (/3,x2) € R(t)}
and show that it can not be attained by any admissible trajectory.
3.5.	Consider the linear control system
(ii, ±2) = (^2, u), u(t) G [-1,1].
Fix any constant A G [—1,1] and a time r > 0. Call xx G IR2 the point
reached at time r starting from the origin, using the constant control
u(t) = A. According to the bang-bang theorem, the point xx can also be
reached using some control t u(t) G { — 1,1} taking values only in the
extreme points. Determine explicitly such control function.
3.6.	On IR2 consider the system
(±1, x2) =	? u2 - a?2), (rri, Я2ХО) = (o, 0), u(t) e U = [-1,1].
(3.84)
On a fixed interval [0,7], consider the sequence of rapidly switching con-
trols as in (3.27). Prove that, as v —> 00, the corresponding trajectories
satisfy
Xi(t,uy) 0 ,	x2(t, Uy) t
uniformly for t G [0,7*]. However, show that the function t 1-* (0, t) is not
a solution of the system (3.84). In addition, prove that, for every r > 0,
the reachable set R(r) C IR2 starting from the origin is not closed.
3.7.	Prove a converse to Theorem 3.3.1, namely: if the set of trajectories of the
differential inclusion (3.4) is closed, then all the sets F(t,x) are convex.
3.8.	Consider the control system
(±1, ±2? £3) = (^1 T x2 + и, sin x% + u2 , x\ + sin x2 + cos x% — 1),
3.10 The Bang-Bang theorem
71
with x(0) = 0 e IR3 and |u| < 1. Prove that it is locally controllable at
the origin.
3.9.	Consider the control system on IRn
x — u, t(0)=0,	u(t) G U = {o>i,...,cu/v} .
Describe the reachable set 7?(t) at any time r > 0. For a given point
у 6 R(t), construct a control function и : [0,т] U that steers the
system to the point у and is discontinuous at no more than n times, say
0 < ti < t? < ... < tn < r.
3.10.	Consider the linear system on IRn
x = Ax + bu, rr(O) = 0, u(t) € U = [0,1].
where b is a fixed vector in IRn. Given a continuous control function u(t) :
[0, T] i—> [0,1], construct a sequence of bang-bang controls : [0, T] i—*
{0,1} as follows. Divide [0, T] into equal subintervals Ik =]tk , tfc+i],
tk = к/v, к = 0,1,... On each Д, define

1 if tk < t < tk +
0 if tk +	< t < tk-^-i
Prove that, as p —* oo, the corresponding trajectories #(-, converge to
x(-,u) uniformly on [0,Т].
Hint: consider first the case where T = 1 and the function tt(-) is constant.
3.11.	Let f : IRn i—► ]Rn be a bounded, smooth vector field, with n > 2. Show
that the system
x = /(ж) и u(t) e [—1,1]
is not locally controllable in the neighborhood of any point x G IRn.
3.12.	Consider the control system
(±1,^2) =
u(t)e[-i,l].
Show that this system does not satisfy the assumptions of Theorem 3.7.1
at the point x = (0,0). Nevertheless, by a direct computation prove that
the system is small-time locally controllable at the origin.
Hint: explicitly compute the trajectories corresponding to the piecewise
constant controls
«(*) =
1 if t e [0, A[
-1 if te [A,r]
or
u(t) =
-1 if t € [0,A[
1 if t 6 [A, t]
72	3 Control Systems
3.13.	Consider the control system
щ X2 = (a/2 — t)(l + (f — л/2)Х|)х2,
with initial data (xi, #2)(0) = (0,0). Show that for every control function
и : [0, \/2] —> U = {-1,4-1} the corresponding trajectory is defined and
does not explode. What happens if we take co(t7) as control set?
3.14.	Consider the linear control system (3.51), assuming that x — 0 and that
the set U C IRW is compact. Fix a row vector p e IRn and call t i—> p(£)
the solution to the linear adjoint problem
p(*) =-Р(0л(*) >	Р(т)=р.
Show that every trajectory of (3.51) starting from the origin satisfies
u) < / ( maxp(s) • B(s)lu ) ds.
Jo	J
Viceversa, prove that there exists a measurable control function 11—> ?i*(t)
such that
p - x(r, u*) = / ( maxp(s) • B(s)tu ) ds.
Jo \^и	/
Conclude that, for every row-vector p E IRn, the reachable set at time r
satisfies	r
max p • у = I maxp(s) • B(s}w ) ds .
ytR(r)	Jo	/
Letting p range over the unit sphere in IRn, this provides a characterization
of the reachable sets for linear control systems.
3.15.	For the equation (3.82) of the forced linear pendulum, assume that the
control function satisfies the constraint u(t) e [—1,1]. Let R(t) C IR2 be
the reachable set starting from the origin. For 0 < t < t', prove that R(t)
is a neighborhood of the origin and R(t) G R(t'). Moreover, show that
Ut>o/?(0 = Di2.
Hint: for any unit vector p 6 IR2, use the previous problem and estimate
max p • у.
3.16.	Consider the control system
±1 = щ
X2 =
X1(O) = -l
X2(O) = 1
|Uj(Z)| — 1 a-e-,
where f is given by
3.10 The Bang-Bang theorem 73
1
®1
/(*1)= 0
if x\ < 0,
/(Xi) = exp
if X\ > 0.
Describe the reachable set R(f) for t > 0. Is it possible to reach the point
(—1,0)? If yes find the corresponding control.
Hint: first reach the zone where Xi > 0, then it is possible to change the
second coordinate by suitably choosing the control.

4
Asymptotic stabilization
Consider a control system described by
x = f(x, u).
Assume that /(z, 0) = 0, so that x E IRn is an equilibrium point when the
null control и = 0 is applied. In general, this equilibrium may not be stable:
a trajectory which starts at a point xq « x may get very far from x at large
times. For many engineering applications, an important problem is to design
a feedback control U = U(x) such that the resulting system
x = /(x, u(x))
is asymptotically stable at the point x.
In the first section of this chapter we recall some basic stability results
in the theory of ordinary differential equation. The second section deals with
linear control systems. We prove here the main theorems on global stabiliza-
tion, by means of a linear feedback control. Using a linearization technique,
in Section 4.3 we prove a similar result on local feedback stabilization, valid
for nonlinear systems.
We conclude the chapter observing that, even if a control system is glob-
ally stabilizable, because of topological obstructions the stability may not be
achieved by any continuous feedback. In some cases, it is thus necessary to
construct discontinuous feedback controls. A special class of discontinuous
controls, in the form of patchy feedbacks, will be discussed in Chapter 9.
4.1 Lyapunov stability
Consider the differential equation
x = g(x)
хЕПп,
(4T)
76
4 Asymptotic stabilization
let t i—> x(t,xo) be the solution taking initial data ж(0) = xq. We say that
the system (4.1) is Lyapunov stable at the origin if the following holds (see
figure 4.1).
(LI) For every e > 0 there exists S > 0 such that if |tq| < 6 then for every
t > 0 we have |я?(£,я?о)| <
(L2) For every x^ e IRn we have lirnt_>+oo x(t, x0) = 0.
The first condition means that for every assigned ball, the entire solution will
remain inside this ball provided that the initial condition is sufficiently close
to the origin. Such condition is also called stability. The second condition says
that the origin attracts every trajectory of the system.
Fig. 4.1. Lyapunov stability.
For a general nonlinear system, checking its stability is not an easy task.
A standard method for proving Lyapunov stability is to construct a positive
function that decreases monotonically along all trajectories of the system. We
review here the basic ideas of this approach.
Given an open set 12 C lRn, a C1 function V : J? i—► IR , is called a Lya-
punov function for (4.1) on 12 if the following conditions hold.
i) V is proper, i.e. for every r > 0 the sub-level set {a; : V(z) < r} is compact;
ii) V is positive definite, i.e. V(0) =0 and V(x) >0 for every x 0;
iii) V is strictly decreasing along trajectories of the system: For x / 0 we
have W(a?) • g(x) < 0.
Theorem 4.1	.1. If the system (4-1) admits a Lyapunov function on IRn then
it is Lyapunov stable.
Proof. Let V be a Lyapunov function for (4.1).
1.	Let us first prove that for every s > 0 there exists 5 > 0 such that
{:r : V(.t) < £} C L?(0,6), where B(0,s) is the ball centered at the origin
with radius e. Indeed, assume by contradiction that there exist e > 0 and a
sequence x„ such that x„ £ B(0, e) and V(a:p) —> 0. Then by i) there exists a
4.1 Lyapunov stability
77
converging subsequence, that we indicate again by xv, and x B(0,s) such
that xu —► x. In particular x 0, but by continuity we get V(x) = 0, thus we
reach a contradiction.
2.	We now establish (LI). For any given e > 0, by 1. there exists 8'
such that {a; : V(x) < <5'} C B(0,s) and, by ii), there exists 8 such that
B(0, <5) C {# : V(x) < J'}. Take xq E B(0,5), then У(яо) < 8' and, by iii),
V\x(t,xo)) < 0, hence V(x(t,xof) < 8' for every t > 0. Finally x(t,xo) e
B(0,e) for every t > 0.
3.	Next, for every xq, we establish the limit V(rr(t,xo)) ~> 0 as t —* oo. By
iii) the function t V(#(£, rro)) is monotone decreasing. If V(x^t, j?o)) > c > 0
for all t > 0, define
A = max |w(x)-<z(ar); V(x) e [c, V(x0)]} •
Observe that A is the maximum of a strictly negative function on a compact
set, hence A < 0. We now have
^-V(x(t,x0)) - VV(i(t,x0)) -g(x(t,xoy) < A,
at
hence
V(z(£,a:o)) < ^(яо) + At —> —oo	as t —» oo .
reaching a contradiction.
4.	Finally we prove (L2). Fix x$ and e > 0. By i) there exists 6 such that
{я : У(ж) < J} C B(0,c) and by 3. V(rr(t,x0)) —► 0 as t —► oo. Therefore
there exists T such that for every t > T we have V(x(t,a?o)) < 8, hence
x(t,Xfy) e B(0,s). This completes the proof.
A complete characterization of Lyapunov stability can be given in the case
of a linear system with constant coefficients:
x = Ax x e IRn.	(4.2)
Theorem 4.1	.2. The system (4-%) is Lyapunov stable if and only if all the
eigenvalues of A have strictly negative real part.
Proof If v is an eigenvector of A whose corresponding eigenvalue A has non-
negative real part, then the function 11—> eXtv is a solution of (4.2) which does
not approach the origin as t —> oo.
To prove the converse, we consider an invertible matrix R such that В =
R~1AR is in canonical Jordan form, as in Example 2.1. If all eigenvalues of
A have strictly negative real parts, say JfA < — 8 < 0, and their multiplicity
is < A:, then
78	4 Asymptotic stabilization
||eM|| = ||ЛегВ7Г1|| < C • tk~xe~St 0
as t —> oc.	(4.3)
Here the constant C depends on the matrix R. By(4.3), every trajectory t i—>
x(fi) = etAx(0) of the linear system approaches the origin.
Using Theorem 4.1.1 we can provide a further characterization the set of
linear systems which are Lyapunov stable. In the following the symbol A* will
indicate the transpose of the matrix A. Similarly, the transpose of the column
vector x E Rn is the row-vector ж*.
Theorem 4.1	.3. For a linear system (4-2) the following conditions are equiv-
alent.
(i)	All the eigenvalues of A have strictly negative real part;
(ii)	There exists a symmetric, strictly positive definite matrix P such that
A*P + PA = -I;
(Hi) There exists a symmetric matrix P such that V(x) = x* Px is a Lyapunov
function.
Proof, (i) implies (ii). By hypothesis
||eM'eM|| <Ce~£t
for some constants C > 0 and s > 0, depending on the matrices R. В in (4.3).
We can thus define the matrix
I etA* etA dt
о
by an absolutely convergent integral. This matrix is positive definite, because
for any x 0
x^Px = / x*etA etAxdt = / |etj4rr|2 dt > 0 .
jo	jo
Moreover,
r+°o г	f+oo (ptA*tA\
A*P+PA=( ^A*etA‘etA + etA‘etAA^ dt — J	---^-dt = -I.
(ii)	implies (iii). Let P be as in (ii). Since the matrix P is positive definite,
the quadratic form V(x) = x*Px is proper, positive definite and it satisfies:
VV(a?) • Ax = x'PAx 4- x'A'Px = -x*x < 0.
(iii	) implies (i). This is a direct consequence of Theorems 4.1.1 and 4.1.2.
4.2 Stabilization of linear control systems
79
Example 4.1 Consider the linear differential equation (4.2) with n = 2
and
A =
-1 1
0-1
The matrix A has a unique eigenvalue A = — 1 of double multiplicity. We can
then apply Theorem 4.1.3. The equation A*P 4- PA = — I can be written as
2pn = ~l,	2p12 = 2p2i =Pu ,	2p22 = P12 + P21,
and the solution is found to be
1/2 1/4
1/4 3/4
The corresponding Lyapunov function is given by:
izz x 1	2	1	3 2
V(x) = -XX + -371^2 + ^2’
4.2 Stabilization of linear control systems
In this section we study the linear control systems
x = Ax F Bu.	ueIRw,	(4.4)
where A is an n x n matrix and В an n x m matrix. If all the eigenvalues of the
matrix A have negative real part, the system is already stable in connection
with the null control и = 0.
In the case where the uncontrolled system (4.2) is unstable, our aim is to
find a linear feedback U(x) = Fx, with F an m x n matrix, such that the
resulting linear system
± = (4 + BF)rr	(4.5)
is Lyapunov stable at the origin. In this case, the control U(x) = Fx is called
a stabilizing feedback. By Theorem 4.1.2, this will be the case if and only
if all eigenvalues of the matrix A 4- BF have negative negative real part. The
main result given below will show that, if the system (4.4) is controllable,
then one can always construct a stabilizing feedback. Toward a general result
on stabilization by linear feedbacks, some preliminary lemmas are needed.
The next result shows that the space IRn can be decomposed as a sum of a
subspace where the system is completely controllable, and another subspace
where the control has no effect.
Lemma 4.2.1. Let d — rank[B, AB. A2B,..., An~1B] be the rank of the con-
trollability matrix for the linear system (4-4)- Then, by a linear change of
variables, the control system can be urritten in the form:
80
4 Asymptotic stabilization
у = A\y + A2z + B\u
z = A3Z,
(4-6)
where (y, z) € IR6' x JR"~d.
Proof. Let Vi be the subspace of dimension d, generated by the columns of the
controllability matrix [B, AB, A2B,..., An~lB], and let {^i,..., a basis
of Vi. Choose n — d additional linearly independent vectors . ,vn so
that {vi,...,vn} forms a basis of IRn, and call V2 the space spanned by the
vectors	, vn. In terms of the basis {t?i,..., ?;n}, the system (4.4) takes
the desired form (4.6). Indeed, the space V\ is А-invariant, namely Ax e Vi
for every x € Ц. Moreover, Bu 6 Ц for every и G IRm, hence the component
of Bu along the subspace V2 is zero.
Lemma 4.2.2. (Pole shifting). If the control system (4-4) ™ controllable,
then for any given set of real numbers Ai,..., An 6 IR one can find an m x n
matrix F such that the square matrix A + BF has Ai,..., An as eigenvalues.
Proof. 1. Assume first that m = 1. In this case the matrix В reduces to a
column vector b. The controllability matrix [6, Ab,.... An-16] is thus an n x n
matrix which, by assumption, has rank n. We can thus use its columns as a
new basis of IRn. With respect to this basis, the system takes the form
x = Ax + bu =
/0 ••• 0 a0 \
1 • •• 0 cti
\0	1	an_i	/
(4-7)
Notice that here the numbers qq, ..., an_i are precisely the coefficients of
the characteristic polynomial of A :
n—1
det (А/ - A) = An - £2 aiXJ 
j=o
Indeed, by the Cayley-Hamilton theorem, the matrix A is a root of its char-
acteristic polynomial. The vector A(An“1b) can thus be written as a linear
combination of b, Ab,..., An~lb in the form
(n—1	\	n—1
^ajAj b^^ajA^b.
j=0	J	j=0
2. Next, consider the auxiliary linear system
x = Ax 4- bu =
/ 0 1	0 \
0 0 •••	1
\«o «1 • • • ^4-1 /
(4-8)
4.2 Stabilization of linear control systems
81
Notice that the characteristic polynomial of A coincides with the one of A
and A. Therefore, using the basis {b, Ab, ..., An~1b}, the control system
(4.8) can be put in the same canonical form (4.7). We conclude that there
exists a change of coordinates that transforms the original system (4.4) into
the form (4.8).
3.	Up to a change of coordinates, it thus suffices to prove the result for the
special system (4.8). Given the numbers Ai,..., An, compute the coefficients
/3j of the polynomial
n—1
(А-Л1)---(Л-Ап) = Лп-^/ЗйЛ
J=o
Defining the row-vector F = (/?q — qq, /?i — <*1, • • • , /3n-i - »n-i) we obtain
A + bF =
/ 0 1 ••• 0 \
0 0 •••	1
\0o 01 • • • 0n-l /
The characteristic polynomial of this matrix is computed as
n—1
det (А/ - (Я+ bF)) = An - 0jxj 
j=o
By (4.9), its eigenvalues are Ai,..., An, as required. This proves the theorem
in the case where m = 1, i.e. the matrix В has just one column.
4.	We now deal with the general case m > 1. As a first step, we construct a
basis of linearly independent vectors tq, г>2,..., vn 6 IRn such that ui = Buq
for some uq e IR™ and
v<+i = Avi 4- Вщ	(4.10)
for some controls щ, u%, ..., ttn-i 6 IR771. This can be done by induction. We
start by choosing uq such that Vi = Buq 0. Next, assume that Vi,..., Vk
have already been constructed and call W the subspace generated by the
vectors Vi,... ,Vk. If к < n, we claim that there exists a control Uk such that
Ufc+i = Avk + Buk £ W, so that the induction can continue.
In the opposite case, we would have Avk + Bu E W for every и E Rm. In
particular, this implies Avk € W and hence Bu G W for every u. By (4.10),
this in turn implies Avi E W for every i = l,...,fc. From the above, we
conclude that Aw E W for every w E W, and В и E W for every и E IRm.
Therefore, all columns of the controllability matrix [B, AB,..., An-1B] lie in
W. By assumption, these columns span the whole space IRn. If к < n we thus
obtain a contradiction.
82	4 Asymptotic stabilization
5.	Recalling (4.10), we now construct an m x n matrix F so that
Fvi = Ui, i = 1,..., n — 1.
The above equations can clearly be satisfied because the vectors Vi are linearly
independent. With this choice, we compute the n x n matrix
[V1, (Л + BF)v\,	, I.A + BFp-'n] = (V1,...,vn).	(4.11)
Notice that this is precisely the controllability matrix for the system with
scalar control
x = (A + BF)x + xiu и € IR .
Since the vectors Vi,... vn generate the entire space IRn, this system is com-
pletely controllable. By the previous steps dealing with the case m — 1, there
exists a row vector f such that the matrix A + BF + iq f has Ai, ... , Xn
as eigenvalues. Since = Bizq, the conclusion of the theorem is achieved by
taking F = F + uq f.
Combining the two previous lemmas we can now prove the main result on
feedback stabilization of linear systems.
Theorem 4.2.3. (Feedback stabilization of linear systems). Consider
the linear control system (4-4)> and let (4-6) be its decomposition, in terms
of a fully controllable component у and a non-controllable component z. If all
the eigenvalues of the matrix A3 have strictly negative real part, then there
exists a linear feedback U = Fx that renders the system Lyapunov stable at
the origin.
In particular, the conclusion holds when the system (4-4) is completely
controllable.
Proof. By construction, the linear system on IR'7
у = Aiy + BiU
is completely controllable. Using Lemma 4.2.2, there exists an m x d matrix
Fi such that all eigenvalues of Ai + ByFi have strictly negative real parts.
Implementing the feedback control U = F^y we obtain the linear system
= (A1 +nB1F1 42)	•	(4.12)
z J \ о As J \z J	7
Consider the eigenvalues of the matrix in (4.12). These are given by the eigen-
values of Aj + B}F[, which by construction have negative real part, together
with the eigenvalues of A3, which by assumption have negative real part.
Therefore the system (4.12) is Lyapunov stable.
We conclude the section with a useful lemma:
4.3 Stabilization of nonlinear systems
83
Lemma 4.2.4. (Hautus) If the system (4-4) ls completely controllable then
rank(AZ — A, B) = n for every A 6 IR. In particular rank(A, B) = n.
Proof If the conclusion does not hold, there would exists a row vector p 0
such that p • w = 0 for every column vector w in the (n + m) x n matrix
(AI — A, B). Then pA = Xp and pB = 0, hence pAkB = XkpB = 0 for
every integer к. In particular p • w = 0 for every column vector v in the
controllability matrix [B, AB, , An-1B]. In view of Corollary 3.6.3, this
yields a contradiction with the assumption of complete controllability.
4.3 Stabilization of nonlinear systems
We now consider a general nonlinear system:
x = f(x,u) u(P) eUc IRm.	(4.13)
The first result gives conditions for the existence of a stabilizing feedback on a
small neighborhood of the origin, and is obtained by a linearization technique.
Theorem 4.3	.1. Consider the control system (4-13), satisfying the standard
conditions (H) in Chapter 3. Assume that f is continuously differentiable and
that U G IRm is contains a neighborhood of the origin. If
(4)/(o,o) = o,
(b) Setting
4 = ^(0,0), В = ^(0,0),
OX	OU
the linearized system x = Ax + Bu with is completely controllable.
Then there exists a neighborhood of the origin 12 such that the system restricted
to (2 admits a continuous stabilizing feedback.
Proof. By Lemma 4.2.2, there exists a matrix F such that A = A + BF has
all eigenvalues with negative real part. Hence by Theorem 4.1.3 there exists a
symmetric matrix P such that A*P + PA = —I. In particular, the quadratic
form V (x) = x* Px is a Lyapunov function for the linear system
x = Ax + BFx.
We claim that V is also a Lyapunov function for the nonlinear system
x — f^x.Fx)	(4.14)
restricted to a small neighborhood of the origin. Indeed, by assumption we
can write
f(x, u) = Ax + Bu + B(x, u)
84
4 Asymptotic stabilization
with
Therefore, every trajectory of (4.14) sufficiently close to the origin satisfies
V(x(<)) = VK(i) • /(x, Fx) = 2x*P  {Ax + BFx + Л(х, Fx))
= —x*x + 2x*PR(x, Fx) < 0 .
By Theorem 4.1.1, this shows that the system is Lyapunov stable, restricted
to a neighborhood of the origin of the form f2 = {# G IRn ; V(x) < 5} .
For linear systems, our earlier results showed that, if the system can be sta-
bilized at the origin, then the stabilization can always be achieved by means
of a linear feedback, say U = Fx. A similar statement may not true, even
locally, in the case of nonlinear systems. Indeed, one can give examples of
nonlinear control systems which can be stabilized, but only using a discontin-
uous feedback.
In a global setting, it is easy to understand how topological obstructions
can prevent the existence of a continuous stabilizing feedback.
Example 4.2. As in Example 1.1 in the Introduction, consider the problem
of steering a boat on a river to a given point P along a shore. Let v(x) be the
velocity of the water at point x, and let и be the speed of the boat relative
to the water. We seek a feedback control U — ф(х) such that al trajectories
of the resulting O.D.E.
x = v(x) + ф(х)	(4-16)
converge to P as t -> oo. It is now easy to see that, if an island is present
in the middle of the river, there can be no continuous feedback ф performing
the desired task. Indeed, there must be a curve 7 separating points whose
trajectories pass to the left or to the right of the island, as in figure 4.2.
Fig. 4.2. Non-existence of a smooth feedback.
In the above example, the non-existence of a continuous feedback was
due to the topological properties of the set L? of admissible states for the
4.3 Stabilization of nonlinear systems
85
control system. One can also identify topological properties of the map f
which prevent the existence of a smooth stabilizing feedback.
Recall that, by the Hautus Lemma 4.2.4, for every controllable linear system
the map (x, u) h-> Ax + В и is onto. A similar necessary condition can be
formulated in the nonlinear case (for a proof see [80])
Theorem 4.3	.2. Consider the control system (4-13), assume U = IRm and
f : IRn x lRm —► lRn continuously differentiable. If there exists a smooth
stabilizing feedback in a neighborhood of the origin, then the map f is open at
0, i.e. for every e > 0 there exists 8 > 0 such that
{y\ |y| < <5} c {f(x,u); |x| + |u|<e}.	(4.17)
Example 4.3 (Brockett) Consider the control system on IR3
x = fi(x)ui + f2(x)u2	(4-18)
with
/i(xi,x2,x3) = (1,0, -x2),	f2(x1,x2,x3) = (0,1,Xj).
Computing the Lie bracket, one finds
= (0,0,2).
Therefore, the Lie algebra £ generated by /1 and /2 satisfies
£(x) D span|/i(x), f2(x), |/i,/2](^)} •
for all x e IR3. According to Theorem 3.8.2, every initial point can be steered
to the origin in finite time.
However, a smooth stabilizing feedback to the origin for system (4.18) does
not exits. Indeed, the map:
(x,U) f(x,u) = (?li, U2, U2Z1 — ^1^2) •
is not open at the origin. To verify this claim, consider the velocity v68 =
(0,0,5), 8 > 0, and assume that v6 = f(x,u) for some (x,u). Then, equating
the first two components, we get щ = u2 = 0. Hence 8 = u2X\ — щх2 = 0,
reaching a contradiction.
Problems
4.1.	Consider the linear differential equation x = Ax on IR2, with
A=(~1 °
0 -2/’
Compute a Lyapunov function for the system on ]R2.
86	4 Asymptotic stabilization
4.2.	Consider the linear differential equation x = Ax on IR3, with
/-1 2 0 \
A= 0-10	.
\ 0 0-3/
Is the system Lyapunov stable?
If yes, find a Lyapunov function for this system, checking that it satisfies
the three conditions (i) (iii).
4.3.	Write the linear control system
±i=2x2 + wi, x2 = -rr3,	±3 = z3 + ui..
in the standard form x = Ax 4- Bu. Find a linear feedback и — Fx which
stabilizes the system to the origin.
4.4.	Consider the linear control system x = Ax + Bu, with
/100\	/1\
A= 0 1 1 , B = I 0 I .
\101/	\° /
Prove that this system is completely controllable. Find a linear feedback
matrix F such that (A+BF)3 = 0. Compute the solutions for the resulting
system with feedback: x = Ax + BFx.
4.5.	Consider the linear control system (4.4), with n > 2, m — 1. Assume that
there exists a constant A G IR such that = A a\j for j = 1,..., n and
bn = A . Is the system controllable?
Hint: Use Hautus Lemma 4.2.4.
4.6.	Consider the equations of a forced pendulum:
x(t) + sinx(^) = u(t),	|u(£)| < 1.
Find a feedback control и = u(x,x) which locally stabilizes the system
asymptotically at the origin. Can the system be also locally stabilized by
a feedback at the (unstable) equilibrium point x = 7Г ?
4.7.	Consider the control system:
x = Ax 4- Bu, у = Cx, z = Dz 4- <p(x) 4- V;(71)?
where x, y, z, и € IRn, А, В, C, D n x n matrices, with В and C invertible
while det(D) = 0. Prove that there exists no feedback stabilizing this
system to the origin.
Hint: Apply Theorem 4.3.2.
5
Existence of Optimal Controls
In this chapter we begin the analysis of optimal control problems, focusing-
first on the existence of optimal solutions.
In Section 5.1 we consider problems in Mayer form, where the cost to
be minimized depends only on the terminal point of the trajectory. For gen-
eral nonlinear systems, following [42] the existence of optimal controls will
be proved under a key assumption. Namely, the sets of admissible velocities
should all be convex.
When this convexity assumption fails, one can still construct an optimal
solution, but within a class of “relaxed trajectories”, corresponding to chat-
tering controls. In the special case of linear control systems, one can push the
analysis a bit further, and show the existence of a genuine trajectory of the
original system which reaches exactly the same terminal point. As in [67], this
argument yields again the existence of optimal controls.
In Section 5.2 all results are extended to the more general Bolza problem,
where a running cost is also present.
5.1 Mayer problems
As usual, we consider the control system
x = /(£,#, гл), и € U,
(5-1)
where
L( = { u(-) measurable,
u(t) e U for all t} .
(5.2)
Given an initial state x, a set of admissible terminal conditions S C IR x
lRn, and a cost function ф : IR x lRn f—> IR, we consider the optimization
problem
min ф(Т, x(T,uY)	(5.3)
u€Z/,T>0
with initial and terminal constraints
88	5 Existence of Optimal Controls
z(0) = x, (Г, ж(Т)) e S.	(5.4)
When, as in (5.3), the performance criterion depends only on the terminal
time T and on the terminal point x(T) of the trajectory, we say that the
problem is in Mayer form.
Remark 5.1 The maximization problem
max ^(T, x(T, u))
иеи,т>о
is of course equivalent to (5.3), choosing ф = —ф.
Under suitable hypotheses, we shall prove the existence of an admissible
control u* whose corresponding trajectory #*(•) = j?(-,u*) satisfies the con-
straints (5.4) and yields the minimum in (5.3). The key assumption will be
the convexity of the sets
F(t,x) = {f(t,x,w)', ai e U},	(5.5)
which guarantees the closure of the set of trajectories of (5.1).
To begin with the simplest case, assume that the terminal time T > 0 is
fixed, and (5.3), (5.4) take the form
inf ф(х(Т, u)), я(0) = x, x(T) € S',
иЕЫ
where ф is continuous and S is a closed subset of IRn. If the assumptions of
Theorem 3.5.1 of Chapter 3 hold and the set of trajectories reaching the target
set S is nonempty, then an optimal control exists.
Indeed, by Theorem 3.5.1 of Chapter 3, the reachable set R(T) is compact.
Hence there exists a point #min in the nonempty compact set R(T) A S where
ф is minimal. Any control u* that steers the system to the point xmin, i.e. such
that a;(T, u*) = a\nin3 is clearly optimal.
A similar result holds for the more general problem (5.1)-(5.4). On the
control system (5.3) we make the following assumptions, somewhat stronger
than the hypotheses (H) at the outset of Section 3.1.
(H) The set U C IRm of control values is compact, f : [0, oo) x IRn x U »—►
IR™ is continuous in all variables, continuously differentiable w.r.t. x, and
satisfies the bound
|/(t, x, w)| < <7(1 -F |^|) for all t,x,u.	(5.6)
Theorem 5.1.1. (Existence of optimal controls). Let the assumptions
hold. Assume that the sets of velocities F(t,x) in (5.5) are convex, the
cost function ф is continuous, and the target set S is closed and contained in
some strip [0, T] x IRn. If some trajectory rr(-) satisfying the constraints (5.4)
exists, then the problem (5.1)-(5.4) has an optimal solution.
5.1 Mayer problems
89
Proof. 1. By assumption, there is at least one admissible trajectory reaching
the target set S. Therefore, we can construct a sequence of controls uy :
[О, Ty\ i—► U whose corresponding trajectories tp(-) starting at x satisfy
(Tu, xp(Tp)) ES, lim <£(TM, ^(Tp)) = inf ф(7\ x(T,u)).	(5.7)
U—>OQ	W(.)
Since S C [0,T] x IR/1, we have Ty < T for every v. We can now prolong
each function xy to the entire interval [0, T] by setting xy(P) = xy(Ty) for
t € [Ty,T].
2.	By the assumption (H), as in Corollary 3.2.4 all trajectories satisfy the
uniform bound
fxy(t, u)\ < (eCt — 1) + ec<|^| for all te[0,T].	(5.8)
Since f is uniformly bounded on bounded sets, the sequence xy(/) is uniformly
Lipschitz continuous. Using Ascoli’s compactness theorem, by possibly taking
a subsequence we can assume that Ty^>T* for some T* < T, and xy(•)—►£*(•)
uniformly on [0, Г*].
3.	Because of the convexity of the sets F(t,x), Corollary 3.3.2 in Section 3.3
implies that #*(•) is an admissible trajectory of (5.1), i.e. there exists a control
u* : [0,T*] i—> U such that
±*(i) = /(t, x*(t),	for a.e. t e [0, T*].
Clearly, rr*(O) = x. Since S is closed, the first relation in (5.7) implies
(T*, x*(T*)) = lim (T„ ^(TJ) 6 S.
v—>oo
4.	Finally, from (5.7) and the continuity of ф it follows
ф(Т*, x‘(T*)) = lim ф(Т„, xv(Ty)) = inf ф(Т, x{T,«)).
v—>oo	u,T
Therefore the control u*(-) is optimal.
Examples of approximate solutions are shown in figure 5.1. We remark
that the main difficulties faced in the above proof are:
•	The sequence of control functions up(-) is bounded, but may be highly
oscillatory, hence it may not converge in L1.
•	Similarly, the sequence of derivatives xy = f(t.xy,uy) is bounded but may
not converge in L1.
On the positive side, taking a subsequence one has:
90	5 Existence of Optimal Controls
•	The sequence of times Ty converges to some T,
•	The sequence of trajectories яД-) converges uniformly to some function
£*(•). whose derivative remains in the convex closure of the velocity sets, i.e.
x*(t) € coF(Z, x*(t)) for a.e. time t.
Hence, if all sets F(t, rr*(Z)) are convex, then there exists an optimal control
such that i*(i) = f(t, #*(£), w*(0) f°r a-e- [0, Т].
Fig. 5.1. Two successive approximations xu to the optimal trajectory x*.
Example 5.1 Consider a system of the form
£(£) = /о(^) +	Л(^) UiW , z(0) = x, e [-1,1], i = 1,..., m ,
i=l
where /0, /i,..., fm are bounded, continuously differentiable vector fields on
lRn. Then, given a time T > 0 and any continuous function ф : IRn IR, the
problem
min </>(x(T))
u()
has at least one optimal solution.
Remark 5.2 The above proof is a typical example of the Direct Method for
proving the existence of optimal solutions. The basic steps are:
(1)	Construct a minimizing sequence x^-).
(2)	Show that some subsequence converges to a function #*(•).
(3)	Prove that .?*(•) is an admissible trajectory and satisfies the appropriate
initial and terminal conditions.
(4)	Prove that ^*(-) attains the minimum value for the optimization problem.
5.1 Mayer problems
91
Several extensions of Theorem 5.1.1 are possible:
(i)	The assumption that S С [0, T] x lRn is used in order to ensure that the
time intervals [0, Tu] are uniformly bounded. The theorem still holds if S
is closed and ф(Т, x) —* oo as T—>oo.
(ii)	The assumption (5.6) on f is used to achieve the a-priori estimate (5.8).
Using Theorem 3.2.3 of Chapter 3, one can replace (5.6) by any other hy-
pothesis which guarantees that, as t e [0,T], all solutions of (5.1) starting
at x remain inside a fixed compact set.
(iii)	The continuity assumption of the cost function ф can be replaced by
lower semi-continuity:
</>(£, re) < liminf ф(1',x')	for all t,x.
t' —>t, x' —*x
The relevance of the various assumptions in Theorem 5.1.1 is illustrated
by the following examples.
Example 5.2. In connection with the system
(±i, ±2) = (u, ^i),	e { — 1, 1} ,
consider the optimization problem:
min x2(T),	z(0) = (0,0), x(T) e IR2 .
Here the terminal time T is fixed but there are no constraints on the final
state x(T). By analysis in Example 3.5, it is clear that for any given T > 0 this
problem has no solution. Using highly oscillatory controls, one can construct
a sequence of trajectories xI/(-) such that xu(T) -+ (0, 0). However, the second
component хъ^Т} cannot be zero, for any admissible trajectory. Therefore
this minimum cost cannot be attained. Notice that in this example the sets
of velocities F(x) = {f(x,w); ш = ±1} are not convex.
Example 5.3. Consider the optimal control problem in one space dimension
max x(T, u)
u(-)
for the control system with dynamics
x = их2, rr(O) = 1,
where u(t) E U = [—1, 1] and T > 1 is any fixed terminal time. For any
e g]0, 1] define the control
1 1\
1 - t ’ £ )
92	5 Existence of Optimal Controls
The corresponding trajectory is found to be
x£(t) = min
If T > 1, there is no uniform bound on this set of solutions. In particular,
x£(T) — e-1, thus there exists no optimal control. This shows the necessity
of the assumption (5.6), or soma alternative assumption providing a uniform
bound on the reachable sets.
Example 5.4. Consider the optimal control problem given by:
min ф(х(Т)), x = и, |u| < 1,	z(0) = 1,
iz(-)
where T is fixed, ф(х) = (x — l)sign(z) and we set sign(O) = 0. For every
T > 1 the infimum of the cost is —1, but is never exactly attained. Notice
that here the cost function ф is not continuous. In fact, it is not even lower
semicont inuous.
The next result is concerned with the case where the velocity sets F(f, x) in
(5.5) are not necessarily convex. In this case, one can construct the associated
chattering system
n
x = /tt(t,x,utt) =	, (uo, • • • ,un,0) - u9 e W = Un+1xAn
i-0
(5-9)
as in (3.72), (3.74) of Section 3.9. Applying Theorem 5.1.1, we can find an
optimal solution t : [0, T*] h-> IR77 to the relaxed problem
min ф(Т\ ж(Т, tt#)),	(5.10)
subject to the initial and terminal constraints (5.4). In general, ^opt(-) is not
a trajectory of the original system (5.1). However, if the control system has
the special form
x(t) + A(t) • x(t) + h(t, u{t)),	(5-11)
then there exists some trajectory #*(•) of the original system (5.1) which
reaches the same terminal point as ^pt. Clearly, this yields an optimal solution
to the optimization problem (5.1)-(5.4).
Theorem 5.1.2. (Existence of optimal controls for linear systems).
Consider the optimization problem (5.3)-(5.4) for the linear system (5.11).
Assume that the functions A. h, ф are continuous, the set U is compact and the
target set S С [0, T] x lRn is closed. If there exists at least one trajectory which
reaches the target set S, then the minimization problem admits an optimal
solution.
5.2 The problem of Bolza
93
Proof. 1. For the linear system (5.11), the corresponding chattering system
(5.9) can be written as
x = f*(t,x,u“) = A(t) -x + ^O^t)- h(t,Ui(ty), u* e 1/“ = U x • • • U x An .
i=0
(5-12)
The optimization problem (5.3)-(5.4) for the chattering system (5.12) satisfies
all the assumptions of Theorem 5.1.1. therefore it admits an optimal solution
4₽t: [0, T] ~ IR’*.
2. By Theorem 3.10.1 of Section 3.10, there exists a trajectory a:*(-) = ж(-,и*)
of the original system (5.11) such that t*(T) =	t(T). Since every trajectory
of (5.11) is also a trajectory of (5.12), we have
Ф(Т, x*(T)) = 0(T, z’pt(T)) = inf ф(Т, x(T,u*)) < inf ф(Т, x(T,u)),
ueu
(5.13)
proving that the trajectory #*(•) is optimal.
We remark that the values of the two infima in (5.13) coincide. Otherwise
stated, the introduction of chattering controls, changing the system (5.11)
into (5.12), does not improve the optimal performance of the system.
5.2 The problem of Bolza
This section is concerned with the minimization problem of Bolza:
mm
L(t, x(t), u(t)) dt + ф(Т, x(T, w))
(5-14)
for the control system (5.1) with initial and terminal constraints (5.4).
Apparently, the performance criterion (5.14) looks more general than (5.3),
because, in addition to the terminal cost ф(Т, ж(Т)), we here also have a
running cost L = L(t,x, u). However, the problem (5.13) can be easily recast
in Mayer form, introducing the additional state variable
#o(t) = / L(s, x(s), u(sf) ds.
Jo
(5.15)
The optimization problem (5.14), (5.1), (5.4) is equivalent to
min
{х0(Т,и) + ф(Т, x(T,u))},
(5.16)
for the system on IR71"1"1
94
5 Existence of Optimal Controls
( ±o = L(t,x, u)
[ x = f(t,x,u)
u(t) G U.
(5.17)
with initial and terminal constraints
жо(О) = 0, x(0) = x, (T, x(T)) G S.	(5.18)
The existence of optimal solutions for the problem (5.14), (5.1), (5.4) can thus
be obtained by studying the equivalent problem (5.16)-(5.18).
Theorem 5.2.1. (Existence of solutions for the Bolza problem). Let
the assumptions (H) on f, U hold. Assume that S С [0, T] x IRn is closed, L
is continuous and the set of trajectories reaching the target set S is nonempty.
If ф is continuous and the sets
F+(t,x) = {(yo,y) G IRn+1; уо > L(t,x,w)y = f(t,x,cS) for some cu G U}
(5.19)
are all convex, then the minimization problem (5.14) for the system (5.1) with
initial and terminal constraints (5.4) has an optimal solution.
Proof. 1. Because of (H), all admissible trajectories remain uniformly bounded
as t G [0, Т]. Since L is continuous and the set U is compact, there exists a
constant M such that
|L(t, x(t), cj)| < M
for all t G [0, Т]. w G U, and every admissible trajectory #(•).
2.	Consider the auxiliary minimization problem on lRn+1, in Mayer form:
inf |х0(Г, u0 ,u) + ф(Т, z(T,w))}	(5.20)
for the control system
f xo = fo(t,x,uo,u) =	+ (1 - u0(t))T(t, x(t), u(t)), z .
[ x = f(t,x,u)	' ’ '
Here the control functions satisfy
(u0(t), iz(t)) € [0,1] x U C Rm+1.	(5.22)
In addition, we impose the initial and terminal conditions
(x0,x)(0) = (0,x) e IRn+1,	(Т,ж(Т))ё5.	(5.23)
3.	We can now apply Theorem 5.1.1 to the problem (5.20)-(5.23), observing
that the sets
5.2 The problem of Bolza
95
F(t,xo,x)-{(fo,/)(t,x , Uq, u); uq E [0,1], и E U
= {(yo,y) e F+(t,x); y0 < м} C IRn+1
are all compact and convex. This yields an optimal control (uj, и*) : [0, T*] i—►
[0,1] x U for the auxiliary problem (5.20)-(5.23). Let #*(•) be the correspond-
ing trajectory.
4.	Since L(t,:r*(t), zz*(t)) < Af, we must have uq(£) = 0 for almost all t, other-
wise the control (0, u*) would achieve a strictly better performance. Therefore,
the control ?/*(•) is optimal also for the original problem (5.14), (5.1), (5.4).
In the case of a system with linear dynamics, the convexity assumption
on the sets of velocities can be relaxed, also in the case of Bolza problems.
Indeed, Theorem 5.1.2 can be extended also to case of a running cost.
Theorem 5.2.2. (Solutions for the Bolza problem with linear dynam-
ics). Consider the control system with linear dynamics (5.11), with initial and
terminal constraints (5.4)- Assume that the functions A,h are continuous, the
set U is compact and the target set S С [0, T] x lRn is closed. If there exists
at least one trajectory which reaches the target set S, and if the functions
ф,ао,Ьо are continuous, then the minimization problem
min < / [ao(^) • #(£) + ho(t, wW)]	u)) 1	(5.24)
U’T [Jo	J
admits an optimal solution.
Proof. Introduce a new scalar variable Яд, satisfying
±o(t) = a0(t) • x(t) + hv(t,u(ty), ^o(O) = 0.
The Bolza problem (5.24) can then be reformulated as a Mayer problem on
IRn+1, namely
mm {х0(Г) + ф(Т,х(ТУ)}.
This problem satisfies all the assumptions in Theorem 5.1.2, hence it admits
an optimal solution.
Problems
5.1.	Consider the optimal control problem (5.1), (5.3) with the more general
constraints
x(0) ек, (T, x(T)) e S, x(t) e P for all t e [о, T].
96	5 Existence of Optimal Controls
Here the initial point is not fixed, but can vary within a compact set
К C IRn. In addition, we require that the entire trajectory remains inside
a closed set J?.
Under the same assumptions as in Theorem 5.1.1, show that if there exists
at least one trajectory satisfying all constraints, then the minimization
problem admits an optimal solution.
5.2.	For the control system (5.1), starting from the origin, consider the problem
of reaching a point in a closed set К G ]Rn in minimum time. Write this
problem in the standard form (5.3), (5.4), for suitable S and ф. Let the
assumptions in (H) hold and assume that the sets of velocities F(f, x) in
(5.5) are convex. If some trajectory reaching a point of К from the origin
exists, prove that the minimum time problem admits an optimal solution.
5.3.	Let t •—> y(t) G lRn be any continuous function. Consider the optimal
tracking problem
fT
min / |?/(i) — x(t,u)\2 dt,
**(•) Jo
for the control system
m
i = /o(®) + 52/i(®)«i(t) Uj(t) e [-1,1],
г=1
with initial data jr(O) = x G JR71. We assume that Jo, • • •,fm are C1 vector
fields with sub-linear growth, so that \fi(x)| < C(1 + |.r|). Prove that this
problem admits an optimal solution.
5.4.	Consider the optimization problem
min / L(t, x(f), x(t}) dt
*(•) Jo
where the minimum is sought among all functions x : [0, T] »—> IRn such
that t(0) = 0, x(T) = £, and which are Lipschitz continuous with constant
«, i.e.
|ar(t) — .t(^)| < n\t — £'|	for all t, t' G [0, T].
Formulate this problem as an optimal control problem with fixed terminal
time. Assuming that the Lagrangean function L is continuous w.r.t. all
variables and convex w.r.t. x, prove that the problem admits an optimal
solution.
5.5.	Consider the car parking problem, modelled by the system (1.16). Given
an initial position (.T](0), x2(0). #(())) = (ah,#2,0), prove that there
exists a control function t ►—> (u(£), o(£)) steering the car to the ori-
gin in minimum time. Notice that in this case, the angle is defined up
5.2 The problem of Bolza
97
to multiples of 2тг. As target set we should thus take {(xi,#2>0) =
(0,0,2Ъг), к integer}.
5.6.	Recall the equations (1.9) describing the motion of a boat on a river. Given
any two points A = (a, — 1) and В = (b, 1) on opposite shores, show that
there exists a control u(-) steering the boat from A to В in minimum
time. Notice that for this control system we have the state constraint
o:2(t) 6 [—1,1] at every time t.
5.7.	Consider the linear control system:
x = Ax + Bu, x € IRn, и € IR™,
and the optimal control problem:
max ^(Т,х(ТУ).
и
Prove that if lim^^^ x) = +00 and the system is controllable then
there exists no optimal control. What happens if the system is not con-
trollable?
5.8.	A satellite moves around the earth on a fixed plane with dynamics, in
polar coordinates, given by:
r(t) = u(t), r(0) = 1,	0(t) = 1, 0(0) = 0,
with 0 < и < 1 the energy given to rockets. The satellite gets energy e by
solar panels, thus:
e(t) = /?(0(t)) — u(t), e(0) = 0, e(t) > 0,
where (3(0) = 1 if 0 € [0,7r] and vanishes otherwise, reflecting the fact
that for half orbit the earth sends shade over the satellite. Consider the
optimal control problem:
maxr(T) + e(T), r(T) > Ci, e(T) > C2.
и
1)	Verify that the set of admissible controls (which ensure all constraints
e > 0, r(T) > Ci and e(T) > C2) for the case T = 2тг, Ci = C2 = 0, is
given by
л	/'2tv
U := < и : [0, 2тг] —► [0,1] : / u(s) ds < тг
I	Jo
Hint: Notice that on [0, я] the increase of e is given by %, while it vanishes
on [7г, 2тг].
2)	Show that if T is allowed to be arbitrarily large, then there exists
98	5 Existence of Optimal Controls
no optimal control.
Hint: is there any a priori bound on r 7
3)	Assume T = 2А?тг. Show that there exists a solution if and only if
A;7t > Ci and fc?r + 1 > (7i + C2.
5.9.	Consider the control system:
±1 = их] + 1, X2 = tz, #i(0) = #2(0) = 0.
with и € U and the optimal control problem:
inax</)(T, 2?(T)),	G S,
и
where ф is continuous and S is a compact set.
1)	Prove that there exists no optimal control if S = {(—1,0)} and
U = [-1,1].
Hint: can 2:1 became negative?
2)	Prove that there exists an optimal control if ф = X2, S — {(2,2:2) •
0 < x2 < 100} and U = {0, +1}.
3)	Prove that there exists an optimal control for ф = Xi — T, S = {(2:1,0) :
2 < 24 < 100} and U = {—1,4-1}.
5.10.	Consider the optimal control problem:
2:1 = 2:2, %2 =	4" .(/(J>1) 4-u, #i(0) = 2:2(0) = 0,
max^(ar(T')),	^2(T} = — 24 (T)2,
и
where |u| < 1, g and ф are continuous.
Prove that if
lim <7(2:1 )/|2:iI = a > 1,
|ii |—>4-00
then there exists an optimal control (regardless of the growth of ф at
infinity).
6
Necessary conditions
In this chapter we introduce the Pontryagin Maximum Principle (PMP) and
show how it can be used in order to compute optimal controls and optimal
trajectories.
Section 6.1 deals with optimization problems where the payoff depends
only on the terminal point, which is not subject to any constraints. This is
a case where the key ideas can be more clearly described. To test the opti-
mality of a control w*(J, one has to consider various possible perturbations,
and check that none of these produces a better payoff. In particular, to derive
Pontryagin’s necessary conditions, one constructs a family of “needle varia-
tions” , changing the values of the control u* only in a neighborhood of a given
time t. Remarkably, in order to compute how all these different perturbations
affect the terminal payoff, it suffices to transport one single “adjoint vector”
backward along the trajectory. Imposing that the change in the payoff (up to
first order) is non-positive for all admissible variations, one obtains the famous
Pontryaging Maximum Principle [72].
In general, the PMP yields a (highly non-linear) system of O.D.E’s for the
optimal trajectory and for a corresponding adjoint vector, which must then be
solved with appropriate boundary conditions. In Section 6.2 we present various
examples where these equations can be explicitly solved, thus determining the
optimal controls and the optimal trajectories.
In Section 6.3 we discuss the PMP in the case where terminal constraints
are present. The proof is now considerably more difficult, ultimately relying
on the use of Brouwer’s fixed point theorem.
Sections 6.4 and 6.5 extend the earlier results to problems with variable
terminal time, and to minimization problems containing both a terminal cost
and a running cost. In particular, we show how the classical Euler-Lagrange
and the Weierstrass necessary conditions in the Calculus of Variations can be
derived from the PMP.
The special case of a linear system with quadratic cost, most important
for engineering applications, is discussed in Section 6.6.
100	6 Necessary conditions
For an alternative approach to necessary conditions for optimal control
problems, see [27], [28].
6.1 The Mayer problem with free terminal point
We consider here an optimal control problem in Mayer form
max^(2?(T, u))	(6.1)
u£U
subject to
±(t) = f(t, rr(t),u(t)),	ж(0)=ж.	(6.2)
For a given set U C IRm, the family of admissible control functions is defined
as
U — {и : [0, T] h-> U, и measurable} .	(6.3)
Notice that here the terminal time is fixed, but we do not put any constraints
on the terminal state x(T). Our main goal is to derive necessary conditions
in order that a trajectory £*(•) = x(-,u*) be optimal. We make the following-
assumptions.
(0) The set 1? C IR x IRn is open, the function f = f(t,x,u) is continuous
on I? x U and continuously differentiable w.r.t. x. The payoff function
ф : IR” i—► BFt is differentiable.
Notice that here we are not making any assumption on the set U of admis-
sible control values. In particular, we may well have U = IR™. Our main goal
is to derive necessary conditions for the optimality of a control £*(•). These
conditions will provide a basic tool for the actual computation of optimal
controls.
Theorem 6.1.1. (Pontryagin Maximum Principle, free terminal po-
int). Consider the optimal control problem (6.1)-(6.3), under the assumptions
(C>). Let u*(-) be a bounded admissible control whose corresponding trajectory
#*(•) = x(-,?i*) is optimal. Call p : [0,T] »—► IRn the solution of the adjoint
linear equation
p(i) = -p(t) • Dxf(t,x*(t),u\t)), p(T) = \7г1>(х*(ТУ).	(6.4)
Then the maximality condition
PW • f(t,x*(t),u*(ty) = max |p(t) • f(t,x*(t), w)}	(6.5)
holds for almost every time t € [0, Т].
6.1 The Mayer problem with free terminal point 101
In the above theorem, rr, /, v represent column vectors, Dxf is the n x n
Jacobian matrix of first order partial derivatives of f w.r.t. x, while p is a row
vector. In coordinates, the equations (6.4), (6.5) can be rewritten as
=	Pi(T) =-^(x’(T)),	(6.6)
U.l j
J=1
n	( n	1
52Pi(^) •	= max < Vpi(t) • fi(t,.	(6.7)
г=1	L i=l	)
Proof. To understand the derivation of the Pontryagin Maximum Principle
(PMP), it is best to consider first the case where the optimal control 1? :
[0, T] i—> U is continuous. In the last step of the proof we shall extend the
result to a general control u*, measurable but possibly discontinuous.
1.	Fix any time т > 0 and any admissible control value w € U. For e > 0
small, consider the perturbed control function
|u*(i) otherwise.
This is called a “needle variation” of u, due to the shape of its graph, as shown
in figure 6.1.
Fig. 6.1. A ’’needle variation” of the optimal control u*.
2.	Let t h-> x£(t) = x(tyue) be the corresponding trajectory of the control
system (6.2). Of course, xe(t) = x*(i) for t < r — e. At time t = т we have
= lim£_0+ JTT_e /(t, ar£(i), w)dt - | f^_£ f(t, x*(t). dt}
= /(r, x*(t), w) - /(r, x*(r),	.
(6-9)
102	6 Necessary conditions
As shown in figure 6.2, at time r the curve e i-> же(т) thus admits the tangent
vector
t?(r) = lim Xe^—	rr*(r), cj) — /(t, x*(t), tz*(t)) . (6.10)
e^o+ e
On the remaining time interval [т, T], all trajectories x£ are solution to the
same O.D.E., namely
x(t) = /(f,z(f),u*(f)).
According to Theorem 2.3.1, for every t € [т, T] the tangent vector
/ x . i. xeU) “ x*(t)
v(t) = lim
£->0+	£
is well defined and provides a solution to the linearized equation
v(t) = Dx/(t,a:*(t),?i*(t)) • v(t),
(6.И)
with initial data given by (6.9).
In particular, replacing u* with the controls u£ defined at (6.8), as e in-
creases the terminal point x(T, u£) of the trajectory is shifted in the direction
of the vector v(T).
Fig. 6.2. The perturbed trajectories xe, corresponding to the needle variations ue .
3.	By assumption, the control «*(•) is optimal, therefore
^(х*(Т))>Ж(Г))
for every e > 0. Differentiating w.r.t. s we obtain
0 > lim	= ^fe(r))
e—»0-|-	e	(is
E—04"
= W(x*(T))-v(T).
(6.12)
6.1 The Mayer problem with free terminal point
103
4.	Summarizing the previous arguments, we have shown the following: For
each т e]0,T] and every e U, let t	be the solution to the linear
Cauchy problem (6.11) with initial data
v(t) = /(Л £*(TX	^*(r), W*(T)) •
Then the terminal value v^T^(T) has non-positive inner product with the
gradient of the payoff function at the terminal point ж*(Т), namely
W(z*(T)) • v{r^(T) < 0.	(6.13)
Here we regard as a row vector and as a column vector.
5.	The family of inequalities (6.13), one for each (t,oj), can be rewritten in
a more convenient form. Instead of transporting each of the (infinitely many)
tangent vectors forward in time along the trajectory ж*(-), it is more
convenient to transport the single row-vector p(T) = V'0(^*(T)) backward in
time. Let t p(i) be the solution of the backward Cauchy problem
Ж = -p(f) • Dxf(t, ?(t)y(t)), p(T) = W(z*(T)).	(6.14)
Notice that (6.14) is the adjoint linear equation for (6.11). According to The-
orem 2.2.2, the inner product
p(t) •	te[r,T]
is constant in time. Therefore, by (6.13) we have
p(r) • г/т’ш)(г) = p(T) • v(T>u,)(T) < 0.
Recalling (6.10), for every т e]0, T] and cu e U we obtain
p(r) • (/(^ я*(т), w) - /(r, x*(r), u*(t))) < 0.	(6.15)
By continuity, it is clear that the above inequality also holds for r = 0. Since
w 6 U is arbitrary, (6.15) yields the maximality condition (6.5), for every
t e [о, T]
6.	To complete the proof of the PMP, we need to extend the previous argu-
ments to the case where the optimal control tz*(-) is measurable, bounded,
but possibly discontinuous.
By a theorem of Lebesgue, at almost every time т e [0, T] the bounded
measurable function t /(£, x*(t), u*(t)) is quasi-continuous, i.e.
lim; [	f(T,x*(T),u*(T))\dt = 0.	(6.16)
£->0+ e jT_e |	I
Fix any such time т and any control value u> e U. For e > 0 small, define the
control function u£(-) as in (6.8) and let же(-) = be the corresponding
trajectory. At time t = r one has
104	6 Necessary conditions
дг£(т) = т*(т) 4- I	- /(£,я*(£),гд*(£))] dt.
Letting e —> 0+, thanks to (6.16) we again recover the existence of a tangent
vector v(r), as in (6.10). The rest of the proof goes through without changes,
also in this more general case. Notice that now we obtain the validity of
the maximality condition (6.5) not at all times t e [0,T] but only almost
everywhere, i.e. at every Lebesgue point t of the map t»—►
Fig. 6.3. Geometric meaning of the optimality condition.
The geometric meaning of the maximality condition (6.5) is illustrated in
figure 6.3. Using the adjoint linear equation (6.14) we transport the row-vector
p(T) = V^(x*(T)) backward along the optimal trajectory. The maximality
condition means that, at each time r, among the possible attainable speeds
±(t) e F(r, x*(t)) = |/(т, ж*(т), cd), Cd€U|
we should choose the one that maximizes the inner product р(т) • ±(т).
Remark 6.1 In addition to the assumptions of Theorem 6.1.1. let U be
convex and assume that f is also differentiable w.r.t. u. Then the maximality
condition (6.5) implies
p(t) • Duf(t, x\t). u*(tY) (id - u*(t)) <0 Vid e U.	(6.17)
Otherwise, one would have u*(£) + e(cd — ti*(t)) 6 U and
p(t) • /(t, x*(t), u*(i) + e(w - w*(t))) > P(t)  /(t &*((), «*(*))
for e > 0 sufficiently small.
6.2 Computation of optimal controls
The Pontryagin Maximum Principle motivates a practical method for find-
ing optimal solutions to the problem (6.1) (6.3). We first define the function
6.2 Computation of optimal controls
105
и = и(1,х,р), from [0,T] x IRn x IRn into the control set U, in terms of the
maximality condition
p • f(t,x, u(t, x,p)) = max p •	(6.18)
In other words,
u(t, x,p) = argmax p • f(t, x, cu).	(6.19)
We then solve the system of 2n differential equations in the variables x,p:
(i = f(t,x,u(t,x,pY)	r
^p- -p. Dxf(t,x,u(t,x,p)),
with boundary conditions
z(0) = x, p(T) = W(z(T)).	(6.21)
By Theorem 6.1.1, if a bounded optimal control и exists, it must be found
among the solutions of (6.20)-(6.21). In general, this method encounters two
main difficulties:
(i) The map (i,x,p) h-> u(t,x,p), implicitly defined by the maximality condi-
tion (6.18), may be multivalued and/or discontinuous.
(ii)The equations (6.20)-(6.21) do not constitute a Cauchy problem on IR2n,
but a (usually harder) two point boundary value problem. Indeed, at time
t = 0 we are assigning the initial value д?(0), but the initial value p(0) is
not explicitly known. Instead, an implicit equation is given, involving the
terminal values p(T), x(T).
In particular cases, however, the equations for p and x can be uncoupled,
and a solution is more easily found.
Example 6.1	(Linear pendulum with external force). Let q be the
position of a linearized pendulum (see figure 6.4), controlled by an external
force u, with magnitude constraint
|u(t)| < 1, Vi.
For simplicity, let us assume that the initial position and velocity are both
zero, and that the motion is determined by the equations
q(«) + q(t) = u(t),	9(0) = 9(0) =0.
We wish to maximize the displacement q(T) at a fixed terminal time T.
Introducing the variables x\ = q, x% = q, the optimization problem can be
formulated as
max^i(T, u),
uea
where the dynamics is described by
106	6 Necessary conditions
{X\ = x2
X2 = -Xi + U
£1(0) = 0,
£2(0) = 0,
and the set of admissible controls is
U = {и : [0,T] »—> [— 1,1], и measurable } .
Fig. 6.4. Trajectories in the phase plane for the linear pendulum.
Notice that here we have
/(^w) =
a?2
—Xi + и
Dxf =
0 1
-1 0
In this case, the adjoint equations (6.6) take the form
Pi — P2
P2 = -Pi
Р1(П = 1-
p2(T) = o.
These equations can be solved for p independently of x, yielding
Pi(t) = cos(T — t), P2(T) = sin(T - t).
By (6.18), the optimal control u* satisfies
Pi£2 + p2(—£i + ti*) = max <! prx2 + p2(~£i + ^)
|u/|<l I
Therefore, the optimal control is
u* = sign(p2(T)) = sign(sin(T - <)).
(6.22)
6.2 Computation of optimal controls
107
Notice that trajectories corresponding to the constant controls и = 1 or
и = — 1 are circles centered at (1,0) or (—1,0), respectively. The case where
T = Зтг/2 is illustrated in figure 6.4. According to (6.22), the optimal control
is
«*(*)={;
if 0 < t < тг/2 ,
if тг/2 < t < Зтг/2.
Example 6.2	. To appreciate the effect of the map и = u(t,x,p) being mul-
tivalued, consider the problem
max а?з(Т, и),
и
for the system
(±i, i2, i3) = (a, -xi, x2 — я?),	(^1,^2,^з)(0) = (0,0,0)
with the control constraint |u| < 1 for all t e [0, Т]. The equations (6.6), (6.7)
here take the form
(Р1,Р2,Рз) = (P2 + 2zip3,p3,0),
(p1,p2,p3)(T) = (0,0,l),
' u(t)	=	1	if	pi	>	0,
< u(t)	=	—1	if	pi	<	0,
k u(t)	e	[—1,1]	if	pi	=	o.
Solving for p3,p2, we find
рз(*) = 1, p2(t)=T-t	Vt6[0,T].
In turn, the value of pi can be found from the equations
Pi = (-1 + 2u)p3 = -1 + 2sign(p!), P1(T) = 0, pi(0) = p2(0) = T, (6.23)
with the convention: sign(0) = [—1,1]. From (6.23) it follows, see figure 6.5.
' _3 (T
<	2 V 3
0
u*(t) =
- t)2 if t G
if t G
M)
.w,
-1 if 0 < t < T/3
1/2 if T/3<t<T.
P1W =
Observe that, on the interval [T/3, T], the value a* — 1/2 is derived not from
the maximality condition (6.18), but from the equation р^ = (—1 + 2u) = 0.
An optimal control with this property is called singular.
Example 6.3	. We now study a case where is not linear, hence the terminal
value p(T) = V^(x(T)) is not a priori known. Consider the problem
108	6 Necessary conditions
Fig. 6.5. Graph of pi.
max [xi(T, u) — x^T, u)] ,
for the system
(±ь±2) = (#2,ti),	(#i,#2)(0) = (0,0),	|u| < 1 Vie[0,T].
Pontryagin’s equations (6.6), (6.7) take here the form
(pi,p2) = (0,-P1),	(P1,P2)(T) = (l,-2x2(T>),
u(t) = sign(P2(t)).
We thus find
Pi = l, P2(t)=P2(T) + (T-i).
Observe that p2 is strictly decreasing function. Hence the corresponding con-
trol и = sign(p2) is either constantly = 1, or constantly = —1, or else it has
the form
( 1 if 0 < t < r	(a o .x
u^~ \ — 1 if r<t<T,	(6’24)
for some r G [0, T] at which p2 changes sign.
The optimality of the constant controls и = ±1 is easily ruled out. Indeed,
for u(t) = 1 we have a;2(Z) = P?(T) = — 2T < 0. For u(t) = —1, we
have xz(t) = —t, p2(T) = 2T > 0. In both cases, the maximality condition
is not satisfied. Now consider a control и satisfying (6.24), for some r. The
corresponding trajectory now is
Г2(/) = {2т-Ип'>т ’ Ж1(<) = / xAs}dS'
Since at the switching time r we have р2(т) = 0, we deduce
6.2 Computation of optimal controls
109
0 = р2(т) = p2(T) + T - т = -2ar2(T) + T - т = -2(2т - T) + T - r,
hence т = 3T/5. The optimal control is
zm _ f 1 if 0 < t < 3T/5,
[-lif 3T/5<f<T.
Example 6.4	. To see that the conditions of Theorem 6.1.1 are not sufficient
for optimality, consider the problem
max a:2(T, zz),
и
for the system described by
= (“j,	= СП > l«(t)l<i vtG[0,T].
y.r2 J \x\ )	\^2(0) )
(6.25)
The constant control zz*(t) = 0 yields the trajectory a;*(£) = 0. The corre-
sponding adjoint vector satisfying
(Pi, P2) = (-2^, 0) = (0,0), (pi,p2)(T) = (0,1)
is (pi, p2)(£) = (0Д), hence the maximality condition (6.7) holds for all t.
However, zz* is not optimal. In fact, any control и 7^ zz* yields
z2(T, u) =
dt
> rr2(T, zz*) = 0.
Remark 6.2. The above example shows that the Pontryagin Maximum prin-
ciple provides only a “first order” necessary condition for the optimality of a
control zz*(-)- Roughly speaking, this means the following. Assume that there
exists a family of perturbed controls u£(•) such that, calling xe(-) the corre-
sponding trajectories, one has
^е(Г))-Ж(П)
Um ---n-----n-----
£^o+	||zZe - 1Z*||L1
Then the control zz*(-), which is clearly not optimal, will not satisfy the PMP.
However, the non-optimality of zz* may not be detected by the PMP if
the increase in the payoff function ф is of higher order w.r.t. the control
perturbation, say
Hm ^(Г))-У>(х*(Т)) = Q
£-*0+	11^-71*11^
j = l,2,...,fc- 1,
110	6 Necessary conditions
^(Г))-У>(Ж*(Т))
e->0+	||?Ze—
> 0
for some к > 2. For a discussion of higher order necessary conditions for
optimality, we refer to [48], [58], [59].
Example 6.4 (continued). For the control system (6.25), consider the per-
turbed control functions
_ ( e if t E [0, T/2],
[ > |-£if te]T/2, Т].
Notice that — u*||Li = H^Hl1 — eT. The terminal points of the corre-
sponding trajectories are (xf(T), ^(T)) = (0, s3T3/12). Recalling that the
payoff function is ^(^1^2) = ^2, in this case we have
d3
= T3/2 > 0.
£=0+
This again shows that the constant control u*(f) = 0 is not optimal, but the
increase in the value function is very small, namely of third order w.r.t. to the
control perturbation. This cannot be detected by the PMP.
6.3 The Mayer problem with terminal constraints
This section is concerned with the optimization problem
max (#(T, ^))	(6.26)
u^U
for the control system described by
x = f(t, x, u),	#(0) = x, u(t)ev, £ e [0, T].	(6.27)
Here the terminal time T is fixed and the terminal point x(T) satisfies the
constraint
x(T) € S = {x e IRn ; 0Дж) = 0, г = 1,..., к}.	(6.28)
In addition to the assumptions (<0>), we now assume that all functions
0o, Фъ • • • j Фк ' Dln ► Dt are continuously differentiable.
6.3 The Mayer problem with terminal constraints 111
Theorem 6.3.1. (Pontryagin Maximum Principle with terminal con-
straints). Let u* be a bounded admissible control, whose corresponding tra-
jectory x* is optimal for the maximization problem (6.26)-(6.28). Assume
that the gradients V0o,... ,^фк are linearly independent at the terminal point
x*(T). Then there exists a nontrivial, absolutely continuous vector function
p(-) which satisfies the equations
p(t) = -p{tyDxf(t,x^(tfiu^t)Y	(6.29)
p(f) • f(t, x*(f), u*(t)) = max |p(t) • fft, z*(£), a;)}	(6.30)
at almost every time t G [0, T], together with the terminal conditions
к
р(Т) = £л^(х*(Т))	(6.31)
i=0
for some constants Ao,..., Xk, wiift Ao > 0.
Remark 6.3. If u* is optimal in connection with the minimization problem
min 0o (^(T, u))
for the system (6.27) with constraints (6.28), then all conclusions of PMP
continue to hold, with (6.30) replaced by
p(t) • f(t, x*(t),	= min (p(0 • f(t, x*(t), a/)} .	(6.32)
The equations (6.29)-(6.31) have a nice geometrical interpretation, which
motivates the subsequent proof (see figure 6.6). For every co € U and every
time r where ?/*(•) and is quasi-continuous, consider the one-parameter family
of control functions u£ defined at (6.8). By changing the control function from
u* to u£, the terminal point of the trajectory will be shifted from x*(T) to
x(T, u£). For e > 0 small, the direction of the shift is approximately given by
the vector:
V(T’W) = j-x(T,ue)
de
= v(T),
£=0
where v(-) denotes the solutions to the linear Cauchy problem
v(t) = Dxf(t, z*(t), u*(£)) • v(t), v(t) = /(т, ж*(т),о?) - /(t,x*(t),u*(t)) .
(6.33)
Otherwise stated, if Af(-, •) denotes the fundamental matrix solution for the
linear system:
v(t) = Dxf(t,x\t),u*(tf) • 'u(t),
then
112	6 Necessary conditions
^-x(T,u£)	=v(r’“) =	• [/(r,x*(r),w) - /(t,x*(t),u*(t))] .
d£	e=0
(6.34)
Define the convex cone Г as the positive span of all such vectors v^r,u;b
Г = span4-	0 < т < T, lj g U, zz*(-) is quasi-continuous at r } .
(6.35)
Next, let Ts+ — Ts+(x*(Ty) be the tangent cone to the set
S+ = 0o(^) > 0о(я*СП)>	— ° Vz = 1,...,
at the point rr*(T). More precisely,
Ts+ = {v; V0oCr*CT)) -v > 0, V0i(x*(T)) -v = 0 Vz = l,...,fc}.
Observe that Г represents a cone of ’’feasible directions”, i.e. a set of direc-
tions along which we could shift the terminal point ж(Т) by suitably changing
the control zz*. On the other hand, Ts+ represents the cone of ’’profitable di-
rections”, i.e. the set of directions along which we should shift the terminal
point, in order to raise the value of 0o(x(7'))^ keeping x(T} inside the target
set S, see Figure 6.6.
Fig. 6.6. The cone Г of feasible directions and the cone Ts+ of profitable directions.
The PMP amounts precisely to the assertion that the two cones Г and
7$+ are weakly separated. More precisely, the conditions (6.29)-(6.31) can be
restated in the form
p(T) • z; < 0 WgF,	(6.36)
p(T) • v > 0 Vz; G Ts+ .	(6.37)
6.3 The Mayer problem with terminal constraints 113
Indeed (6.36) holds if and only if p(T) •	< 0 for every admissible control
value ш G U and every time т at which u* is quasi-continuous, Recalling that
the product p(t) • v(t) remains constant in time, this is the case if and only if
p(r) • [/(г, £*(t), u>) — /(т,	и*(т))^ <0 Vt G [0,T], cj G U,
hence if and only if (6.30) holds. The equivalence of two conditions (6.31) and
(6.37) is proved in Lemma A.9.2 of Section A.9.
The proof of Theorem 6.3.1 relies on the possibility of “combining” con-
trol variations. More precisely, let To,... ,тдг € (0,T] be distinct times where
u*(-) is quasi-continuous. For any given uo, • • •	€ U and вц,..., Ок > 0,
consider the control function
„	if t e [Tj -eOj, Tj] for some j € {0,..., 2V},	,
( u*(t) otherwise.
Replacing the control u* by we will show that, as e increases, the terminal
point of the trajectory is shifted in the direction
N
= 52^v(T'’Wj)-	(6-39)
£=0+	j=0
Therefore, by suitably choosing the values of Tj, cjj, 6j, one can shift the ter-
minal point x(T) along any direction v of the cone Г. Assuming that this
cone of feasible directions is large, so that it cannot be separated from 7^+,
the theorem will be proved by contradiction, showing that u* is not optimal.
Proof. The Pontryagin Maximum Principle with terminal constraints will be
proved in several steps.
1. By Lemma A.9.3 in the Appendix A.8, the assumption that Г and T§+ are
not weakly separated implies that there exists finitely many control values
£ U and times tq,...,t/v where ?/*(•) is quasi-continuous, such
that, setting
v(Tj,Wi) =	- /(^^(тДиЧт;))]	(6.40)
as in (6.34), the cone
Г* = span+ |v(To’"1 * * o) * * * * * *,...; v(TN’“w)|	(6.41)
is not weakly separated from Ts+. Observe that, for any 3 > 0 and every
Lebesgue point т for u*(-), there exists another Lebesgue points т' т such
that
d	X
—ar(T,u£ie)
114
6 Necessary conditions
Therefore, again by Lemma A.9.3 of Section A.8, it is not restrictive to assume
that the times Tj in (6.40) are all distinct, say,
0 < T0 <
<rN <T.
2. Consider the parametrized family of control variations u£j defined at (6.38),
where £ € [0, f] and 0 ranges over the standard TV-dimensional simplex
{N	1
O = (0o,...,ON)-, 0,>О ^2^ = 11.	(6.42)
г=0	J
One eventual goal is to apply Lemma A.9.4 of Section A.8 to the map
X(s,0) = z(7>^).
(6.43)
3. With u£j as in (6.38), we claim that the limit
x(T,uej) - x(T,u*)
lim-------—------------
E—>0	£
j=0
(6.44)
holds uniformly w.r.t. 0 € A^. Indeed, for Z? = 0,..., TV define the controls
if t E [Tjfor some j e {0,.
e,e( (u*(£) otherwise.	' ’	'
By induction on Д we will show that the limit
,.	x(T,u*e)—x(T,u*)
lim --------------------=
e-»0+	£
£
^0jAf(T,Tj) [/(трХ*(тД^) - /(г^х’(т,),и*(т7))] (6.46)
j=0
is valid for all t € [т^,Т], uniformly w.r.t. 0 6 An. Indeed, assume that (6.46)
holds for some index L Then
«eV) -a:(^+1’ue,e)
fTe+i
J Tf+i ~E0f+l
[/(t,x(t,u^V),^+i) - /(t,x(t,i4 e),u*(t))] dt

+ [/(t,x’(t),^+l) - /(t,x‘(t),u*(t))J
+ [/(t,x*(t),u‘(i)) - f(t,x(t,uee e),u*(t))] | dt.
6.4 Variable terminal time
115
The assumption that is bounded allows us to use Theorem 2.3.2 of Section
2.3. Together with the inductive hypothesis, this yields
..	-x*(t€+i)
lim -------------------------
£->0+	£
Га;(т€+1,^ 0)-a?‘(r€+i)	- a:(r€+i, u* 0)
= Inn ---------------------------1----------’-------------------
£->o+	e	£
i
=	[/(7>>a:*('5),%) - f(rj,x*(rj'), u*(r,))]
j=0
+ lim I	[f(t,x*(t),CLie+i) - /(t,x*(t),u*(t))]dt
e^0+ Jre+1-e0e+l
+ lim I	- f(t,x*(t),we+i)] dt
+ lim /	[/(£, #*(£),	— f(t, x(t^ Д w*(t))] dt
£^0+ A£+1-£^+1
£
j=0
+&e+i [f(.t,x*(t),ue+i) -
€+1
=	- /(т,-,х*(т^),и*(т,))] .	(6.47)
j=o
Observing that for every t e [r^_|_i,T], it holds
..	-x*(t)	aj^+i,^1)-x*(r£+i)
lim -----------------= Af(t, * lim -------------------------------
e->0+	£	£->0+	£
from (6.47) it follows (6.46) with £ replaced by £ T 1. By induction, when
£ = N, t = T, we recover (6.44), proving our claim.
4. We can now apply Lemma A.9.4 in the Appendix A.8 to the map X intro-
duced at (6.43), with £ > 0 suitable small. Since Г* and T$+ are not weakly
separated, there exists (e, 0) such that
0o(X(f, 0)) > </>o(^*(T)), &(X(e, 0)) = Фг^{Т}\ Vz = 1,..., fc. (6.48)
Recalling (6.41), this shows that the control u~g steers the system to some
point in S, and achieves a value фо(х(Т, иё 0)) strictly better than u*. Hence,
tz* is not optimal. This contradiction proves the theorem.
6.4 Variable terminal time
This section is concerned with the optimization problem
116	6 Necessary conditions
max фо (T, x(T, u))	(6.49)
u&A
for the control system described by
x = f(t, x(t), u(tf), #(0) = £, u(t) E U a.e.,	(6.50)
where the terminal time and the terminal point are subject to the constraints
фг(Т,х(Т,и)) =0, г=	(6.51)
An optimal solution of (6.49)-(6.51) is now a pair (T*,rz*), where u* :
[0,T*] I—» U is measurable and the corresponding trajectory x*(-) yields the
maximum in (6.49) among all those which satisfy (6.51).
Theorem 6.4.1. (PMP, variable terminal time). Consider the opti-
mal control problem (6.49)-(6.51), under the usual assumptions (ф). Let
u* : [0,T*] i—> U a bounded optimal control for the problem, (6.49)
(6.51), and let #*(•) be the corresponding optimal trajectory. Assume that
f is continuously differentiable w.r.t. both t and x, and that the vectors
=	• • • > fa2-)’ i = 1,..., k, are linearly independent at the point
(T*, x*(T*)). Then there exists a nontrivial absolutely continuous row-vector
p(-) such that
p(t) = -p(t) • Dxf(t,x*(t\u*(tf),	(6.52)
p(t) • f(t, z*(t), zz*(t)) = max |p(i) • f(t, z*(£), c^)|	(6.53)
at almost every time t E [0,T*]. Moreover, there exist constants Xo,...,Xk
with Ao > 0 such that
(Pl, • • • .p„)(T*) = £ Л,	(T*, х*(Г)) / (0,..., 0), (6.54)
max р(Г*)./(Г’,ж*(Т*),и;) = -^А^(Т*,а;’(Г*)) .	(6.55)
wEu	*—'	(Jt
i=0
Finally, the function t i—► p(£)-±*(£) in (6.53) coincides a.e. with an absolutely
continuous function, satisfying
u*(t))} = P(0  Dtf(t, x*(t), u*(t)).	(6.56)
Proof. 1. We shall apply Theorem 6.3.1 to an auxiliary optimization problem
in n + 1 space variables with fixed terminal time T*. Set x = (xq, x) E IRn+1,
u = (и$,и) E IRm+1 and consider the problem
max </>o(x(T*, u)),
(6.57)
6.4 Variable terminal time
117
for the (n + 1)-dimensional system
f ±o(r) = u0(r),
[ i(r) = uo(t)/(xo(t), x(t), u(t)) ,
(ar0, z)(0) = (0,5)
subject to the constraints
</>i(x(T*)) = 0, i = 1,..., k.
(6.58)
(6.59)
(6.60)
(6.61)
uo(t) e |, 2 ,
u(r) e U for a.e. r G [0,7*].
The rationale behind these definitions is the following. The additional state
variable xq plays the role of the time t in (6.50). Indeed, (6.58) implies
dx	dx	dr
dx0	dr dx0	U
Moreover, the new time variable г is a reparametrization of the old time t.
In this way, we can allow t to range in a variable time interval [0, T], while т
always ranges on [0,T*].
2. Assume now that и* : [0, T*] U is optimal for the original problem
(6.49)-(6.51). Then u* = (l,u*) is optimal for (6.57)-(6.61). Indeed, let v =
(vo,v) be another admissible control, with
0o(x(T*,v)) > </>0(T*,a:(T*,w*)),	&(x(T*, v))=0 (1 < i < k).
(6.62)
Since Xot'r') = Vq(t) e [1/2, 2], one can invert the function r i—> ^o(^) = £(t)
and construct a control
u“(t) = v(r(t)).
Using this control in the original system (6.50), at the terminal time T =
xq(T*) by (6.62) one obtains
</>o(T, x(T,rz“)) ></>0(Г*, x(T*,u*)),	<к(Т,х(Т,и*)) = 0 (l<i<fc),
against the optimality of u*.
3. Applying Theorem 6.3.1 to the optimal control u* = (l,tz*) for the prob-
lem (6.57)-(6.61) and recalling that xq = t, we obtain the existence of an
absolutely continuous adjoint vector p = (po?p) with the properties
Po(<) = ~p(t)  Dtf(t,	,
(6.63)
118	6 Necessary conditions
PoW • 1 + p(t) • f(t,	=
max (po(£)u>o + p(i) • cvof(t,x*(t'),cu)\,	(6.64)
j<w0<2, ueu I-	J
k	/
(po(T*),.. . ,pn(T*)) - £ A, (^, gi,..., g J (T*,x*(T*)). (6.65)
i=0	4	'
for some constants Ло,..., Afc with Aq > 0. Since the maximum in (6.64) is
attained when cuq = 1 and cj = a*(i), we must have
PoW = ~P(C • № z*(f),	a.e.
(6.66)
Since po is absolutely continuous, from the first equation in (6.63) and
from (6.66) we deduce the identity (6.56).
The adjoint linear equation (6.52) follows from (6.63).
The maximality condition (6.53) is derived from (6.64).
The terminal condition (6.54) is a consequence (6.65).
Finally, (6.66) and (6.53) together imply
-po(0 = max p(Z) • /(t, x*(t), w)
Vte [0,T*].
Indeed, the two sides are continuous and, by (6.53), they coincide almost
everywhere. Together with (6.65), at t = T* this yields (6.55).
Remark 6.4. Defining the payoff function </>o(T, x) = — T in (6.49), one
obtains a minimum time problem. In this case, дфц/dxi = 0 for all i =
I,..., n. Therefore, defining the target set at time T as
5(T) = {x; фг(Т, x) = 0, i = 1,..., к} ,	(6.67)
the conditions (6.54) states that the adjoint vector p(T*) is perpendicular to
S(T*) at the terminal point x(T*).
Remark 6.5. In connection with the system (6.50), define the Hamiltonian
H(t, x,p, u) = p • /(t, x, u).
If u*(-) is an optimal control and x*(-),p(-) denote the corresponding trajec-
tory and adjoint variable, from (6.50) and (6.52) it follows that the corre-
sponding system of O.D.E’s has the Hamiltonian form
dx* 3H(t,x*(t),p(t), u*(Z))
dt	dp
dp	dH(ty x* (t), p(t), u* (t))
dt	dx
(6.68)
6.5 The problem of Bolza 119
6.5 The problem of Bolza
This section is concerned with the Bolza problem with running cost
min I L(t, x(t\ u(t))dt	(6.69)
uezv /л
subject to
x = f(t, x(t), u(t)),	д:(0) = ж, ?z(t)eU, (6.70)
with the terminal constraints
Фг(Т, x(T, it)) = 0, i = l, ...,fc.	(6-71)
Assuming that L is continuous in all variables and continuously differentiable
w.r.t. t,x, the above problem can be recast in Mayer form, introducing the
auxiliary variable
xn+i(t) = / L(s, a:(s),u(s)) ds
Jo
This yields a maximization problem with (n + 1)-dimensional state variable:
min жп+1(Т),
uEu
(6.72)
subject to the terminal constraints (6.71), for the system with dynamics
( Xi = fi(t, x(£), u(tY) i = 1,..., n
[±n+i = L(t,x(t),u(t))
e u
(6.73)
and initial conditions
(j?i,...,xn+i)(0) = (ж1,...,жп,0).	(6.74)
From Theorem 6.4.1 we deduce
Theorem 6.5	.1. (PMP, Bolza problem). Let f and L be continuous in
all variables and continuously differentiable w.r.t. t,x. Let the bounded con-
trol и* : [0, T*] U be optimal for the problem (6.69)-(6.71) and assume
that vectors	...,	(T*, rc*(T*)), i = 1,..., k, are linearly inde-
pendent. Then there exists a nontrivial adjoint vector p = (p1?... ,pn) and
constants Ло,..., Afc with Ao > 0 such that, for almost every t e [0, Г*],
UXi	uXi
(6.75)
120
6 Necessary conditions
p(0‘ /(^	u*(0) + АоД*, rr*(i), u*(t)) =
min [p(t)  fit, #*(<), ш) + A0L(t, x*(t), w)|,
(6.76)
d г . . _z + ..	. T.	*	z 4 df(t,x*,u*) x dL(t,x*,u*)
-jAp(t)-f(t,x ,u ) + XoL(t,x ,u ) !> = p(t)-——- + A0 —— ,
(6.77)
(Pi.... ,p„)(T*) = £ Аг ^(Т*,а?*(Т)),..., ^(Г ,Z(T))^ , (6.78)
min {p(T*)  /(Г*, s*(T*), w) + Aoi(T*, x’(T*), u>)}

(6.79)
Proof. Notice that min f L = — maxj(—L), thus the control u* solves also
a maximization problem for the cost with changed sign. We can then ap-
ply Theorem 6.4.1 to the corresponding problem (6.71), (6.73) and (6.74)
on JRn+1 with cost —jrn+i. In this case, we would have an adjoint vector
p = (рь ... ,pn+1) satisfying the evolution equation (6.52). However, since
neither f nor L depend on Tn+i, the evolution equation for pn-i-i is trivial:
4;Pn+i(t) = - ]Tpj(t)-^—(t,x*(t),u*(t))
Ol	‘	OX^^. i
~Рп+1(^)д( ~ (t,X*{t),U*{t)}=0.
ОХп^\
Hence pn+i(t) = pn+i(T*) = Ao for all t G [0, T*]. We then set pi = — рг for
г = 1,..., A: and pn+i = pn+i = Ao.
The identities (6.75) (6.79) now follow from the corresponding statements
in Theorem 6.4.1. For example, (6.76) follows from
p(t) • /(i, x*(i), u*(t)) 4- Ao (-b)(t,
= max {p(0 •	u) + Ao x*(t), w)
multiplying by —1 and recalling the definition of p.
Remark 6.6. For the control system (6.50), consider the problem of reaching
a (possibly moving) target set S in minimum time. If S(T) is described by
(6.67), this can be formulated either in the Mayer form (6.49)-(6.51), taking
</>o(T, ж) = T, or in the Bolza form, taking L(t,x, u) = 1.
Assume that the trajectory rr* = #(-,?/*) is optimal for the problem of
minimizing the time T subject to
6.5 The problem of Bolza
121
яг(Т, и) € S = {x; Фг(х) = 0, г = 1,	,	(6.80)
± = /(ж,и), ar(O) = x, u(t) e U.	(6.81)
Notice that here f does not explicitly depend on time. Using (6.75), (6.77)
and (6.78), in this special case where L = 1 we obtain
p(t) = -P(t)  Dxf(x*(t),
p(t) • /(#*(£), !£*(£)) = COriSt.
к
p(T*) =
i=l
Remark 6.7. If L(i, x, u) > 0 for every t, x, u, then the Bolza problem (6.69)-
(6.71) can be reformulated as a minimum time problem. Indeed, consider first
the autonomous case
min / L(a?(£),w(£)) df,	(6.82)
u Jo
subject to (6.80)-(6.81), with free terminal time T. By a simple rescaling of
time, (6.82) is transformed into the minimal time problem for the auxiliary
system
dx =	G и,	=	e s
dr L{x,u)
To handle the general case, just observe that any time-dependent problem
can be rewritten as an autonomous one, introducing the auxiliary variable
x„4-i = t, together with the equations £n+i = 1, жп+1(0) = 0.
As an application of Theorem 6.5.1, we shall derive the usual necessary
conditions for an extremum, for the standard problem in the Calculus of
Variations:
min / L(t, x(t), ±(t)) dt,	(6.83)
*(•) Jo
subject to
z(0)=z,	x(T)=y,	(6.84)
with x, у € lRn.
Theorem 6.5	.2. (Euler-Lagrange and Weierstrass necessary condi-
tions). Assume that L is continuously differentiable w.r.t. all variables t,x,x.
Let #*(•) be a Lipschitz continuous function which attains the minimum for
the problem (6.83) (6.84)- Then
(i)	The function t i—> ^(t, x*(t), £*(£)) coincides a.e. with an absolutely con-
tinuous function, such that
d \dL(+ * •*<
—	-T— lt.X ,X )
dt dx.
ox
(6.85)
122	6 Necessary conditions
(ii)	The function t ►—> L(t,x*(t), ±*(t)) — |4(t, x*(i), £*(£)) • x*(t) coincides
a.e. with an absolutely continuous function, such that
d r/ * V- dL ,	*
— L(t,X ,X )- > 7— (t,X ,X )• Xa
dt	dxi
L	i=l
(6.86)
(Hi) For almost every t G [0, T] and every ш G IRn, one has
L(t,x*(t),w) > L(t,x*(t),x*(t)) +	J*)’-(ш-х*(0)• (6.87)
Proof. The problem (6.83)-(6.84) is a special case of (6.69) (6.70), for the
system with simple dynamics
i(t) = u(t), u(t) G HV1
with fixed terminal time and terminal point. By Theorem 6.5.1 there exists
a nontrivial, absolutely continuous adjoint vector (pi,... ,pn) and a constant
Aq > 0 such that
dL
Pi(t) = -Xo — (t,x*(t),x*(t)),	(6.88)
A0L(t,a:*(i),i*(Z))+^2pi(t)i’‘(<) = min < A0L(t,ar*(Z),w) + ^p^Z)^ > .
i=i	weIR" I	«=i	)
(6.89)
If Aq = 0, since p is nontrivial and iv is arbitrary, the infimum in (6.89)
would be identically equal to — oo, and could never be attained as a minimum.
Therefore, multiplying Ao and all components of the adjoint vector p by the
same positive constant, we can assume Aq = 1. In order that w = ±*(f) yield
the global minimum in (6.89), we must have
d
dwi
L(i,ar*(t),w) + 57pi(t)
i=l
hence
Pi(t) = -f^(«,	±*(t))	(6.90)
dxi
at a.e. time t. Taking the derivative w.r.t. time and using (6.88) we obtain
(6.85). From (6.89), with Aq = 1, we also deduce
L(t,x*(t),u) > L(i, rr*(t), x*(t)) + p(f) • (£*(*) - Vcj G IRn. (6.91)
By (6.90), p = —dL/dx, hence (6.91) yields (6.87). Finally, since Aq = I,
(6.77) and (6.90) together imply (6.86).
6.5 The problem of Bolza 123
Fig. 6.7. In dimension n = 1, the graph of L must lie above the tangent line
at In this example, the condition (6.87) holds provided that the derivative
satisfies i*(t) < a or x*(t) > b.
Remark 6.8. The necessary condition (6.87) has a clear geometric meaning
(see figure 6.7). Namely, the graph of the function w >—> L(f, a?*(t), a;) lies
entirely above the tangent plane at the point u = x*(t).
Example 6.5 A landing vehicle separates from a spacecraft at time t = 0 at an
altitude h from the surface of the planet, with initial velocity vq. For simplicity,
consider vertical motion only and assume that the gravity acceleration g is
constant. Let Xi denote denote the altitude, X2 the velocity and let u(t) be
the thrust exerted by the rocket motor, subject to |tz(£) | < 1, with a suitable
rescaling. The equations of motions are
= (aj2, u — g),	(a;1,a:2)(0) = (Ji, v0)
For a soft landing at some time T we require
(x1,a;2)(T) = (0,0).
As performance index, we choose a linear combination of fuel consumption
and total time. This leads to the problem
min f (|iz(t) | + k) dt.
Jo
The Pontryagin’s equations (6.75)-(6.76) here take the form
(P1,P2) = (0, -Pl),
124
6 Necessary conditions
Pi^2 + P2(u* - g) + A0(|u*| + k) = min {Pix2 +р2(ш - g) 4- А0(|ш| + к)}.
kl<i
These in turn yield
Pi(0 = рг, p2(t)=p2 + (T-t)pi,
for some constants Pi,p2, while the control law is determined by
-1 if p2(t) > Ao,
0 if - Ao < p2(t) < Ao,
1 if p2(t) < -Ao.
Since p2 is a linear function of t, observing that u* = 1 for t sufficiently close
to T (to avoid crashing), by the minimality condition the optimal control must
have the form
u*(t) = < 0
if л < t < t2,
if и < t < T,
for some switching times 0 < Ti < т2 < T. The terminal condition (6.78)
here does not yield any additional information. Indeed, we have <^i(x) = #i,
ф2(х) = x2, and (6.78) states that the vector (pi,p2)(T) is a linear combi-
nation of V^i = (1,0) and V</>2 = (0,1). On the other hand, from (6.79) we
further deduce
Pi(T>2(T) +P2(T)(u*(T) - p) + A0(|u*(T)| + k) = p2(l - p) + A0(l + k) = 0.
From the relations
P2(n) = Ao,
P2(t2) = -Ao,
x 1 + к
P2\T) = -Ao j _
=
' — 1 if t < Ti,
1
and the fact that p2 is a linear function of t it now follows
ti + r2
2
1 - g _ t2 - л
1 + к ~	2
Using this additional condition on the switching times, a unique optimal con-
trol iz* is determined, satisfying the terminal conditions (xi,x2)(T) = (0,0)
at some time T.
Example 6.6. An enemy airplane flies at constant speed V at an altitude h
above the ground. Using a?i,a;2 as coordinates in a vertical plane, its position
at time T is given by
(rr1,ar2)(T) = (fc + VT, h).
(6.92)
A rocket is launched from the origin at time t — 0, with zero initial speed.
Its motor can produce an acceleration of magnitude A whose direction can be
6.6 Linear-quadratic optimal control
125
controlled. Calling x = (^1,^2) its position and v = (^1,^2) its velocity, the
equations of motion are:
(±i, X2, Vi, ^2)^) = (*h> ^2, A cos zi(i), Asin u(t) — g),
where g is the gravity acceleration and the angle и is the (unrestricted) control
function. One seeks the control law и = u*(t) that will intercept the airplane
in minimum possible time.
We are thus considering a minimum time problem, with terminal constraint
(xi,£C2)(^) = (H VT, h). Pontryagin’s equations here take the form
(Р1,Р2,91,9г) = (0,0, -pi,-p2),
</i(i)cos u*(t) + q2(f)sin u*(t) = min {<?i(f)cos w + (ftWsin w},
i.e., tan u*(f) — — 92(£)/9i(i). The terminal conditions (6.92) further imply
91(7') = 92(7) = 0, hence
91W =Pi(7-t),	92(t) =p2(T-t).
Since <7i, (72 are linear functions of /, the optimal control law thus has the form
u*(t) = arctan —
This shows that optimal controls must be constant.
6.6 Linear-quadratic optimal control
In this last section we apply the Pontryagin Maximum Principle to a spe-
cial class of optimal control problems, frequently encountered in engineering
applications, with linear dynamics and quadratic cost. Consider the system:
±(t) = A(f) x(t) + B(t) u(t), x e IRn, и e IRm, (6.93)
where A is a n x n matrix and В is a n x m matrix. The optimal control
problem, with fixed terminal time T, takes the form
• fT
mm /
Jq
[г? R(t)u + xr*Q(£)^] dt,
a:(0) = x,
(6.94)
where U is the class of all measurable controls, R is a symmetric m x m matrix,
Q is a symmetric n x n matrix, and t denotes the transpose.
We assume that each Q(t) is positive semi-definite and that the matrices
R(t) are uniformly positive definite, so that
xfQ(t)x > 0,
ulR(t)u > 0|tt|2
126	6 Necessary conditions
for some 0 > 0 and all x € IRn, и E Rm, t E [О, Т]. In particular, this implies
that the symmetric matrices R(t) are invertible.
Roughly speaking, we wish to keep the system close to the origin during
the whole interval [0, Т]. The integral in(6.94) penalizes the distance of x from
the origin, and the energy spent by controlling the system.
The problem (6.94) is in Bolza form, hence we shall apply Theorem 6.5.1.
The minimality condition (6.76) can be written as
p(t)- [4(t)x*(t) + B(t)tz*(0] +Ao[(M*(i))t^)M*(Z) + (a:*(Z))t(2(^*(i)] =
= min |p(t) - [x(t)x*(i) +	4- Ao|o?7?(£)w + (a;*(t))<Q(f)a;*(t)j
Notice that Aq 7^ 0. Otherwise, since there is no final constraint, (6.75) and
(6.78) would imply
р(«) = -p(t) • лад, p(T) = 0,
hence p(i) = 0, reaching a contradiction. Multiplying Aq and each component
of the adjoint vector p by the same positive constant, we can assume Aq = 1.
Since the dynamics is differentiable w.r.t. и and и can vary in the whole
space IR™, the minimization condition implies that the following quantity is
vanishing:
This, in turn, implies
p(t) • Z?(£) + 2(u*(t)//?(£) = 0.
Hence the optimal control satisfies
u‘(i) = -17?-1(^(i)pt(f).	(6.95)
The optimal trajectory for the linear-quadratic optimal control problem is
thus found by solving the two-point boundary value problem
x = A(t)x -	,
p = — pA(t) - 2xtQ(f),
with initial and terminal conditions
x(0) = x , p(T) = 0 •
The corresponding optimal control is then obtained by inserting the solution
p(-) in (6.95).
6.6 Linear-quadratic optimal control
127
Problems
6.1.	In Theorem 6.3.1, assume that the target is defined also by means of some
one-sided constraints, i.e.
S = {a; G IR71; ф^х) = 0 i = l,...,fc, </>j(rr) > 0 j =
Let ж*(-) = a;(-,?z*) be optimal. At the terminal point x*(T), assume that
^(x*(T)) = 0 > = 1,...,£,	ф^х\Т))>0 j =
and assume that the k + £ gradient vectors
V<^o(^(T)),..., V0fc(^*(T)), V4>i(z*(T)),..., V^*(T)),
are linearly independent. Prove the existence of a nontrivial adjoint vector
p(-) such that (6.29)-(6.30) hold, together with
к	£
p(T) = £ Aj;V^(x*(T)) + £ A'V^(X*(T))
i=0	j=0
with Ai,..., А^ G IR, Ao, A^, ..., A^ > 0.
6.2.	Show that the trajectory (xi(t), x2(£)) = (0,0) is optimal for the problem
max xi(T)
и
for the system
(±i,±2) = (tt,O), u(t) e [—1,1],
with initial and terminal constraints
(^i,^2)(0) = (0,0),	(^i,a?2)(T) G S = {ж G IR2; ф(х) > 0} ,
where ф^х^х?) = x% — x*.
However, there exists no adjoint vector p(-) which satisfies (6.29), (6.30)
together with
p(T) • V > o Vv e ts+
where Ts+ = {(3/1,372); У1 > 0} is the tangent cone at the origin to the
set
S+ = {(rri,x2) G S, хг > 0} .
Explain why this does not contradict the maximum principle proved in
the previous exercise.
128	6 Necessary conditions
6.3.	As in (1.9) consider the control system
(ii,i2) = (l-arl + ux, u2)
modelling the motion of a boat on a river. Here the point (^1,^2) is
constrained to the strip {(^1,^2)» ^2 £ [— 1, 1]} and the set U of admissible
controls consists of all measurable functions и : IR 1—> IR2 taking values
inside the closed disc
U — { (cJl, 1л12 ) * у CJ j	M } •
Assume that the initial position is (jti, ж2)(0) = (—1,0).
Write the Pontryagin necessary conditions, for a trajectory which reaches
a given point у = (1,6) on the opposite shore in minimum time.
Find the trajectory which reaches some point у — (1,^2) 011 the opposite
shore, in minimum time (Notice that in this second case the coordinate
X2 is free). Write the Pontryagin optimality conditions for this problem.
6.4.	A population of fish in a lake evolves according to the equation
x — x(a — x) — x и .	(6.96)
Here the control u(t) 6 [0,1] denotes the intensity of fishing activity.
An initial population x(0) = x and a terminal population x(T) = у are
assigned and one wants to maximize the payoff function
0(T) = I [x(t)u(t) — k?z2(£)] dt
Jo
i.e. the total amount of fish caught from the lake minus the cost of fishing.
We assume here that о. к > 0 and 0 < x, у < a. Write the PMP necessary
conditions for this problem.
6.5.	As in (1.14), consider the system
(±1,^2) = (^2/tt),	u(t) e [-1,1]
modelling the control of a cart on a straight rail. Given any point x =
(^1,^2), find the control which steers the system from the origin to x in
minimum time. Compute the minimum time function T = T(x). Show
that it is smooth on the entire plane IR2 except along the two curves
7+ = {(^i,.t2); xi = x%/2, x2>0},
7" = {(a?i,x2); Xi = -x^/2, x2 < 0}.
6.6 Linear-quadratic optimal control 129
6.6.	Consider the system

e [-1,1],
discussed in Example 6.4. Fix any point x = (Si,x2) € IR2 with x% > х^/З.
Write the Pontryagin equations satisfied by a control u(-) steering the
system from the origin to x in minimum time T. Show that there exists a
control reaching x, having one of the forms
u(t) =
+1 if t € [0, r],
-lif te]r,T],
u(t) =
(-1 if t e [o,r],
( +1 if t €]т, T],
(6.97)
satisfying the additional condition т > T/2. Show that such control is
unique if 0, while there are two such controls if x\ =0. These are
the only optimal ones.
Hint: if u(-) is a control of the form (6.97) but with т < T/2, show that
there exists a second control u(-) of the same type, which steers the system
to the same point x(T, u) = x(T,u) but at an earlier time T < T.
6.7.	Let A, В be an n x n and n x m matrix, respectively. Let z : [0, T] i—> IRn
be a smooth function, a > 0. Write the Pontryagin necessary conditions
for the optimal tracking problem
min [ \x(t) - z(t)|2 + a|u(t)|2 dt,
u Jo
in connection with the linear system on IRn
x = Ax 4- Bu, z(0) = 0, u(t) 6 IRm.
6.8.	A control ?/*(•) is called a strong [weak] local maximizer for the problem
(6.1)-(6.3) if there exists 6 > 0 such that i/?(a;(T,u*)) > ^(t(T, u)) for
every control и 6 U such that ||u — u*||li < d [respectively: for every
control и e U such that supte[0 Tj \u(t) — u*(£)| < J].
(i)	Assuming that u* is a strong local minimizer, prove that all state-
ments of Theorem 6.1.1 still hold.
(ii)	Assume that the set U C IRm of admissible controls is convex, and let
u* be a weak local minimizer. Prove that the statements in Theorem 6.1.1
still hold, with the maximality condition (6.5) replaced by (6.17).
6.9.	Recalling the definitions introduced in the previous problem, consider the
optimal control problem:
max x(T),
utu '
130	6 Necessary conditions
x = и3 — и2,	ж(0) = 0,
with Т > 0 and Ы = {и : [0, Т] —> [—2, 2] и measurable}.
(i)	Prove that u*(^) = 0 is a weak local maximizer.
(ii)	Show that (6.5) fails. In particular, u*(t) = 0 is not a strong local
maximizer, and it does not satisfy the Pontryagin necessary conditions.
6.10.	Assume that the function #*(•) provides a minimum for the standard
problem of the Calculus of Variations (6.83)-(6.84). Prove that, for almost
every t € [0, T],
n
QU) = £	x*(t), x\t))^ > 0	€ IRn,
iJ=0
i.e. the quadratic form Q is positive semi-definite (Legendre necessary
condition).
6.11.	Let £*(•) afford the minimum for the standard problem of the Calculus
of Variations (6.83)-(6.84). Assume that the derivative x* is piecewise
continuous, with a jump at some intermediate time t. Prove that the right
and left limits i(t+), x(t~) of the derivative satisfy the Erdmann corner
conditions:
OX	ox
L(t, x*(t), i*(f+)) — L(t, rr*(t), £*(£+))
i*(t+) — ±*(i_)
6.12.	A particle of mass m moves on a smooth horizontal plane with rectangu-
lar coordinates y. Initially the particle is at rest at the origin. During
a fixed time interval [0,T], the particle can be accelerated by applying a
force F(t) of constant magnitude |F| = к and arbitrary direction. The
angle 0(t) made by the force with the positive a?-axis is the (unrestricted)
control variable. At the terminal time T, we want the particle to be mov-
ing along a given line parallel to the ж-axis (say, the line {y = c}), with
maximum speed.
(i)	Find conditions on к, с, T which guarantee the existence of at least
one control $(•) satisfying the requirements.
(ii)	Show that the optimal control is given by 0*(t) = arctan(a T bt)
for suitable constants a, b.
6.13.	Suppose that a cup is initially filled for 4/5 of its capacity with hot coffee,
at temperature 0^. Cold milk is then poured inside at a rate u(t) e [0,1],
6.6 Linear-quadratic optimal control
131
until the cup is completely full. The coffee can be drank as soon as its
temperature decreases to a prescribed value 0± < 0q. We wish to minimize
this time.
Calling x(t) the amount of liquid in the cup and 0(t) its temperature, we
have the equation
x = u,
x(0) = 4/5,
0(0) = 0O •
Find the control function rz(-) which minimizes the time at which the state
(ж,0) = (l,0i) is reached.
6.14.	As a variant to the previous problem, suppose that the cup is initially
already full of coffee, at the temperature 0q. Milk is poured into the cup,
at a controlled rate u(t) 6 [0,1], so that the liquid overflows. The total
amount of milk is limited by
u(t) dt — ttiq.
In this case the total amount of liquid in the cup x(t) = 1 remains constant
in time. The thermodynamic equation describing the temperature of the
liquid in the cup is bilinear, namely
0 = — 3 — и — Ou .
Here the first term on the right hand side corresponds to the heat loss to
the surroundings, the second term is due to the inflow of cold milk, and
the third term is overflow. Determine the admissible control u(-) which
reduces the temperature of the cup to the value 0i in minimum time.
6.15.	Consider the linear quadratic problem
min / [jq(£) + x£(t) + cu2(t)] dt
Ц) Jo
for the system
±1 = X2	( #i(0) = Xi
±2 = и	[ £2(0) = ^2
Write the necessary conditions for the optimality of a control u*(-).

7
Sufficient Conditions
Consider the optimization problem in Bolza form
inf < I L(t, x(t, u),-u(t)) dt + (T, x(T,u))
U^u [Ло
(7-1)
(7.2)
(7-3)
for the system
x = f(t,xyu),
with initial and terminal constraints
u(t) E U a.e.,
x(t) = x,	(T,x(T)) e S.
It is well known that the PMP provides only a necessary condition for op-
timality: a control tt*(-) may satisfy Pontryagin’s conditions and yet not be
optimal. For example, it may provide a local minimum to the functional (7.1),
but not the global one. In this chapter we describe four techniques for proving
global optimality.
(I)	Prove that an optimal solution exists. If u*(-) is the only admissible control
that satisfies the PMP, then u* must be the optimal one.
(II)	Consider a Mayer problem with fixed terminal time. If the set of ad-
missible controls reaching the target S is convex, and if the functional
и ^(^(T, u)) is convex, then any control satisfying the PMP is in fact
optimal.
(Ill)	Embed the problem (7.1)-(7.3) in a family of problems, by varying the
initial data (f, £)• Compute trajectories satisfying Pontryagin Maximum
Principle. If such trajectories cover in suitably regular way the state space,
then all trajectories are optimal.
(IV)	Compute the optimal value function V = V(t, ж), for all initial data
(7,ж), by solving a related Hamilton-Jacobi partial differential equation.
Then, any control ?/*(•) which attains the optimal value V(t,x) is optimal.
134	7 Sufficient Conditions
The first three techniques will be covered in the following sections. The fourth
approach, based on the value function, will be discussed at the end of this
chapter in a special case, i.e. for linear-quadratic problems. A more general
treatment, based on the theory of viscosity solutions to Hamilton-Jacobi equa-
tions, will be given in Chapter 8.
As in Chapter 3, we shall always assume that the control system satisfies
the basic assumptions (II) below, except for the special Linear-Quadratic
problems treated in Section 7.5.
(H) The set U C IRm of control values is compact, is an open subset of
IR x П1п, the functions f : J? x U i-> IRn, L : L2 x U IR are continuous
in all variables and continuously differentiable w.r.t. x.
7.1 Existence + PMP.
Theorem 7.1.1. Let the hypotheses (H) hold. Assume that an optimal so-
lution for the problem (7.1)-(7.3) exists. Assume that the control functions
,n/c(-) are the only admissible ones that satisfy the PMP. Then,
among the controls ,Uk, the one which yields the lowest value of the
cost (7.1) is optimal.
Indeed, by Theorem 6.3.1, the optimal control u*(-) satisfies the Pontryagin
Maximum Principle. By the above assumptions, we must have u* = Uj for
some j e {1,..., k}, and the result is obvious.
We illustrate the theorem with a simple example.
Example 7.1 Consider the system
(®1,Ж2) = (u,Xj),	(xi,x2)(0) = (0,0),	|u(t)| < I te [0,2].
We seek an optimal control for the problem
max {xi(T, и) + x%(T, tz)}
и
with fixed terminal time T = 2.
Observe that the reachable set at any time t is bounded. Indeed
|a;i(«)| < t, |x2(<)| < j s2 ds = y.
Moreover, for each t, x, the set of admissible velocities
F(t,x) = {(u,: — 1 < и < 1}
7.2 Convexity + PMP.
135
is a compact, convex subset of IR2. Therefore, by Theorem 5.1.1 in Chap-
ter 5, an optimal solution exists. The PMP yields the adjoint equation and
maximization condition
(P1,P2) = (-2xiP2,0),	(Р1,рг)(2) = (1,1), u = sign(pj).
These in turn imply p?(t) =
pi = -2±ip2 = -2u = -2sign(pi), pi(0) = 0, pi(2) = 1.
The graph of pi(-) therefore is the union of finitely many arcs of parabolas.
By direct inspection, we find the only two solutions:
Pi(i) = 5-t2,

if 0 < t < 1,
if 1 < t < 2.
The corresponding extremal controls are
ut(f) = | ! 1
= 1,
if 0 < t < 1,
if 1 < t < 2.
The trajectory corresponding to the control u*(-) is
(xi,x2)(t) = (t, t3/3) te [0,2].
The trajectory corresponding to ?J(-) is
/	t3\
I —t, — ]	if 0 < t < 1,
(xi,X2)(t) = < z q	/, _	\
( t - 2,	(	} ) if 1 < t < 2.
\	3	3 J	~ ~
Computing #i(2) + #2(2) in the two cases, it is found that iz*(-) is the only
optimal control.
7.2 Convexity + PMP.
We consider here the Mayer problem with fixed terminal time:
inf </>o(*(T,u)),	(7.4)
uQU
for the system (7.2), with initial and terminal constraints
x(to) = xq, x(T) € S = {x;	= 0, i = 1, • • • , k}.	(7.5)
We recall that a function f : A —> IR, where A is a convex subset of a
vector space, is called convex if for every x, x' G A and Л € [0,1] one has:
f(Xx + (1 - A)x') < A/(x) + (1 - A)/(a/).
136	7 Sufficient Conditions
Theorem 7.2.1. In addition to the basic hypotheses (H), let Duf be contin-
uous and assume that the set of admissible controls which steer the system to
the target set
Us = {и : [0, T] U, x(T, u) € S}
is convex. In addition, assume that the functional и > фо(х(Т, и)) from Us
into IR is convex. Then, any trajectory &*(•) = a?(-,u*), which satisfies Pon-
tryagin’s equations, with
к
p(T) = V^0(a:*(T')) + £ Х^ф&*(ТУ)	(7.6)
2=1
for some Ai, • • • , Xk € IR, is optimal.
Proof. Let the control «*(•) and its corresponding trajectory x*(*) satisfy the
PMP. Assume that there exist a different control u^ E Us whose trajectory
t (£) = x(t, г?) satisfies
0о(^(Т)) <0о(х‘(П)-	(7-7)
A contradiction is then obtained as follows. Define the 1-parameter family of
controls
ti«(i) = eu+(t) + (i-e)u*(t) ce [0,1].
Notice that E Us, because Us is convex. Let M(•, •) denote the fundamental
matrix solution of the linear problem
v(t) = Dxf(t,x*(t),u*(t)} • v(t).
By Theorem 3.2.6, we then have
Фф^х(Т,и^) _	dx(T,ut)
£=0
= V^(a-*(T))  /	• Duf(t,x*(t),u*(t)) • (г?(<)-u*(t))dt
Jo
= 0	=	(7.8)
because x(T, u^) E S for all Using (7.6) and (7.8) we now obtain
<У0о(а;(Т,ы?))	dx(T>ut)
-----—------	= Уф0(х (Г))--------—-----
f=0
= V<Ao(x*(T)) • I	 Duf(t,x*(t),u*(t))  (u\t) - u*(t))dt
Jo
= p(T)- I Af(T,i)-D,J(i,2:*(0,u*(f))-(wt(t)-u*(t))</i
Jo
= [ p(t)  Duf(t, x*(f),u*(t)) • (uf(t) - u*(t))dt
Jo
7.3 Dynamic Programming 137
because, by the PMP (see Remarks 6.2 and 6.3 in Chapter 6), the last inte-
grand is a.e. nonnegative. In particular, this implies
фо(х(Т,и*У) > ф0(х(Т,и*)) - | [</>о(^(Г,и*)) - ф0(х(Т,и]У)] ,	(7.9)
for all £ > 0 sufficiently small.
On the other hand, the convexity of the map и i—► ф0(х(Т, и)) implies
<£о(я(Т,?/)) < </>0(x(T,u*)) -£ [ф0(х(Т,и*У) - <£0(а:СГ,г?))] V£ € [0,1].
This yields a contradiction with (7.7) and (7.9), proving the optimality of the
control ?z*.
Observe that the assumptions of Theorem 7.2.1 are satisfied whenever
фо : IRn и-> IR is convex, the control set U and the target set S are both
convex and the system is affine, having the form
x = A(t)x + B(t)u + c(t).
Indeed, if u, u' e ZY, £ 6 [0,1], one has
Ф0(х(т,£и+(i -e)u')) = Фо^^и) + (i - ew,tz'))
< С Фо(х(Т, u)) + (1 - £)ф0(х(Т, и')).
This technique applies in particular to the Examples 6.1 and 6.3 in Chapter
6.
7.3 Dynamic Programming
This section is concerned with the minimization problem
inf ф(Т,х(Т,иУ)	(7.10)
uet/
for the control system
x = f(t,x,u), u(t) e U a.e.,	(7.11)
subject to the terminal constraints
(T,j:(T))€S,	(7.12)
where S C IR71-*-1 is a closed target set. We always assume that the basic
hypotheses (H) hold and that ф : S •—► IR is continuous and bounded below.
Given an initial condition
ж(^о) = xo,
(7-13)
138	7 Sufficient Conditions
call £4o.To the family of all measurable controls и : [to? T] U, for some
T > to? such that the corresponding trajectory of (7.11), (7.13) satisfies the
condition (7.12) at the terminal time T. In order to keep track of the initial
conditions, we write x(-,u,to,xo) for the solution of
x = f(t,x,u) x(to) = Xq-	(7-14)
We now consider a family of optimization problems with different initial
conditions
z(s) = y,
and study how the values of these optimization problems vary with s, y. We
thus define the Value Function as
V(s,?/)= inf x(T, u, s, ?/)),	(7.15)
adopting the convention that V(s, y) = oc if Us>y is empty. Some basic prop-
erties of V are proved in the next theorems.
Theorem 7.3	.1. (Properties of the Value Function). Let V = V(s,y)
be the value function for the problem (7.10)-(7.12). Then
(i)	For every и € U and every admissible trajectory x(-,u), the map t
V(t, xft, u)) is nondecreasing.
(ii)	If и* : [t0,T] i—> U is optimal for the problem (7.10)-(7.12), then
V(t,x(t,u)) is constant for t E [fo?^1]-
Proof. 1. Let и : [to? U be any admissible control. We use the notation
xi = x(ti, u, to'Xo) and assume that
V(t^xx) = V(t0,x0)-6	(7.16)
for some e > 0. A contradiction is then derived as follows. Let v : [ti, T] i—> U
be a control in for which
^(T,x(T,v,tx,x^ < V^Xi) +	(7.17)
Define the control й : [t0,T] •-> U by setting
uft) if te [t0,ti],
vft) if te^T],
Then u e UtQ,xG because x(T. u. to, Xq) = x(T,v,ti,Xi). Moreover, (7.16) and
(7.17) together imply
г[>(Т, x(T,u,to,xo)) < V(to,xo) - |,
u(t) =
contrary to the definition of V. This proves (i).
7.3 Dynamic Programming 139
2. Let t h-> u*(t) be an optimal control, for the initial data z(£q) = #o- Calling
t и-> x*(t) = x(t,u*the corresponding trajectory and using (i), we
deduce
V(t0,x0) = i/>(T,x*(T)) > V(t,x*(t)) > V(to,xo) e [to, Т].
This proves (ii).
Our main goal is to derive a partial differential equation satisfied by the
value function V = V(s,y). In the following, we write
dsV= £-V,

respectively for the partial derivative of V w.r.t. the time s. and for the gra-
dient of V w.r.t. the space variables у = (t/i,..., yn). We regard 4yV as a
row vector, while f(s,y,u) E IRn is a column vector.
Theorem 7.3	.2. (P.D.E. of Dynamic Programming). Assume that the
value function V is C1 on some open set Q C IR x IRn, not intersecting the
target set S. Then, at every point (s,y) € Q, the function V satisfies the
Hamilton-Jacobi equation
dsVM)+ inf{VyV(S,y)-/(S,y,^)} = 0.	(7.18)
Proof. 1. Let (s,y) E Q and consider any constant control u(t) = w E U.
Then by Theorem 7.3.1
at
= dsV(s,y) + VyV(s,2/) • f(s,y,u) > 0.
Since uEU was arbitrary, the left hand side of (7.18) is > 0.
2.	To prove the converse inequality, consider a sequence of controls uy :
[s, Tv\ h-> U such that
^(Tp,x(Tp,Up,s,z/)) < V(s,7/) -I-1/”3, (Tp, x(Ty,uu,s,y\) e S. (7.19)
Since (s,y) S, we can assume Ty > s + 6 for some 6 > 0 and all у > 1.
Calling x„(t) = x(t,uy,s,y), we claim that, for all v sufficiently large, there
exists ty E [s, s + i/-1] such that
•^(L/) --- f(fv, ^1/(^1/)» ^1/(^1/)),
(7.20)
a5V(tp,Xp(tp)) + VyV(tp,j:p(^)) • /(^,Xp(Zp),Up(tp)) < i. (7.21)
Indeed, if our claim does not hold, one would have
140	7 Sufficient Conditions

for a.e. t 6 [s, s + и !],
hence
> V (.s + p 1,xJ/(s + b' *)) > V(s,y) +
in contradiction with (7.19).
3.	By (7.21) we have
inf (tu,xu{tu)) + 7?уУ(^,^(^)) • f(tu,Xy(ty),w) <
As y—>oc we have ty—*s, ху(1у)—+у. Therefore, the continuity of dsV. 4yV
and f, we conclude that the left hand side of (7.18) is < 0. This completes
the proof.
Remark 7.1 (Bolza Problem). Consider a minimization problem in the
more general form (7.1). Then the value function
V(y, s) = inf
L(t, x, u) dt + ^(T, rr(T))
Ф) = у (7.22)
satisfies the Hamilton-Jacobi equation
asV(S,y)+ inf{VvV(m) -f(s,y,iv) + L(s,y,a>)} = 0.	(7.23)
cue и
Remark 7.2 (Minimum Time Problem). If the problem is autonomous,
i.e., f = f(x,u), tl) =	L = L(t,ii). S = {(f,.r); t € IR, x € £},
then the value function V is independent of time. Hence (7.23) takes the form
inf {VV(y)-/(y,w) + L(y,u?)} =0.
cueu
In particular, consider the minimum time problem:
x = /(x,u), u(t) E U for a.e. ,t
x(s) = y, x(T) 6 S'.
Then the value function V = V(y) describes the minimum time taken by a
trajectory starting at у in order to reach the target set S'. On a domain where
V is C1, one has
inf{VV(7/)./(t/,iv) + l} = 0.	(7.24)
7.3 Dynamic Programming
141
Remark 7.3 (Optimal Feedback Controls). By theorem 7.3.1(ii), if a
trajectory x*(-) = rr(-,u*) is optimal, then £V(t, x* (t)) = 0. Therefore,
^^W) + W^W)‘/(^*W^*(0) = 0
for a.e. t at which V(t,rr*(t)) is differentiable. The knowledge of V therefore
allows us to derive an expression for the optimal control u* = u*(t,x) in
feedback form, solving the equation
dtV(t,x) + VxV(Z,x) • f(t, я, u*) = 0
in terms of u*. Equivalently,
u*(£, x) = arg —max VxV(t, x) • /(£, жси).
cuEU
In general, however, the trajectories of
± = f(t, x, u*(t, x))
need a careful interpretation. Indeed, the function (£,x)	u*(t, x) may be
discontinuous, multivalued, or not everywhere defined.
By definition, a control u* : [to, T] h-> U is optimal for the problem (7.10)-
(7.13) if and only if
(T,z(T,u*)) € S, ^(T,x(T,u*)) = V(t0,z0).
The knowledge of the value function V thus provides a straightforward crite-
rion for optimality. Using Theorem 7.3.2, one may hope to recover V as the
(unique) solution to the Hamilton-Jacobi equation (7.18), with boundary data
on the target set S:
V(s,y) =t/>(s,y)	(s,y) e S.
(7-25)
This approach runs into a major difficulty. Since (7.18) is strongly nonlinear,
globally defined C1 solutions do not exist, in general. In fact, in most cases, the
value function is continuous but only piecewise differentiable, with derivatives
dsV, V.V having jumps along a finite number of submanifolds of positive
codimension.
Example 7.2 Consider the minimum time problem on IR:
for the system
dt,
x = u, u(t) € [-1,1]
x(s) = у,
142
7 Sufficient Conditions
with target set
x(T)eS = {xelH\ |ж| = 2}.
The value function independent of time, is then equal to the distance
of у from the set consisting of two points {—2, 2}:
V(S/) = ||»|-2|.
As in (7.24), it satisfies the Hamilton-Jacobi equation
min {VyV-u>+l} = 0	(7.26)
uc[—1,1]
almost everywhere on IR2\S, together with the boundary conditions
V(y)-0 if 12/1 = 2.	(7.27)
Clearly, V is not differentiable at у = 0. Observe that V cannot be charac-
terized as the unique piecewise C1 solution of (7.26)-(7.27). Indeed, we can
choose any piecewise constant function Л : IR i-> {—1,1} such that
y* h(x) dx = 0,
and define	y
W(?/) = [ h(x)dx,
J-2
This provides another piecewise C1 solution to the same equation with the
same boundary conditions.
Given a continuous function W : IR x IRn IR, we shall seek sufficient
conditions which guarantee that W is the value function for the minimization
problem (7.10)-(7.12). If W is globally C1 and satisfies the equation (7.18),
then a uniqueness theorem for classical solutions of Hamilton-Jacobi equations
would guarantee the equality W = V. If W is only piecewise Cl, the problem
is more subtle. A result in this direction is
Theorem 7.3.3. Consider the problem (7.10)-(7.12). Let Q C IRn+1 be an
open set containing the closed target set S, and letW : Q i—♦ IR be a continuous
function such that
(i)	W > V,
(ii)	W = on S,
(iii)	At every boundary point (t,x) € dQ, one has
W(t, x) = M = sup W(s, y),	(7.28)
(s,2/)€Q
7.3 Dynamic Programming
143
(iv)	There exist finitely many manifolds Mi, • • • ,Mw C Q with dimension <
n, such that W is continuously differentiable and satisfies the H-J equation
Ws(s,y) + min{%(s,y) • /(s,y,u>)} = 0
cuGLJ
at every point (s, y) in the open set <2\ U Л4,.
Then W coincides with the value function V on the closure of the domain
Q.
Fig. 7.1. A nearly-optimal trajectory is approximated by one with piecewise con-
stant control. In turn, this is approximated by a trajectory having transversal inter-
sections (or no intersection at all) with the manifolds Mt where W is not smooth.
Proof. 1. Because of (i), it suffices to prove W < V. Assume, on the contrary,
that W(tQ,x$) > V(Iq,Xq) for some (to^o) € Q- Since И7(/0,т0) < M, this
of course implies V(to,^o) < Af. By the continuity assumptions on W, there
exist e, 6 > 0 such that
V(t0,z0) + e < M
(7.29)
|z - x0| < 6
W(t0,x) > V(t0,x0) + e.
(7.30)
2.	Let iz* : [to,!1] U be a control whose corresponding trajectory £*(•) =
z(-, w*, to, xq) satisfies
(T,x*(T)) e S, il>(Tyx*(T)) < V(t0,x0) + e. (7.31)
If uy : [to ,T] i—> U is a sequence of piecewise constant controls approaching
w* in the L1 norm, by Theorem 3.2.1 in Chapter 3 the trajectories
x„(t) = x(t, Uy, T, x*(T)),
144	7 Sufficient Conditions
with the same terminal point as x*, converge to #*(•) uniformly on [to, Т].
In particular, there exists a piecewise constant, left continuous control :
[to?T] *—► U such that the trajectory а;й(-) = x(-, u\ T, x^(T)) satisfies
(ar^to) - zo| <
(7.32)
3.	Let t"] be an interval on which the control u$(t) = ш is constant, and
such that (t,x#(t)) € Q for all t	Then
- IV(t',a;l,(t')) > 0.	(7.33)
Indeed, by the Transversality Theorem 2.4.1 in Chapter 2, there exists a se-
quence of points zm^>x^(t') such that the solution xm(-) to the Cauchy prob-
lem
tu), Xffi^t ) — Zm
has only transversal crossings with each manifold Mj. Hence (t, xm(t)) €
Uj-Mj only at finitely many times t. We can thus use the assumption (iii) and
obtain
W",M*")) - W(t',xm(t'))	(7.34)
= / [Ws($,zm(s)) + Wy(s,j:m(s)) • /(МпМu)] ds > 0.
(7.35)
Since W is continuous, letting m—>oc in (7.34), we recover (7.33).
4.	We now consider two separate cases. First, the graph {(t, x^(t)) : t G [to? ^]}
is assumed to be entirely contained in Q. If is constant on the intervals
with to < ti < • • • < t/v = T, from (7.33) it follows
N
W(T,x\T)) - W(t0,x*(t0)) =	[Wi.^(ti)) -	> 0.
i=l
(7.36)
By construction, хЦТ) = x*(T), hence (7.36) and (7.31) together imply
Wo,^(*o)) < W(T,x*(TY) = Ж*‘СП) < V(to,xo) + £.	(7.37)
Recalling (7.32), this yields a contradiction with (7.30).
5.	It remains to consider the other case, where (t,x$(t)) is not always inside
Q. Since Q is a neighborhood of the closed target set S, the time f where the
trajectory rH(-) makes its last entrance in Q satisfies
f = inf{i<T; (S,x*(s))eQ Vse[t,T]}<T.
7.3 Dynamic Programming
145
Moreover, by the assumption (ii),
lim	= M.
t^>T+
Choose т e ]т, T] such that
(7.38)
W(r, ^(t)) > M - E.
(7.39)
Recalling (7.29), the same argument used in (7.37) now yields
Ж(т,^(т)) < РИ(Т,^(Т)) = ^(T,xtt(T)) < V(t0,x0) + e< M - e,
a contradiction with (7.39).
Remark 7.4. In the case where Q C 1R+ x IRn, the conclusion of the theorem
still holds if we assume that (7.28) is satisfied only for the boundary points
(t, x) € dQ with t > 0.
In practice, one can use Theorem 7.3.3 in the following way. Assume that,
for every initial data (t, y) in a neighborhood Q of the target set S we can
construct a trajectory 11—> xs,y(t), which reaches the target set and achieves a
cost W(s,y). Even if this trajectory satisfies the Pontryagin Maximum Prin-
ciple, a priori there is no guarantee that it be optimal. However, assume that
the function IT is piecewise smooth, and satisfies the P.D.E. of dynamic pro-
gramming (7.18), outside finitely many smooth manifolds All,..., of
lower dimension. Then, using the theorem, we would like to conclude that our
function W actually coincides with the value function, i.e. W(t, x) = V{t,x)
for all (2, x) € Q. By construction, the assumption IT > V is automatically
satisfied. The only remaining condition to achieve is (iii), requiring that the
global maximum of W on Q should be attained at all boundary points of Q.
For this purpose, it is often useful to replace the domain Q with the sublevel
set
Q' = {(t,x) 6 Q; W(t,x) = M},
for a suitable value of the constant M.
Another situation where the same arguments in Theorem 7.3.3 can be
applied is illustrated below.
Corollary 7.3.4. Consider the problem (7.10)-(7.12), where the target set is
S = {T} x lRn. Let Q =]£,T[xIRn and let W : Q i—> IR be a continuous
function such that
(i)	W > V,
(ii)	W = onS={T}x IRn.
(iii)	There exist finitely many manifolds A4i, * • • , C Q with dimension <
n, such that W is continuously differentiable and satisfies the H-J equation
146	7 Sufficient Conditions
Ws(s,y) + min{Wy(s,p) • f(s,y,a>)} = 0
at every point (s, y) in the open set Q\ U Mi.
Then W coincides with the value function V, on the closure of the domain
Q-
Proof. By (i), again it is enough to prove the inequality W < V. All steps 1.
to 4. in the proof of Theorem 7.3.3 remain valid. Indeed, they are based on the
continuity of W and on the assumption (iii). By the form of the domain Q, it
is obvious that any trajectory (£,a^(£)) cannot leave or re-enter Q before the
terminal time T. Therefore, the situation considered in step 5. never occurs,
and the proof is complete.
Example 7.3 For the linear system
(±i,±2) = (x2,u), u(t) € [—1,1] a.e.,
(7-40)
consider the problem of reaching the origin in minimum time
min{T; (aq,a;2)(T) = (0,0)}.
Since the system is autonomous, the value function does not depend on t and
satisfies the Hamilton-Jacobi equation
min W(yi,2/2) • (y2,w) =-I-
(7-41)

Pontryagin’s equations yield
(РьРг) = (0, -Pi), w(t) = -sign(p2(i)).
Hence pi is constant and p2 is a linear function of t. which can change sign at
most once. Therefore, the only controls which may be optimal are bang-bang
with at most one switching:
Observe that the trajectories of (Jq, ±2) = (rr2,±l) are arcs of parabolas
aq = C ±^2/2* Any initial condition (?/i,?/2) can be connected to the origin
by two such arcs. More precisely, call
If У1 > ^(^2), the point (?/i, У2) can be steered to the origin along the two
parabolas
I	X if A
Xi = I Pl + у
7.3 Dynamic Programming
147
The point of intersection (xi,^) of the above parabolas (see figure 7.2) is
computed as
/	2
z- - \	/ У1 , У2
(xi,x2) = I — + — ,
Recalling that |^21 = Ы = 1, the total time taken to travel along the two
arcs is
^(271,3/2) = I3/2 - x2\ + Ы = 3/2 + 2
-4
For 2/1 < 92(2/2) the analysis is entirely similar, showing that W(y\,y2) =
W(—2/1, —y2). Observe that W is smooth except at the origin and on the two
manifolds
Г	2/2
Л41 = < (2/1,3/2) : 3/2 = у, 3/2 < 0
f	3/2
A42 = < (3/1,3/2) : 3/1 = ~y5 3/2 > 0
Moreover, in the region where 2/1 > ^(^2), the Hamilton-Jacobi equation
(7.41) takes the form
f dw	dw
^€[-1,1] [ дуг	dy2
= -1.
+ min
Therefore, on Q = IR2 all of the assumptions in Theorem 7.3.3 hold. Hence
all trajectories are optimal.
Fig. 7.2. Optimal trajectories reaching the origin in minimum time.
148
7 Sufficient Conditions
7.4 Relations between the P.M.P.
and the P.D.E. of Dynamic Programming
Consider an optimization problem in Mayer form
max	(7-42)
uElA
for the control system
x = f(t,x, u),	w(£)cU.	(7.43)
Call V — V(r,y) the maximum payoff attainable with the initial condition
x(t) = y. By Theorem 7.3.2, on regions where V is smooth, this value function
satisfies the Hamilton-Jacobi equation
dTV + H(r,y,W) = 0,
(7-44)
where
H(r,y,p) = max p  f(r,y,u>).
(7-45)
Aim of this section is to establish a basic connection between the P.D.E.
(7.44) and the Pontryagin equations (6.4)-(6.5).
V=V
T
Fig. 7.3. A characteristic curve.
As a preliminary, we recall here the method of characteristics, for the
solution of the first order P.D.E. (7.44). Consider any differentiable curve
t (£,#(£)) in t-x space (see fig. 7.3). We seek a system of O.D.E’s describing
how the function V varies, restricted to this particular curve. From (7.44) it
follows
~V(t,x(t)) = dtV + i-VV = -H(t,i,VV) + iVV. (7.46)
at
Of course, (7.46) cannot be solved by itself, because the right hand side con-
tains the unknown quantity W(t, x(t\). We thus introduce the gradient vector
p(t) = W(£,z(£)), so that
7.4 Relations between the P.M.P.and the P.D.E. of Dynamic Programming 149
Differentiating (7.44) w.r.t. хг, i = 1,... , n, we obtain
	d	dH	dp, dH air' + дх, + Эх, Op, ~	'
and we also have	^t,x(t)) = %+± ir£.	p.48) Cot-	С/i J=1	J
The identity VXiXj we thus obtain	— VxjXi yields dpi/dxj = dpj/dxi. From (7.47) and (7.48)
d /	/ \ \
—Pi(t,x(t)) =
ЭЯ	ЭЯ ул. d^_
dxi “ dxi dpj " lj dxj
(7.49)
Notice that, in order to compute the time evolution of p = W along an
arbitrary curve, according to (7.49) one also needs to know all the quantities
dpi/dxj. However, if we choose a special curve such that x = dH/dp, then
the last two terms on the right hand side of (7.49) cancel out. We thus obtain
the closed system of n + n O.D.E’s
Xi =
Pi =
dH
dpi ’
dH
dxi >
i = 1,..., n .
(7.50)
For each terminal point x, the hamiltonian system (7.50) can be solved with
terminal conditions
x(T) = x,
p(T) = VV(T,x).
(7-51)
Next, we show that, under suitable regularity conditions, the hamiltonian
system of O.D.E’s (7.50) coincides with the Pontryagin equations. Indeed,
consider the hamiltonian function (7.45) and write
H(t,x,p) = p • f(t, x,u(t,x,p)),	(7-52)
where
u(t, x, p) = arg max p • f(t, x, u).	(7.53)
To avoid technicalities, we suppose the above maximum is assumed at an
interior point of U. This is certainly the case if U = lRm. The above definitions
then imply
ЭИ d f	d f du	d I
(7.54)
150	7 Sufficient Conditions
Indeed, p • df /ди = 0 at an interior point where the maximum is attained.
Moreover, by the same argument we obtain
yy	Q f du
— = f(t,x,u(t,x,p)) + p- — (t,x,u(t,a:,p)) = f(t,x,u(t,x,p)).
op	OU op
(7.55)
By (7.54)-(7.55), the hamiltonian system (7.50) is equivalent to the two vector
equations
x = f(t,x, u(t, х,рУ) ,
df
p = —p • — (t,x.u(t,x,pY),	(7.56)
dx
with u(t,x,p) defined by the maximality condition (7.53).
In conclusion, the above analysis has shown that the equations of char-
acteristics for the Hamilton-Jacobi P.D.E. of dynamic programming coincide
with the Pontryagin’s equations (7.56), (7.53).
7.5 Linear-quadratic case
In this section we apply the methods of Dynamic Programming to a special
class of optimal control problems with linear dynamics and quadratic costs.
More precisely, we consider the system:
x(t) = A(t) x(t) + B(t) u(t), x e ИГ, и e IRm, (7.57)
with A n x n matrix and В n x m matrix, and the optimal control problem
mm /
Js
[??#(t)u + xfQ(t)x] dt + х(ТУ5х(Т),
z(s) = У, (7.58)
where U is the class of all measurable and a.e. bounded controls, T is fixed, R
is a m x m matrix, Q and S arc n x n matrices, and * denotes the transpose.
We make the following assumption:
(H*) The functions t A(t), t —> B(£), t	R(t), t —> are all measurable
and bounded on compact intervals. The matrices R(t),Q(t) and S are
symmetric. Q(t) and S are positive semi-definite, i.e. xtQ(t}x > 0 for every
x e ПГ and same for S', while R(t) is positive definite, i.e. z?H(t)u > 0
for every и G ПГ and equality holds only for и = 0.
Using the expression for solutions to linear systems, we notice that the cost
function is quadratic in и and y. With this in mind, as a candidate for the
value function, we consider
W(s,y) = у'Р(з)у,
(7.59)
where P is a symmetric n x n matrix to be determined.
7.5 Linear-quadratic case
151
Theorem 7.5	.1. The function (7.59) is continuously differentiable and is the
value function if and only if matrix valued function t h-> P(t) is a solution to
the system of O.D.E’s
( P(t) =	- P(t)A(t) - A4t)P(t) - Q(t) (	.
\P(T) = S	’ U J
The matrix equation (7.60) is known as the Riccati Differential Equation.
Proof. 1. Assume first that the function W in (7.59) is differentiable and
coincides with the value function. The terminal condition at time t = T clearly
implies P(T) = S.
Recalling Remark 7.1, we have:
0 = <9.SIT + min • [A(s)z/ + B(s)cu] Tcuf/?(s)cj + ylQ(s)y^. (7.61)
Since the expression in braces is a convex quadratic function of cu, the mini-
mum is attained for the value of satisfying:
° =	{^yW '	+	+ Wffl(s)w + y'<2(s)i/j
= [p(S)y + P‘(s)y] + 7?(s)w + R\s^.
Recalling that P(s) is symmetric and R(s) is symmetric and invertible, we
obtain
(7.62)
Substituting this value of w in (7.61), for every у 6 IRn we have
0 = y* [p(s) - P(s)B(s)/?-1(s)Bt(s)P(s) - 2X‘(s)F(s) - Q(s)]
У-
Since А*(з)Р(8)у is scalar, it is equal to its transpose yl P(s)A(sfy. In con-
clusion, if the value function has the form W(s, у) = у*Р($)у, then the differ-
ential equation in (7.60) must hold.
2.	Next, assume that P solves (7.60). Observe that the matrix P(t) is
symmetric for every time t. Indeed, its transpose Pl solves the same equation
and has the same terminal condition P*(T) = S = P(T). We want to apply
Corollary 7.3.4 to the function W(s, y, z) = W(s, y) + z on Q =] — oo, T[xIRn x
IR and with dQ = S — {ffT,y,z) : у € IRn,z 6 IR}. Define xSiV as the
trajectory corresponding to the feedback control
u(t,x) = -R~1(t)Bt(t)P(t)x
152
7 Sufficient Conditions
as in (7.62), with initial condition xs,y(s) = y. Then the same computation
as in 1. shows that W is constant along (xs,y, z) for z(s) = 0. In particular:
W,?/,0) = W(T,x„,v(T),z(T)) =
T
I [us,y(t)R(t)ua,y(t) + x*s,y(i)Q(t)arSiJ)(O] dt + x^y(T)Sxs,y(T),
J s
where us,y(fi) = w(xs,y(t)Y Hence the assumption (i) holds.
Since P(T) = S, the condition (ii) is also satisfied. Finally, the computations
of 1. show that W satisfies the P.D.E. of dynamic programming on the entire
domain Q. Therefore, W coincides with the minimum value function.
To solve LQ problems we only need to show that there exists a solution
to the Riccati Differential Equation:
Theorem 7.5	.2. The backward Cauchy problem (7.60) has a unique solution
defined for all t € ] — ос, Т].
Proof. The Riccati equation is an O.D.E. on Rn*n whose right hand side is
continuous w.r.t. time and smooth (a quadratic polynomial) w.r.t. the space
variable. Therefore, the Cauchy problem admits a unique local solution. This
solution can be continued backwards up to a maximal interval [f, T], with
lim ||P(f)|| = +oo.	(7.63)
t—T+
We claim that (7.63) cannot happen for a finite time t. Indeed, for every
t < T, the value function W(t,y) is bounded by the cost of the trajectory
corresponding to control и = 0, i.e.
0 < W(t,y) = у1Р(у)у
< f y'dT + у1М*(Т,1)8М(Т,1)у.
By assumption (//*) there exists C > 0 such that ||M(t, t)|| < 1 + C(r - t).
This shows that ||P(t)|| is bounded on every bounded interval of time. Hence
no blow-up occurs and the solution of (7.60) is well defined for all times t <T.
Once the solution to (7.60) is found, an optimal control can be easily recovered
from the HJB equation. In fact since W in (7.59) is differentiable the optimal
control is given by (7.62) thus we have:
Corollary 7.5.3. The optimal feedback control for the problems (7.57) (7.58)
is given by
u\s,y) = -R~1(s)Bt(s)P(s)y,
where P solves (7.60).
7.5 Linear-quadratic case
153
Next, we show how the asymptotic stabilization problem can be solved
also in terms of a linear-quadratic optimization. Consider now the linear au-
tonomous system
i(t) =	+	zeIRn,	(7.64)
where A is an n x n matrix and В is an n x m matrix. For given strictly
positive, symmetric matrices P Q consider also the optimal control problem
min, y* [u*Ru 4- xlQx\ dt. x(0) = x.	(7.65)
i.e. problem (7.57) (7.58) with A.B.R.Q constant and S = {0}. We assume
that the linear system (7.64) is controllable. We shall study the limit of optimal
trajectories, as T tends to oo. More precisely, we claim that, as T —> oo, the
optimal trajectories tend to the solutions of a stabilizing feedback.
Given T > 0 let us denote by the value function of (7.64)-(7.65) and by
P^ the solution to (7.60) with S = {0}. First, since (A, B) is controllable,
for every x and T we can find a control й : [0, T] —> lRm such that the
corresponding trajectory satisfies x(T) = 0. For every T > T. we can prolong
й on [0, T], setting u(t) = 0 for t > T, thus we get:
Гт
х^Р<т)(0)^ = V^T\x) < / [й*Рй 4- xlQx\ dt=	4- x*Qx] dt.
Jo	Jo
This proves that the function T i—> xJP(T\tyx is uniformly bounded.
Given Ti < T2 let u be the optimal control for x and time T2, then:
гТг
xJP^'^x — V^T2\x) = /	4- x*Qx\ dt
Jo
fT1
> / [й*Яй + xfQx] dt > V(Tl\x) = xJP^^x.
Jo
Hence, for each x € IRn, the map T н-> х1Р^т\0)х is nondecreasing and uni-
formly bounded. Therefore, it admits a limit as T —► 00. Since x is arbitrary,
we conclude that P^1 —► P^ for some symmetric, positive definite matrix
Poo- We claim that the feedback control defined as
u(x) = -R~1BtP(Xix.	(7.66)
stabilizes the system (7.64) asymptotically to the origin.
The existence of a monotone limit implies that the time derivative of the
map T 1—> P(T)(0) tends to zero. Since the map t 1—> P^T\t) satisfies (7.60)
and p(T-e)(0) = P(t\e), we have
0 = - lim inf p(T\0) — lim -^-(P^T\s))
T—>oo dT	v 7 T^oodsy
154	7 Sufficient Conditions
= lim (P^XtyBR-^pC1")^) - Р(Т\О)Л - A(P(T)(0) - (?),
T—»oo
Therefore the matrix satisfies the algebraic Riccati equation
0 = PooBR-1BtPao-PxA-AtPoo-Q.	I
(7.67)
Now fix an initial point x(0) = x, and let x(-) be the trajectory of the
linear system (7.65), using the feedback control (7.66).Using (7.67) we find
^xXt)Pxx(t) = (xt(t)At+ut(x(t))Bt)P00x(t)+xt(t)P00(Ax(t')+Bu(x(t))) =
= -x^^^BR^B^^ + Q)x(t).
The definition of value function \РТ) implies
rT
xl
о
0
oo + Q]x(t)dt
Г d
dt
=	- —хХ^РоаХХ) dt = XtP<x)X - х‘(Т)РооХ(Т) .
Jo
Taking the limit as T —► oc we obtain
lim х1Р^х < x^P^x - lim х*(Т)Роож(Т'),
T—>oo	T—>(x>
and hence
lim xt(T)Poox(T) = 0.
T -too
Since Рю is positive definite, this implies we conclude lim^oo #(7") = 0. The
other property of a stabilizing feedback is also easily checked.
7.6 Optimal syntheses
In this section we illustrate an alternative method to ensure optimality for
various initial conditions. We consider the optimal control problem:
inf [ L(x(t, u), u(t)) dt,	(7.68)
«(•)ew Jto
for the system
x = f(x,u),	a.e.,	(7.69)
with initial and terminal constraints
z(0) = x0, x(T) = 0.	(7.70)
7.6 Optimal syntheses 155
We assume hypothesis (H) holds.
A synthesis on an open set J? is a collection of trajectories
Г — {^х : x € 12,7* steers x to S},
and Г is optimal if every is. The main goal of this section is to prove
that a synthesis, formed by trajectories satisfying the Pontryagin Maximum
Principle, is optimal provided it covers the entire space in a regular fashion.
Given a single trajectory satisfying PMP there is no regularity condition
which ensures optimality, as shown by next example.
Example 7.4 Consider the control system
(±i,±2) = (u, 1 + xl\ (xi(0),z2(0)) = (0,0), |it| < 1
and the problem of reaching the point (0,1) in minimum time. If we choose
the control и = 0 then we get the trajectory 7 given by:
^i(f)=0,	x2(t) = t,	(7.71)
that reach the final point in time 1. The equation for the adjoint variable is:
(Ai, A2) = (-2a;iA2,0)
and, using (7.71), one easily verifies that defining Ai(Z) = 0, A2(t) = 1, 7
satisfies Pontryagin Maximum Principle. Moreover 7 and the corresponding
control are analytic, even more: polynomial. However, one easily check that
any control й satisfying
u 7^ 0,	/* й($) ds = 0, t + I ( I й(т) dr\ ds = 1,
Jo	Jo \Jo /
steers the origin to (0,1) in time t < 1, thus 7 is not optimal.
The degeneracy of the example can not happen for a synthesis, indeed we
introduce a concept of regularity ensuring optimality. Our interest is to con-
sider the case in which Г is generated by a feedback which is piecewise smooth
in the following sense. We can partition the space into regular manifolds of
various dimensions, such that the feedback is smooth on each region. To give
a precise definition we need some notation.
A set P G IRn is said a curvilinear open polytope of dimension p, if there
exists a polytope (i.e. bounded closed region intersection of a finite number
of half-spaces) P' G IRP and a smooth map ф : 1RP —> lRn, injective with
jacobian having maximal rank at every point, such that ф(Р' \ дР') = P.
Let 12 be an open set containing the origin. We say that P is a Boltyanskii-
Brunovsky regular synthesis, briefly BB synthesis, if the following holds. There
exists a 6-tuple S = (P, Pi,Р2,П, 27, it) such that
156
7 Sufficient Conditions
(BB1)	P is a collection of curvilinear open polyhedra and <2 is disjoint union
of elements of P. If P3 Pk e P and Рк П Pj 0 then Pk C dPj and
dim(Ffc) < dim(Pj).
{0} € P and the elements of P are called “cells”.
(BB2)	P\{{0}} is the disjoint union of P\ (the set of “type I cells”) and P^
(the set of “type II cells”),
(BB3)	the feedback и : {x : 3Pi E Pi,x E Pi} —> U and П : Pi —> P are
maps, 27 : P2 —> Pi is a multifunction, with non empty values, such that
the following properties are satisfied:
i)	The function и is of class C1 on each cell.
ii)	If P] € Pi then f(x, u(x)) E TxPi (the tangent space to Pi at x) for every
x E Pi. In addition, for each x £ Pi, if we let £x be the maximally defined
solution to the initial value problem
£ = /«, £(0) = ®, ее Fl,	(7.72)
and define tx = sup Dom(£x), then the limit £x(tx —) := lim^ £ж(£)
exists and belongs to 77(Pi).
iii)	If P2 E P2, then for each x E P2 and P E ^7(P?) there exists a unique
curve : [0,	[ —► J? such that the restriction of to ] 0, t? [ is a
maximally defined integral curve of the vector field /(-,!/(•)) on P, and
ef (o) = x.
iv)	On every cell Pj E P\, x —> tx is a continuously differentiable function, and
(t,x) —> £x(£), (£,#) “►	:= ?z(&rW) are continuously differentiable
maps on the set
E(P) := {(t,x) : x E Pi , t E [0,^]}.
If P2 E P‘2 the same holds for every , u* , with P E 27(P2).
v)	For every x E f?\{0}, the trajectory : [0, Tx\ —> IRn, yx e Г, is obtained
by piecing together the trajectories on every single cell. Moreover, yx
changes cell a finite number of times.
Remark 7.5. Notice that condition (BB1) essentially means that the collec-
tion of open polyhedra form a ’’triangulation” of the set 12, see Figure 7.4.
In the same figure we represent the two type of cells to illustrate properties
(BB2) and (BB3). The cell Pi is of type I and dimension equal to 2, then there
are trajectories running on it, corresponding to the feedback u, which end on
the cell 77(Pi). On the contrary, the cell P2 is of type II and dimension equal
to 1, hence from every point there start some trajectories entering other cells.
More precisely, from every point of P2 it starts one trajectory entering the
2-dimensional cell P3 and one entering the 2-dimensional cell P4, this means
S(P2) = {P3,P4}.
Theorem 7.6.1. (Boltyanskii-Brunosky sufficiency theorem) Let Г be
а В В synthesis on IRn formed by trajectories satisfying PMP, then Г is opti-
mal.
7.6 Optimal syntheses 157
Fig. 7.4. Example of BB synthesis.
Proof. We define a candidate value function Wr in the following way:
^(1) = I L(4x(t),u(?ix(t)))dt.
Jo
We claim that Wr satisfies the assumptions of Theorem 7.3.3.
1. First, fix x belonging to a cell Pi of maximal dimension n (which nec-
essarily is of type I). By BB3) ii), from every x, in a neighborhood of ж, it
starts a trajectory £x corresponding to it(rr), defined up to time tx and run-
ning on the cell Pi. Then, by BB3) iv) the functions x —> tx, (t,x) —► £x(t)
and (t,x) —» ux(t) := u(^x(t)) are continuously differentiable. By BB3) ii),
the trajectories £x end on the cell 77(Pi) with x £x(tx) continuously differ-
entiable. Then we can use again BB3 ii), or iii), and iv) for the cell 77(Pi) and
prolong the functions x —> tx, (t,x)	£x(t) and (t,x) —> ux(t) in a continu-
ously differentiable fashion. Using BB3) v), by recursion we prove that these
functions are defined and continuously differentiable up to times Tx. Then, it
easily follow that Wr is differentiable at x.
2. Let us denote by (A, Ao) the adjoint covector along (7z,7fc)- Setting
x = x + ev, we want to compute:
pTx	/>тх
Xo(Wr(x+Ev)-Wr(xy) = /	A0L(7x(t),?ix(t))dt— /	XqL(^x (t), ux(t))dt.
Jo	Jo
From 1. it follows:
||7x(t) -7И011 =O(e)-
(7.73)
Assume Tx < Tx, the other case being similar. In the following we consider
integrands which are also of order O(e), hence we can compute all integrals
up to Tx, possibly defining the integrands to be zero after Tx and adding a
o(e) term. Hence we can write:
158	7 Sufficient Conditions
Ao(Wr(z + ev) - ВД =
= [ X0(L(yx(t),ux(t)) - L(7S(f),uI(t)))di
Jo
+ I A0(i(7i(t),uI(t)) - L(^x(t),ux(t)))dt + o(e)
Jo
= Л + /2 + o(e).	(7.74)
We start estimating Д:
Ii = Ao [ I DyL(yx(t) + 0(yx(t)• (yx(t) - ix(t))d0dt
Jo Jo
= Ao У У [£>JZ£(7x(0 + ^(7x(0-7®(0)>ux(0)
-DyL^x(t) + #(7x(0 ~ 7*(0)> «*(<))] ’ (7x(0 - ^x^dOdt
+Ao У У [£>y£(7S(t) + 0(^x(t)-yx(t)),ux(t)) - Иу£(7±(/),г4г(0)]
•(7x(0 -'Ti^dO dt
+ [ X(jDyL(yx(t),ux(t)) • (7x(0-7x(t))dt,	(7.75)
Jo
and the first two terms can be estimated as o(s). The equation (6.75) can be
compactly written as
A(f) = —A(t) • Dyf(^x(t),ux{x)) - X0DyL(yx(t),ux(t)). (7.76)
In turn, this yields
Ii= [ <-A(0 - A(t) • Dy/(7S(i),tti(0),7x(<) “ 7xW)dt + o(e)
Jo
JTs d
= -y ^(А(0,7х(0-7гда
+ I (A(t),/(7x(t),«x(0) -/(7х(0,«х(0))л
Jo
- [ (A • £>v/(7i(t),tti(O),7x(0-7x(0M* +°(e)
Jo
= (A(0),7i(0) - 7г(0)) - (А(Тг),7х(Г^) - 7s(rx)>
+ [ (A(0,/(7s(<),Wx(0) - f(li(t),ux(t)))dt
•h
+ I (A(t)r /(7x(0, ux(t)) - fbiit),ux(t)))dt
Jo
+ / (A • £>уУ(7г(0,иг(0),7х(«) - 7ж(0)^ + «(£)•
Jo
7.6 Optimal syntheses
159
Since 7x(Ti) = 7±(T±) = 0, the second addendum vanishes, while the sum
of the last two integrals is of order o(s), indeed we can argue as for (7.75)
replacing L with f. From the minimization condition of PMP (6.76) it follows
(A(t) , /(7x(<),WzW)> + A0L(7x(t),«®(i))
< (A(i),/(7£(t),ux(t))) + A0L(7s(f),Ux(t)),
for every x and almost every t. Therefore
yT£
Л > (A(0),x - x) - Aq / A0(L(7^(t),ux(i)) - L(7^(^),Ux(f)))df + o(e).
Jo
Notice that the second addendum is precisely the term /2 in (7.74). Dividing
(7.74) by e, using the above inequality, and passing to the limit as e goes to
zero, it follows:
X0(DyWr(x),v) > (A(0),v),
and equality holds, since both terms are linear in v. Now Aq 0 otherwise
A = 0, but this would contradict the non triviality of adjoint covector. Hence,
it is possible to normalize Ao = 1 and finally obtain:
DyWr(x) = A(0).
Using again the the minimization condition of PMP (6.76), we have:
(A(t),/(7x(0>w)> + M7x(t),w) > 0
for every ш € U and almost every t, with equality holding for ш = ux(t). From
the continuity of /, L, A, and ux near 0, we get:
{DyWr^x^f^x^}) 4-T(rr,cu) > 0,
for every w with equality holding for cj = ux(t). Since x and Pi are arbitrary,
this proves iv) of Theorem 7.3.3.
3. Conditions ii) and iii) hold trivially. Finally, since Wr(x) is the cost of
the trajectory 7X, by definition of value function we have i). Therefore we can
apply Theorem 7.3.3 and conclude that Wr coincides with the value function,
but this exactly means that Г is optimal.
Remark 7.6 Theorem 7.6.1 can be proved also for synthesis on an open set
12, assuming that Wr satisfies (iii) of Theorem 7.3.3. Various generalizations
can be find in [71].
Example 7.3 (continued) Consider again the minimum time to origin prob-
lem for the controlled equation (7.40). After straightforward computations,
one checks that the optimal trajectories, represented in Figure 7.2, correspond
to the discontinuous feedback (1.15) of Chapter 1. The collection of optimal
trajectories form a BB synthesis, see Figures 7.5. There are four cells all of
160	7 Sufficient Conditions
type I, on which it is defined the feedback (1.15). The first two Pi and P2
are of dimension 2 and are located, respectively, below and above the curve
= —sign(x‘2)x’2/2. The cells P3 and P4 are of dimension 1 and form the
same curve. The trajectories on cell Pi reach cell P3, hence Я (Pi) = P3, while
trajectories on cell P2 reach P4, hence = P4. It is easy to check that
assumptions (BB1)-(BB3) hold, hence this synthesis is аВ В synthesis.
Example 7.5 Consider the minimum time problem to the origin for the
control system:
(	= и
1 •	12 5
[ X2 = —#1 - |^i
where и e U = [—1,1]. The trajectories corresponding to constant controls
±1 can be described giving X2 as a function of x\. They are, respectively,
cubic polynomials of the following type:
X2 —	T a a e IR
#2 — “б" T + Q Ct G IR.
(7.77)
Consider the curve S = {(^1,^2) € IR2 • #1 = — 1}- It is easy to check
that any trajectory running on S satisfies the PMP. In fact if 7 : [to, ^1] is
such curve, then the corresponding control is constantly equal to 0 and the
evolution equation for the covector is
Ai — A2(l + ^i), A2 — 0.
Hence the covector A is constant along 7. Moreover the maximality condition
maxA • /(7(^),o;) = Ai?i(£) + ^A2,
is always satisfied taking Ai = 0.
7.6 Optimal syntheses 161
Given b > 0, consider the trajectories 71 : [—6,0] »—> IR2 for which there
exists to e [0,6] such that 71 corresponds to control -Fl on [—6, —to] and 71
corresponds to control —1 on [—to, 0]. Define also the trajectories 72 : [—6,0] 1—►
IR2, b > 2, for which there exists ti e [b. 2] such that 72 corresponds to control
— 1 on [—6, — ti] and 72 corresponds to control -hl on [—ti,0]. For every b > 2,
these trajectories cross each other in the region of the plane above the cubic
(7.77) with a = 0 and determine a curve К of points admitting two optimal
trajectories.
We use the symbols хл (b, to) and x~+(b, ti) to indicate, respectively, the
initial points of 71 and 72. Explicitly we have
1	_	.	. (2to — 6)3 (2to — &)2	2	^0
x+- =2t0-b	x+~ =	-I- - v 0 J- +$ + -%	(7.78)
о	2	о
+	,	„	,	(b-2tO3	(£>-2<i)2	2 tf
xi+=b-2t,	z2+ = -----+	(779)
О	2	о
As b varies in [2, 4-oo[, the equation
£+”(Mo) = z"+(Mi),	(7.80)
describes the set K. From (7.78), (7.79) and (7.80) it follows:
to — b — ti ty — 2t2 T (2 4- 36)ti ~h (—62 — 26)^ — 0.
Solving for ti we obtain three solutions:
t\ = 0, t\ — 6,	= 1 ~h —.
The first two solutions are trivial, while the third determines a point of A, so
that:
f	21
К = Haq, x2) : zi = —2, x2 > -- > •
I	о I
The optimal synthesis is a BB synthesis and is portrayed in Fig. 7.6. Notice
that both S and К are cells of the BB synthesis, with S of type I and К of
type II.
Remark 7.7 A complete theory of two dimensional time-optimal syntheses,
with many examples, can be find in [12].
Problems
7.1.	Consider the optimal control problem:
x = A x + В и, x(0) = ж,
162
7 Sufficient Conditions
s
Fig. 7.6. BB synthesis for Example 7.5.
min / \xTQx 4- uTRu] dt
ueu ,/0 L	J
where x E IRn, и E IRm, U — L1([0, T]; IR™), Q and R positive definite.
Prove that Theorem 7.1.1 can be applied to this case.
7.2.	Consider the optimal control problem:
±1 =	4- T2, x<2 — 2X2 + w, |u| < 1,
max^j(T') +	z(0) = 0,
with T fixed. Determine the optimal control.
7.3.	Prove that Theorem 7.3.3 can be applied to the case of unbounded value
functions, replacing assumption hi) with:
lim IV(ir) = 4-oo.
x—^dQ
7.4.	Consider a geostationary satellite and assume that the local motion
around a stable orbit is given by the linear system:
x = Ax 4- Bu.
To ensure transmission, the satellite must track a given trajectory ?/(£),
for t E [О, Т]. Assuming that the fuel consumption amounts to ulR.u for
given matrix R positive definite, find the trajectory that minimizes the
running cost sum of the fuel consumption and the distance from y(t).
(Hint: Write the running cost and reduce to an LQ problem by adding a
fictitious variable.)
7.5.	Consider the minimum time to origin problem for the linearized pendulum
with external force:
x = x 4- u, x E IP, |u| < 1.
7.6 Optimal syntheses 163
(a)	Use PMP to compute a candidate optimal trajectory for every initial
point (x,i).
(b)	Compute a (discontinuous) feedback u(x) so that all trajectories of
(a) solve x = x 4- u(x).
(c)	Compute the cost function W for trajectories of (a). Show that W
is C1 outside two piecewise smooth curves contained in the first and third
quadrant.
(d)	Show that we can apply Theorem 7.3.3 to W, thus all trajectories
of (a) are optimal.
(e)	Show that trajectories of (a) form a BB synthesis and compute the
corresponding cells. How many cells cover the set {(#,£) : x2 T x2 < 7r}.
7.6.	Consider the minimum time to origin problem for the controlled equation
(7.40).
(a)	Is the value function Lipschitz continuous?
(b)	A function W : IRn —> IR is said semi-concave if for every com-
pact convex К C IRn there exists a constant Ck > 0 such that:
W(скЕ-f- (1 — a)y) > aW(a?) + (1 — a)W(y) — Ck(x — y)2 for every x,y€K
and a € [0,1]. Is the value function semi-concave?
7.7.	Recall Example 7.5 and consider the open region 1? and the synthesis
represented in Figure 7.7. The open region L? does not contain the point
Fig. 7.7. BB synthesis on a region Г2.
B, while the synthesis is the same as the optimal ones except for points on
the left of К, for which we take the trajectories 71 instead of 72 (defined in
Example 7.5.) We can define a candidate value function W computing the
164	7 Sufficient Conditions
time along the trajectories of such synthesis. Prove that all assumptions
of Theorem 7.3.3 are verified except iii).
8
Viscosity solutions for Hamilton-Jacobi
equations
Aim of this Chapter is to provide a concise introduction to the theory of vis-
cosity solutions for first order nonlinear PDEs, and illustrate its applications
to problems of optimal control.
In the first section we review the classical method of characteristics, to
construct solutions of the first order P.D.E.
F(x, u, Viz) = 0 x G Q C IRn .	(8.1)
In general, the local smooth solutions obtained by this technique cannot be
extended globally to the entire domain f2. Indeed, when two or more charac-
teristic curves meet at a same point, a singularity occurs.
In a typical situation, a boundary value problem for (8.1) will thus have no
global smooth solutions. On the other hand, it may well have infinitely many
piecewise smooth solutions, which satisfy the equation at almost every point
of the domain. We then face the question of how to single out a unique “good”
solution, relevant for whatever application we may have in mind. An answer
is provided by the theory of viscosity solutions, introduced by Crandall and
Lions in [34]. In essence, the main results show that
•	Letting e ОТ, the solutions ue(-) to the parabolic problems
F(x, ue, Vue) = e Au£
converge to a unique limit zz(-).
•	This limit function и can be uniquely characterized by imposing certain
inequalities on its upper and lower differentials, at each point x G L? where
they exist.
In Sections 8.2 and 8.3 we discuss some definitions and properties of super-
and sub-differentials, and introduce the notion of upper and lower viscosity so-
lution. The stability of viscosity solutions w.r.t. uniform convergence is proved
166	8 Viscosity solutions for Hamilton-Jacobi equations
in Section 8.4. A basic comparison theorem, between an upper and a lower
viscosity solution, is proved in Section 8.5. In turn, this yields the uniqueness
of solutions in the viscosity sense.
Applications to control systems are worked out in the last three sections.
Namely, we characterize the value function for an optimal control problem
as the unique viscosity solution to the corresponding first order P.D.E. prob-
lem. This provides an alternative approach to the construction of optimal
trajectories and to the study of sufficient conditions for optimality.
In addition to [35], and [36], for a comprehensive monograph on the subject
we refer to [8].
8.1 The method of characteristics
Consider a first order, scalar P.D.E., having the general form (8.1). It is conve-
nient to introduce the variable p = Vn, so that (pi,... ,pn) = (uX1,..., uXn).
Throughout the following, we assume that the F — F(x,u,p) is a continuous
function, mapping IRn x IR x IR" into IR.
Given the boundary data
u(x) = й(х) x e 312,	(8.2)
a solution can be constructed (at least locally, in a neighborhood of the bound-
ary) by the classical method of characteristics. The idea is to obtain the values
u(x) along curves s h-> j;(s) starting from the boundary of 12, solving a suitable
O.D.E. (see figure 8.1).
Fig. 8.1. The method of characteristics.
Fix a point у E 312 and consider an arbitrary differentiable curve s i—> x(s)
with x’(0) = y. Call
u(s) = u(x(s)),	p(s) = p(x(sY) = Vu(x(s)).
We seek an O.D.E. describing the evolution of и and p = Vu along the curve.
Denoting by an upper dot the derivative w.r.t. the parameter s, we clearly
have
8.1 The method of characteristics 167
Pj — 'U'XjXi •
(8.3)
In general, pj thus depends on the second derivatives of u, which at this stage
are not available. Differentiating the basic equation (8.1) w.r.t. Xj we obtain
dF dF
dxj + du Uxj
= 0.
Hence
dF	dF	dF
dpi XjX' dxj	du ^J
(8.4)
Instead of taking an arbitrary curve, we now choose a curve such that Xi =
dF/dpi. By this specific choice, the right hand side of (8.3) is computed by
(8.4), and can thus be expressed in terms of the variables x,u,p. We thus
obtain a closed system of n+l+n equations where the second order derivatives
uXiX. do not appear:
( +. - dF.
1 ~ дрг
< й = Ё.гргт£-
p 9F-. 9Fp
dxj du
i = 1, . . . , П
j = 1,... ,n.
(8-5)
This leads to a family of Cauchy problems, which in vector notation take the
form
i=f^
dp
й = p-
op
•n =	_ dF
* dx du
P
' x(0) = у
< u(0) = u(p)
p(0) = Vu(j)
yedn. (8.6)
The resolution of the first order boundary value problem (8.1)-(8.2) is thus
reduced to the solution of a family of O.D.E’s, depending on the initial point
y. As у varies along the boundary of 12, we expect that the union of the above
curves x(-) will cover a neighborhood of Э12, where our solution и will be
defined.
Remark 8.1. If F is linear w.r.t. p, then the derivatives dF/dpt do not depend
on p. Therefore, the first two equations in (8.5) can be solved independently,
without computing p from the third equation.
Example 8.1. The equation
|V-u|2 -1 = 0 x e n
on R2 corresponds to (8.1) with F(x,u,p) = p2 + p| — 1. Assigning the
boundary data
168	8 Viscosity solutions for Hamilton-Jacobi equations
и = 0 x € dfi,
a solution is provided by the distance function
u(x) — dist (ж, dii).
The corresponding equations (8.6) are
x = 2p, и — p • x — 2 ,
p = 0.
Choosing the initial data at a point у we have
.r(0) = 7/,	u(0) = 0,	p(0) — n ,
where n is the interior unit normal to the set f? at the point y. In this case,
the solution is constructed along the ray x(s) = у + 2sn, and along this ray
one has u(.x) = |rr - y\. Assuming that the boundary dii is smooth, in general
the distance function will be smooth only on a neighborhood of this boundary.
If J? is bounded, there will certainly be a set 7 of interior points x where the
distance function is not differentiable (fig. 8.2). These are indeed the points
such that
dist (.r, dQ) = |x — 7/11 = |.t — 7/21
for two distinct points 7/1,7/2 € dii.
Fig. 8.2. Singularities of the distance function.
The previous example shows that, in general, the boundary value problem
for a first order P.D.E. does not admit a global C1 solution. This suggests
that we should relax our requirements, and consider solutions in a generalized
sense. We recall that, by Rademacher’s theorem, every Lipschitz continuous
function и : ii 1—> IR is differentiable almost everywhere. It thus seems natural
to introduce a concept of generalized solutions. A function и is a generalized
solution of (8.1)-(8.2) if и is Lipschitz continuous on the closure f?, takes
8.1 The method of characteristics 169
the prescribed boundary values and satisfies the first order equation (8.1) at
almost every point x G f?.
Unfortunately, this concept of solution is far too weak, and does not lead to
any useful uniqueness result. Recall Example 7.2. There exist infinitely many
generalized solutions to the equation:
x E [0,1],
rr(O) = #(!) = 0,
see figure 8.3 (left).
Fig. 8.3. Left: Infinitely many generalized solutions. Right: a solution and a smooth
approximation.
Therefore, one seeks a new concept of solution for the first order equation
(8.1), having the following properties:
1.	For every boundary data (8.2), a unique solution exists, depending contin-
uously on the boundary values and on the function F.
2.	This solution и coincides with the limit of vanishing viscosity approxima-
tions. Namely, и =	ue, where the ue are solutions of
F(x, ue, Vue) = s Au£ .
3.	In the case where (8.1) is the Hamilton-Jacobi equation for the value func-
tion of some optimization problem, our concept of solution should single
out precisely this value function.
In connection with Example 8.1, we see that the distance function
if xe [0, 1/2],
if xe [1/2, 1],
is the only one, among those shown in figure 8.3 (left), that can be obtained
as a vanishing viscosity limit. Indeed, any other generalized solution и with
polygonal graph has at least one strict local minimum in the interior of the
interval [0,1], say at a point x. If u£ —► и uniformly on [0,1], for some sequence
of smooth solutions to
|14| — 1 = euxx ,
170	8 Viscosity solutions for Hamilton-Jacobi equations
then each u£ will have a local minimum at a nearby point x£, as shown in
figure 8.3 (right). But this is impossible, because
|«x(^e)| - 1 = “I /	> °-
In the following sections we shall introduce the definition of viscosity solution
and see how it fulfils the above requirements.
8.2 One-sided differentials
Let u : J? »—> ]R be a scalar function, defined on an open set 12 C IRn. The set
of super-differentials of и at a point x is defined as
D+«(x) = LelR"; limsup ~ "W ~	~	< o) .
I	y->x	\y ~	J
In other words, a vector p e IR" is a super-differential iff the plane у i—►
u(x) +p-(y — x) is tangent from above to the graph of и at the point x (fig. 8.4
(left)). Similarly, the set of sub-differentials of и at a point x is defined as
D~u(x) = pE IR";
lin[in{Ufa)-«W-P-(»-»)
У-+Х	\y — a? I
so that a vector p 6 IR" is a sub-differential iff the plane у нч- u(x) + p- (y — x)
is tangent from below to the graph of и at the point x (fig. 8.4 (right)).
Fig. 8.4. Super and sub-differentials.
Example 8.2. Consider the function (fig. 8.5)
u(x) =
if
if
if
x < 0,
xe [0,1],
X > I.
In this case we have
= 0,
D tt(O) = [0, oo[,
8.2 One-sided differentials
171
Fig. 8.5. Example of super and sub-differentials.
D+u(x) = D u(x') = {1/2\/t}	rr G]0,1[,
D+u(l) = [0, 1/2],	p-tz(l) =0.
If tp e C1, its differential at a point x is written as V<p(x). The following
characterization of super- and sub-differentials is very useful.
Lemma 8.2.1. Let и e C(J?). Then
(i) p e D+u(x) if and only if there exists a function tp 6 C1(f?) such that
Vcp(z) = p and и — (p has a local maximum at x.
(ii) p € D~u(x) if and only if there exists a function ip e such that
\7(p(x) = p and и — <p has a local minimum at x.
By adding a constant, it is not restrictive to assume that <p(x) = u(x). In this
case, we are saying that p E D+u(x) iff there exists a smooth function ip > и
with V<p(rr) = p, <p(x) = u(x). In other words, the graph of <p touches the
graph of и from above at the point x (fig. 8.6 (left)). A similar property holds
for subdifferentials: p G D~u(x) iff there exists a smooth function < ?/, with
V<p(x) = p, whose graph touches from below the graph of и at the point x.
(fig. 8.6 (right)).
Fig. 8.6. Characterization of super and sub-differentials.
Proof of Lemma 8.2.1. Assume that p E D+u(x). Then we can find 5 > 0 and
a continuous, increasing function ст : [0, оо[ь-> IR, with cr(0) = 0, such that
u(y) < u(x) + p  (y - x) + a(\y - ar|)
172	8 Viscosity solutions for Hamilton-Jacobi equations
for \y — ж| < 6. Define
p(r) = f ff(t) dt
Jo
and observe that
p(0) = pz(0) — 0,	p(2r) > a(r)r.
By the above properties, the function
4>(у) = «(or) + p • (y - x) + p(2|y - x|)
is in and satisfies
<^(x) = tt(rr),	= p.
Moreover, for \y — <t| < 6 we have
w(y) - V’(y) <	- ж|) |j/ - x| - p(2|-y - x|) < 0.
Hence, the difference и — ip attains a local maximum at the point x.
To prove the opposite implication, assume that D<p(x) — p and и — <p has
a local maximum at x. Then
Ito Supа(э)-„(х)-р.(,-х) s Um y(jO-y(»)-p-(!>-x) _ 0
y^x	y^x	|г/-ж|
This completes the proof of (i). The proof of (ii) is entirely similar.
□
Remark 8.2. By possibly replacing the function ip with <p(y) = <p(y)±|?/—x|2,
it is clear that in the above lemma we can require that и — ip attains a strict
local maximum or local minimum at the point x. This is particularly important
in view of the following stability result.
Lemma 8.2.2. Let и : L? > IR. be continuous. Assume that, for some ф € C1,
the function и — ф has a strict local minimum (a strict local maximum) at
a point x € fl. If um —> и uniformly, then there exists a sequence of points
xm —► x with um(xTn) —> u(x) and such that um — ф has a local minimum (a
local maximum) at xm.
Proof. Assume that и — ф has a strict local minimum at x. For every p > 0
sufficiently small, there exists ep > 0 such that
u(y) — ф(у) > u(x) - 0(x) + ep whenever \y - a;| = p .
By the uniform convergence um —> u, for all m > Np sufficiently large one has
um(y) - u(y) < ep/^ for \y - x| < p. Hence
8.2 One-sided differentials 173
«m(У) - 0(У) > Um(x) - 0(x) + y
1У - *1 =P,
This shows that ит—ф has a local minimum at some point with |жт— x\ <
p. Letting p,ep —> 0, we construct the desired sequence {а?т}.
This situation is illustrated in fig. 8.7 (left). On the other hand, if x is a point
of non-strict local minimum for и — 0, the slightly perturbed function um — ф
may not have any local minimum xm close to a?, see fig. 8.7 (right).
Fig. 8.7. Convergence of strict local minima.
Some simple properties of super- and sub-differentials are collected in the
next lemma.
Lemma 8.2.3. Let и G C(J2). Then
(i) If и is differentiable at x, then
D+u(x) = D u(x) = {Vu(a?)} .
(8-7)
(ii) If the sets D+u(x) and D~u(x) are both non-empty, then и is differen-
tiable at x, hence (8.7) holds.
(iii) The sets of points where a one-sided differential exists:
P+ = {zGJ2; P+u(x)^0}, Q- = {xeQ; D~u(x) ± 0}
are both non-empty. Indeed, they are dense in 12.
Proof. Concerning (i), assume и is differentiable at x. Trivially, Vu(x) G
D±u(x). On the other hand, if p G C1(I2) is such that и — p has a local
maximum at x, then V<p(rr) = Vu(e). Hence D+u(x) cannot contain any
vector other than Vu(rc).
To prove (ii), assume that the sets D+u(x) and D~u(x) are both non-
empty. Then there we can find 6 > 0 and <^i. p2 C C^J?) such that (fig. 8.8
(left))
<£l(z) = u(x) = p2(x\
^1(Z/) < tz(i/) < p2(y) \y - z| < S.
174	8 Viscosity solutions for Hamilton-Jacobi equations
By a standard comparison argument, this implies that и is differentiable at x
and Vu(x) = V^i(x) = V<p2(#)-
To prove (iii), consider any open ball B(xo,p) C j? and define the smooth
function (fig. 8.8 (right))
, . . 1
PW = ~2------1-------12
P2 - F - ZO|2
Since —> +oo as |rr—a?o | P, the the continuous function u—p attains a local
maximum at some interior point у 6 B(xo.p). By Lemma 8.2.1, the super-
differential of и at у is non-empty. Indeed, \7<р(у) E D+u(y). The previous
argument shows that, for every Xq E ii and p > 0, the set has non-empty
intersection with the ball Z?(xo,p). Therefore is dense in P. The case of
sub-differentials is entirely similar.
Fig. 8.8. Left: a function и having an super- and a sub-differential at a given point
x is differentiable. Right: pushing down the graph of until it touches the graph of
u, one finds a point у where D+u is non-empty.
8.3 Viscosity solutions
In the following we consider the first order partial differential equation
F(x, u(x), Уф)) = 0
(8.8)
defined on an open set Г2 G IRn. Here F : £? x IR x IRn IR is a continuous
(nonlinear) function.
A function и E C(<2) is a viscosity subsolution of (8.8) if
F(.r, u(x),p) < 0 for every x E 12, p E D+u(x).
Similarly, и E C(L2) is a viscosity supersolution of (8.8) if
8.3 Viscosity solutions
175
F(j:, u(x),p) > 0 for every x e /2, p 6 D u(x).
We say that и is a viscosity solution of (8.8) if it is both a supersolution
and a subsolution in the viscosity sense.
Similar definitions also apply to evolution equations of the form
ut + H(t, x, u, Vu) = 0,	(8.9)
where Vu denotes the gradient of и w.r.t. x. Recalling Lemma 8.2.1, we can
reformulate these definitions in an equivalent form:
A function и G С(Г2) is a viscosity subsolution of (8.9) if, for every C]
function <p = <p(t,x) such that и — <p has a local maximum at (£,x), there
holds
<Pt(t, x) + H(t, x, u, V<p) < 0.
Similarly, и G C(f2) is a viscosity supersolution of (8.9) if, for every C1
function ip = <p(t,x) such that и — p has a local minimum at (t,x), there
holds
<pt(t,x) T H(t,x,u, V92) > 0.
Remark 8.3. In the definition of subsohition, we are imposing conditions
on и only at points x where the super-differential is non-empty. Even if и
is merely continuous and nowhere differentiable, there are infinitely many of
these points. Indeed, by Lemma 8.2.3, the set of points x where D+u(x) 0 is
dense on <2. Similarly, for supersolutions we impose conditions only at points
where D~u(x] 0.
Remark 8.4 If и is a C1 function that satisfies (8.8) at every x 6 12, then и
is also a solution in the viscosity sense. Viceversa, if и is a viscosity solution,
then the equality (8.8) must hold at every point x where и is differentiable.
In particular, if и is Lipschitz continuous, then by Rademacher’s theorem it
is a.e. differentiable. Hence (8.8) holds a.e. in Г2.
Example 8.3. Set F(x,t4,nx) = 1 — |ux|. Then the function u(x) = |#| is a
viscosity solution of
1 —|nx|=0	(8.10)
defined on the whole real line. Indeed, и is differentiable and satisfies the
equation (8.10) at all points x 0. Moreover, we have
P+w(0) = 0,	F"u(0) = [-1, 1].
To show that и is a subsolution, there is nothing else to check. To show that
и is a supersolution, take any p G [—1, 1]. Then 1 — |p| > 0, as required.
It is interesting to observe that the same function u(x) = |ж| is NOT a viscosity
solution of the equation
|tzx|-l = 0.	(8.11)
Indeed, at x = 0, taking p = 0 e	we find |0| -1 < 0. In conclusion, the
function u(x) = |ж| is a viscosity subsolution of (8.11), but not a supersolution.
176
8 Viscosity solutions for Hainilton-Jacobi equations
8.4 Stability properties
For nonlinear P.D.E’s, the set of solutions may not be closed w.r.t. the topol-
ogy of uniform convergence. In general, if un —> и uniformly on a domain
Г2, to conclude that и is itself a solution of the P.D.E. one should know, in
addition, that all the derivatives Daun that appear in the equation converge
to the corresponding derivatives of u. This may not be the case in general.
Example 8.4. A sequence of solutions to the equation
|ux| -1=0,
u(0) = ?z(l) = 0
(8.12)
is provided by the saw-tooth functions (fig. 8.9)
Clearly um —> 0 uniformly on [0,1], but the zero function is not a solution of
(8.12). In this case, the convergence of the functions un is not accompanied
by the convergence of their derivatives.
Fig. 8.9. A sequence of saw-tooth functions.
The next lemma shows that the uniform limit of viscosity solutions is itself
a viscosity solution. Quite remarkably, nothing at all is assumed here about
the convergence of derivatives.
Lemma 8.4.1. Consider a sequence of continuous functions um, which pro-
vide viscosity sub-solutions (super-solutions) to
^Wrn) — 0
X E •
Asm —> oo, assume that Fm —> F uniformly on compact subsets of J?xIRxIRn
and um —► и in C(f2). Then и is a subsolution (a supersolution) of (8.8).
Proof. To prove that и is a subsolution, let ф 6 Cl be such that и — ф has a
strict local maximum at a point x. We need to show that
F(x,</>(jr), V0(a:)) < 0.
(8.14)
By Lemma 8.2.2, there exists a sequence —► x such that um — ф has a local
maximum at xw, and um(a?m) —♦ u(x) as m —► oo. Since um is a subsolution,
8.4 Stability properties 177
(8.15)
Taking the limit in (8.15) as m —> oo, we obtain (8.14).
The above result should be compared with Example 8.4. Clearly, the func-
tions un in (8.13) are not viscosity solutions.
The definition of viscosity solution is naturally motivated by the properties
of vanishing viscosity limits.
Theorem 8.4.2. Let u£ be a family of smooth solutions to the viscous equa-
tion
F(x, ue(x\	= e Au£ .	(8.16)
Assume that, as e —> 0+, we have the convergence u£ —> и uniformly on an
open set ft C HU1. Then и is a viscosity solution of (8.8).
Proof. Fix x e ft and assume p E D+u(x). To prove that и is a subsolution
we need to show that F(x, u(x), p) < 0.
1.	By Lemma 8.2.1 and Remark 8.2, there exists <p E C1 with V</?(x) = p,
such that и — <p has a strict local maximum at x. For any S > 0 we can then
find 0 < p < 8 and a function ф E C2 such that
|V^)-V^)|<<5 if \y-x\<p,	(8.17)
<<5	(8.18)
and such that each function u£ — ф has a local maximum inside the ball
В(х; /?), for e > 0 small enough.
2.	Let x£ be the location of this local maximum of u£ - ф. Since u£ is smooth,
this implies
\7ф(х£) = Vu(r£), Ди(х£) < Аф(х£),
hence from (8.16) it follows
F(x,u£(xe), Wfe)) < e Аф(х£).	(8.19)
3.	Extract a convergent subsequence x£ —> x. Clearly |i—a?| < p. Since ф E C2.
we can pass to the limit in (8.19) and conclude
F(x, u(x), V^(i)) < 0	(8.20)
By (8.17)-(8.18) we have
|V^(i) — p\ < |VV>(£) — Vy?(^)| 4- |V<p(ai) - V^(x)| < 8 + 8.
Since 8 > 0 can be taken arbitrarily small, (8.20) and the continuity of F
imply F(x, u(x),p) < 0, showing that и is a subsolution. The fact that и is a
supersolution is proved in an entirely similar way.
178	8 Viscosity solutions for Hamilton-Jacobi equations
8.5 Comparison theorems
A remarkable feature of the notion of viscosity solutions is that on one hand
it requires a minimum amount of regularity (just continuity), and on the
other hand it is stringent enough to yield general comparison and uniqueness
theorems.
The uniqueness proofs are based on a technique of doubling of variables,
which reminds of Kruzhkov’s uniqueness theorem for conservation laws [60].
We now illustrate this basic technique in a simple setting.
Theorem 8.5.1. (Comparison). Let 12 C IR71 be a bounded open set. Let
U\,U2 € C(12) be, respectively, viscosity sub- and supersolutions of
и + II(x, Vu) = 0	x € 12.
Assume that
Ui(rr) < U2(x)	for all x 6 dS2.
Moreover, assume that H : 12 x IRn i—> IR is uniformly continuous in the
x-variable:
\H(x,p) - H(y,p)\ <w(|x-j/|(H-|p|)),	(8.21)
for some continuous and non-decreasing function uj : [0, оо[ь-► [0, oo[ with
cu(0) = 0. Then
ui(x) < U2 (x)	for all ж e 12.	(8.22)
Proof. To appreciate the main idea of the proof, consider first the case where
ui,U2 are smooth. If the conclusion (8.22) fails, then the difference Ui — U2
attains a positive maximum at a point 6 12. This implies p = Vui(rro) =
Vu2(^o)- By definition of sub- and supersolution, we now have
ui(x0) + H(x0,p) < o,	.	.
U2(x0) + H(x0,p) >0.	1 (ii)
Subtracting the second from the first inequality in (8.23) we conclude щ (a?o) ~
^2(^0) < 0, reaching a contradiction.
Next, consider the non-smooth case. We can repeat the above argument
and reach again a contradiction provided that we can find a point xq such
that (fig. 8.10 (left))
(i) ui(rro) > ^2(^0),
(ii) some vector p lies at the same time in the upper differential D+u1(xq)
and in the lower differential D~U2(xo\
A natural candidate for tq is a point where Ui —U2 attains a global maximum.
Unfortunately, at such point one of the sets D+u\(xq) or D~U2(xq) may be
empty, and the argument breaks down (fig. 8.10 (right)). To proceed further,
the key observation is that we don’t need to compare values of щ and U2 at
exactly the same point. Indeed, to reach a contradiction, it suffices to find
nearby points xe and y£ such that (see fig. 8.11)
8.5 Comparison theorems
179
Fig. 8.10. Comparison of viscosity solutions.
(f) Ui(z£) > U2(t/e),
(ii’) some vector p lies at the same time in the upper differential D+ui(x£)
and in the lower differential D~U2(yeY
Fig. 8.11. Geometric motivation for the proof of Theorem 8.5.1.
Can we always find such points? It is here that the variable-doubling tech-
nique comes in. The trick is to look at the function of two variables
&e(x,y) = Ui(x) - U2(y) -
(8.24)
This clearly admits a global maximum over the compact set ft x J?. If щ > U2
at some point xo, this maximum will be strictly positive. Moreover, taking
e > 0 sufficiently small, the boundary conditions imply that the maximum is
attained at some interior point (xe,ye) 6 ft x Г2. Notice that the points x£, y£
must be close to each other, otherwise the penalization term in (8.24) will be
very large and negative.
We now observe that the function of a single variable
180	8 Viscosity solutions for Hamilton-Jacobi equations
X b-> Ui(x) - ( и2(?/г) + ——— } = Ul(rr) - </?i (x)	(8.25)
attains its maximum at the point x£. Hence by Lemma 8.2.1
——— = V<^i(xe) e £)+uj(xe).
£
Moreover, the function of a single variable
/	I £   у | 2 \
У ” U2(y) - I tii(Xe) - ——-------- = U2(l/) - ^(у)	(8.26)
\ /
attains its minimum at the point y£. Hence
e D~U2(ye).
We have thus discovered two points x£, yE and a vector p = (xe — у£)/e which
satisfy the conditions (i’)-(ii’).
We now work out the details of the proof, in several steps.
1.	If the conclusion fails, then there exists Xq € 42 such that
ui(x0) _ «2(^0) = max {tzi(x) - u2(x)} = 6 > 0.	(8.27)
For £ > 0, call (x£,y£) a point where the function Ф£ in (8.24) attains its
global maximum on the compact set 42 x 42. By (8.27) one has
&e(x£,yE) > 6 > 0.	(8.28)
2.	Call M an upper bound for all values |ui(x)|,	|- as x e 42. Then
la? — vl2
Ф£(х,т/)<2М-Ц-^,
Z£
Фе(х,у) <0 if \x — y\2>M£.
Hence (8.28) implies
\x£ - y£\ < \/M£ .	(8.29)
3.	By the uniform continuity of the functions u2 on the compact set 42, for
£f > 0 sufficiently small we have
|w2(x) - li2(y)|
whenever |x — y\ < VM£f.
(8.30)
We now show that, choosing £ < £r, the points xe, y£ cannot lie on the bound-
ary of 42. For example, if x£ e dQ. then by (8.29) and (8.30)
8.5 Comparison theorems
181
Фг(хе,у£) < (ui(a:e)-u2(xe)) + |игке) -«г(?/г)| - ~	< 0 + 5/2 + 0,
against (8.28).
4.	Having shown that xe^ye are interior points, we consider the functions of
one single variable <pi,(p2 defined at (8.25)-(8.26). Since xe provides a local
maximum for щ —(pi and y€ provides a local minimum for U2 — <£2. we conclude
that
p£ =	~~ 6	О В~Ы2(Уе)-
From the definition of viscosity sub- and supersolution we now obtain
UiM +	< 0.
u2(&) + /%,p£) >0.
5.	Observing that
Ui(a:£)-U2(Ze) <Фг{х£,у£) < щ(хе)-и2(хе) + \и2(х£)-и2(Уе)\
(8.31)
ke ~ tfel2
2г
by (8.27) we see that
|w2(^e) - U2(ye)|
ke -&|2
2e
> 0.
Hence, by the uniform continuity of U2,
4^^°
2s
(8.32)
as s —> 0.
6.	Recalling (8.28) and subtracting the second from the first inequality in
(8.31) we obtain
6 < Фе(х£,Уе) < tZi(2?£) - U2(ye)
< \H(xe,p) - H(ye,p)\ < w((|are - ye\  (1 + ke - ye|e-1)).
(8.33)
This yields a contradiction, Indeed, by (8.21) and (8.32) the right hand side
of (8.33) becomes arbitrarily small as s —► 0.
An easy consequence of the above result is the following uniqueness result
for the boundary value problem
и + H(x, Vu) = 0 x e P,
(8.34)
U =
x e <ЭР.
(8.35)
182	8 Viscosity solutions for Hamilton-Jacobi equations
Corollary 8.5.2. (Uniqueness). Let Pl C IRn be a bounded open set. Let the
Hamiltonian function H satisfy the equicontinuity assumption (8.21). Then
the boundary value problem (8.34)-(8.35) admits at most one viscosity solu-
tion.
Proof. Let be viscosity solutions. Since iq is a subsolution and U2 is a
supersolution, and щ = U2 on <ЭР, by Theorem 1 we conclude ui < U2 on 12.
Reversing the roles of iq and /q* we deduce U2 < iq, completing the proof.
By similar techniques, comparison and uniqueness results can be proved
also for Hamilton-Jacobi equations of evolutionary type. Consider the Cauchy
problem
utTH(Lx,Vu) =0	(Lj:) e]0,T[xIRn, (8.36)
u(0, x) = й(х) хеПп.	(8.37)
Here and in the sequel, it is understood that Vu = (uXi,..., uXn) always refers
to the gradient of и w.r.t. the space variables.
Theorem 8.5.3. (Comparison). Let the function H : [0, T] x IR” x IR”
satisfy the Lipschitz continuity assumptions
\H(t,x,p) - H(s,y,p)\ < C(|i - s| + |x - y|) (1 + |p|),	(8.38)
\H(t,x,p)-H(t,x,q)\ < C\p — q\.	(8.39)
Let u,v be bounded, uniformly continuous sub- and super-solutions of (8.36)
respectively. Ifu(f),x) < v(0, x) for all x e lRn, then
u(t,x) < v(t,x)	for all (t,x) e [0, T] x IR”. (8.40)
Toward this result, as a preliminary we prove
Lemma 8.5.4. Let и be a continuous function on [0, T] x IR”, which provides
a subsolution of (8.36) for t e]0,T[. If ф G Cl is such that и — ф attains a
local maximum at a point (T, Xq), then
<j>t(T,x0) + H(T,x0,V</>(T,x0)) <0.	(8.41)
Proof. We can assume that (T, x'o) is a point of strict local maximum for и — ф.
For each e > 0 consider the function
Each function и — ф£ will then have a local maximum at a point (t£,x£), with
t£<T,	(te,x£) -> (T,x0) as e -> 0 T .
Since и is a subsolution, one has
0e.t(te,xe) + H(^,a:e.V0e(te,a:e)) <	•	(8-42)
Letting e —> 0T, from (8.42) we obtain (8.41).
8.5 Comparison theorems
183
Proof of Theorem 8.5.3.
1.	If (8.40) fails, then we can find A > 0 such that
sup < u(t, x) — v(t, a?) — 2AO = a > 0.
t,X t	J
(8.43)
Assume that the supremum in (8.43) is actually attained at a point (to?^o),
possibly with to = T. If both и and и are differentiable at such point, we
easily obtain a contradiction, because
uf(fo,^o) +	Vu) < 0,
vt(to^o) +	Vv) >0,
Vu(to,^o) = Vr(Zo^o),
ut(to,xo) - vt(t0, xq) - 2A > 0.
2.	To extend the above argument to the general case, we face two technical
difficulties. First, the function in (8.43) may not attain its global maximum
over the unbounded set [0, T] x Moreover, at this point of maximum
the functions u, v may not be differentiable. These problems are overcome by
inserting a penalization term, and doubling the variables. As in the proof of
Theorem 8.5.1, we introduce the function
Ф£(1,х,з,у) = u(t,x)-v(s,y)-A(t+s)-e(|a;|24-|j/|2)-^(|t-s|2 + |2:-y|2) .
Thanks to the penalization terms, the function Ф£ clearly admits a global max-
imum at a point (t£, x£,	y£) € (]0, T] x IRn) . Choosing e > 0 sufficiently
small, one has
<P£(t£,x£,s£.y£) > тахФ£^,х^,х) > a/2 .
t,x
3.	We now observe that the function
(t,x) u(t, x) —
[«(se,tfe) + A(t + .sj +c(|.r|2 + Ы2) + Fqt _ <J£|2 + |x -
= u(t, x) — ф(1,х)
takes a maximum at the point (t£^x£). Since и is a subsolution and ф is
smooth, this implies
A + 2(*e /е) + Я (t£, xe, 2(*£ 2	+ 2£aA < 0.	(8.44)
Notice that, in the case where t£ = T, (8.44) follows from Lemma 8.5.4.
Similarly, the function
184	8 Viscosity solutions for Hamilton-Jacobi equations
(s,y) v(s,y) -
[u(te,xe) - A(ie + s) - e(|®e|2 + |a|2) - ^j(l*e - «I2 + l®e - 3/|2)]
= v(s,y) -il>(s,y)
takes a maximum at the point (t£,x£). Since v is a supersolution and ф is
smooth, this implies
. 2fa-se)
О
+ н s£, y£,
Уе) n A > n
-----2-------2ey£ 1 > 0.
£z-----------J
(8.45)
4.	Subtracting (8.45) from (8.44) and using (8.38)-(8.39) we obtain
2A < H (se, ye,	- 2ey£') - H (te, xe,	+ 2sxe)
< Co(|l-£| + lifel) + C(\t£ - se| + |are - ad) (1 +	+ c(|zd + |ad))
(8.46)
To reach a contradiction we need to show that the right hand side of (8.46)
approaches zero as e —> 0.
5.	Since u, v are globally bounded, the penalization terms must satisfy uniform
bounds, independent of e. Hence
kel, lad < -^= • |ie - sd, !•'=- - ad < c'e. (8.47)
for some constant C". This implies
е(|х£| + |Уе|) <2CZ4/i.	(8.48)
To obtain a sharper estimate, we now observe that
Ф£ (££, ;r£, se, ?/£) > Ф£ (Ze, же, t£, )T£),
hence
u{t£.xe) - v(se,y£) - X(t£ + se) - E(|a?e|2 + lad2)
“^2 (I*® “ S®|2 + Iх® “ ^®|2)
> u(t£,x£) — v(t£1xe) — 2Xt£ — 2e|:re|2,
72 (1^ -»d2 + l^- ad2) < v(ie,a?e)-v(se,ae) + A(te-Se)+E(|a:e|2 - ladT
(8.49)
By the uniform continuity of v, the right hand side of (8.49) tends to zero as
e —> 0, therefore
8.6 Dynamic programming (revisited)
185
Re ~ «el2 + l^e ~ &|2
£2
as e —> 0.
(8.50)
By (8.47), (8.48) and (8.50), the right hand side of (8.46) also approaches
zero, This yields the desired contradiction.
□
Corollary 8.5.5. (Uniqueness). Let the function H satisfy the assumptions
(8.38)-(8.39). Then the Cauchy problem (8.36)-(8.37) admits at most one
bounded, uniformly continuous viscosity solution и: [0. T] x IR" > IR.
8.6 Dynamic programming (revisited)
Consider again a control system of the form
x(t) = f(x(t),	e U.	(8.51)
We now assume that the set U C IRW of admissible control values is compact,
while f : IRn x U Hl" is a continuous function such that for all x. у E IR"
and и E U
|f(x,u)|<C,	\f(x,u)~ f(y,u)\<C\x-y\	(8.52)
for some constant C. Given an initial data
ar(s) = T/eIR",	(8.53)
under the assumptions (8.52), for every choice of the measurable control func-
tion zz(-) € U the Cauchy problem (8.51)-(8.53) has a unique solution, which
we denote as t i—> x(t; s, y, u) or sometimes simply as t x(t). We seek an
admissible control function u* : [s, T] »—> U. which minimizes the sum of a
running and a terminal cost
J(s,y,u) = У h(x(t), u(t)) dt + д(х(Т)У	(8.54)
Here it is understood that x(t) = z(t; s, y, u), while
h:IRn xUh+IR,	g : IR" i—> IR
are continuous functions. We shall assume that the functions h.g satisfy the
bounds
|/i(z,u)| <G	\g(x)\ <C,	(8.55)
\h(x, u) - h(3/,u)| < C\x-y\,	\g(x) -^(t/)| <	(8.56)
for all x, у E IR", и E U. As in the previous sections, we call
186	8 Viscosity solutions for Hamilton-Jacobi equations
Z7 =	: IR i—> IRrn measurable, u(t) G U for a.e. 1j*
the family of admissible control functions. According to the method of dy-
namic programming, an optimal control problem can be studied by looking
at the value function:
V(s^y)= inf J(s,y,u).
(8.57)
We consider here a whole family of optimal control problem, all with the same
dynamics (8.51) and cost functional (8.54). The main interest is on how the
minimum cost varies, as a function of the initial conditions (8.53). Indeed,
we will show that the value function V can be characterized as the unique
viscosity solution to a Hamilton-Jacobi equation. Toward this goal, a basic
step is provided by Bellman’s principle of dynamic programming.
Fig. 8.12. Dynamic Programming Principle.
Theorem 8.6.1. (Dynamic Programming Principle). For every r e
[s,T] and у G IR", one has
V(s, t/) = inf < / s, y, u), u(t)) dt T V(т, <г(т; s, ?/,))
(8.58)
In other words (fig. 8.12), the optimization problem on the time interval [.$•, T]
can be split into two separate problems:
•	As a first step, we solve the optimization problem on the sub-interval [т, T],
with running cost h and terminal cost g. In this way, we determine the
value function V(t, •), at time r.
•	As a second step, we solve the optimization problem on the sub-interval
[s, r], with running cost h and terminal cost V(r, •), determined by the
first step.
At the initial time s. by (8.58) we are saying that the value function V(s, •)
obtained in step 2 is the same as the value function corresponding to the
global optimization problem over the whole interval [s,Т].
8.6 Dynamic programming (revisited)
187
Proof. Call Jr the right hand side of (8.58).
1.	To prove that JT < V(s, y), fix s > 0 and choose a control и : [s,T] i—► U
such that
J(s, у, и) < V(s, y) + e.
Observing that
V(t, x(t',s, y, uf) < h(x(t; s, y, u), u(ty) dt + g(x(T; s, y, u)),
we conclude
JT < h(x(t; s, y, u), u(t)) dt + V (т, ж(т; s, y, u))
<J(s,y,u) < V(s,?/) + s.
Since £ > 0 is arbitrary, this first inequality is proved.
2.	To prove that V(s,?/) < JT, fix г > 0. Then there exists a control u' :
[s, t] U such that
У h(x(t, s, y, u'), u(tf) dt + V(т, ж(т; s, ?/, u7)) < JT + £.	(8.59)
Moreover, there exists a control и" : [т, T] i—► U such that
J(t, .t(t; s, ?/, u7), u")<V(t, х(т;з,у,и')) Те. (8.60)
One can now define a new control и : [s,T] »—► A as the concatenation of
u\u”‘.
if £e[s,r],
u(t) “[«"(t) if fe]r,T].
By (8.59) and (8.60) it is now easy to check that
V(s, y) < J(s, y, u) < JT + 2e.
Since e > 0 can be arbitrarily small, this second inequality is also proved.
The next lemma establishes the Lipschitz continuity of the value function.
This property will be later used, to characterize the value function as the
unique viscosity solution to the corresponding Hamilton-Jacobi equation.
Lemma 8.6.2. Let the functions f,g,h satisfy the assumptions (8.52), (8.55)
and (8.56). Then the value function V : [0, T] x IRn i—> R in (8.57) is bounded
and Lipschitz continuous. Namely, there exists a constant C' such that
|V(M|<C',	(8.61)
| V(s,y) - V(s',у')| < C'(|s - s'| + |y - y'\\	(8.62)
188	8 Viscosity solutions for Hamilton-Jacobi equations
Proof. 1. Let s > 0 and у E IR” be given. By (8.55), for every control function
и : [s, T] U, the corresponding cost satisfies
|J(.s.7/. w)| <C(T-.S) + C.
Hence (8.61) holds with C = C(T + 1).
2.	To prove (8.62), let s, у be given. Fix any s > 0 and choose a near-optimal
control function и : [s, T] h-> U such that
J(s, и) < V(s, у) T £.	(8.63)
If we use the same control function и in connection with another initial con-
dition ($.?/'), by (8.52) the corresponding trajectories satisfy
|x(t; .s, y, u) — x(t, s,u)| < ес^~^\у — y'\ .
Hence, from (8.54) and (8.54) we deduce
J(s, y', «) < ./(s. y, u) + ^r(7eC(t-s)|y' - y\ds + Cec(r~s}\y - y'\.
By (8.63), this implies
V(s,y') < V(s.y) + e(C + l)eCT\y — y'\.	(8.64)
Since e > 0 was arbitrary, interchanging the role of y' we conclude that the
value function V is Lipschitz continuous w.r.t. y:
\V(s,y)- V(.M/)| <C0|y-y'|.	(8.65)
with Co = (C + 1) eCT.
3.	To prove the Lipschitz continuity of V w.r.t. the time variable s, let у €
IR” and 0 < s < s' < T be given. Choose a near-optimal control function
и : [s, T] и-> U for which (8.63) holds, and call t x(P) = x(t\s,y,u) the
corresponding trajectory. We now have
V(s', x(.s')) < J(.s, у, и) - Г h(t, x(t)) dt < V(s, y) + e - C(.s' - «).
J s
On the other hand, by the dynamic programming principle,
V(.s, у) < Г h(t, x(t)) dt + V(s', x(.s')) < V(s',x(.s')) - C(.s' - .s).
The uniform bound on f stated in (8.52) implies
|r(S')-x(.s)H|T(.s')-y|<C'|.s'-S|.
8.7 The Hamilton-Jacobi-Bellman equation
189
Using the above inequalities we conclude
IV(s,y) - V(S',y)| < |V(s,y) - У(з',ф'))1 + |V(.<;r(.S')) - V(e',y)|
<С0СУ-з|+С0СК-^|.
This proves the Lipschitz continuity of the value function w.r.t. time, com-
pleting the proof.
8.7 The Hamilton-Jacobi-Bellman equation
The main goal of this section is to characterize the value function as the unique
solution of a first order P.D.E., in the viscosity sense. In turn, this will provide
a sufficient condition for the global optimality of a control function u(-). As
in the previous section, we assume here that the set U is compact and that
the functions fig.h satisfy the bounds (8.52), (8.55) and (8.56).
Theorem 8.7.1. In connection with the control system (8.51), consider the
value function V = V(s,y) defined by (8.57) and (8.5f). ThenV is the unique
viscosity solution of the Hamilton-Jacobi-Bellman equation
+	=0	(t,x) €]0,Т[х1Б'“,	(8.66)
with terminal condition
V(T.x) = g(x) xE Rn,	(8.67)
and Hamiltonian function
H(x, p) = min { f(x, w) • p + hfx, o>)}.	(8.68)
Proof. By Lemma 8.6.2, the value function is bounded and uniformly Lipschitz
continuous on [0, T] x IR". The terminal condition (8.67) is obvious. To show
that V is a viscosity solution, let € C1 (]0, T[ xlR"). Two separate statements
need to be proved:
(Pl)	If V	-	ip	attains a local maximum at a point (to,^o) e]0, T[ xIRn, then
<a(*o,Zo) + nun {/(я70,cu) • V</?(to,^o) + h(x0,w)} > 0.	(8.69)
(P2)	If V	-	tp	attains a local minimum at a point (to,^o) €]0, T[ xlRn, then
+ niin {/(ж0,й?) • V<p(io,^o) + h(xG,w) < 0.	(8.70)
190	8 Viscosity solutions for Hamilton-Jacobi equations
1.	To prove (Pl), we can assume that
V(to-^o) =	V(£,t) < <p(t,x) for all t,x.
If (8.69) does not hold, then there exists cu € U and 0 > 0 such that
9?t(fo,#o) + №o,^) * W(to,£o) + h(xQ,(jj} < -0.	(8.71)
We shall derive a contradiction by showing that this control value ш is “too
good to be true”. Namely, by choosing a control function iz(-) with u(t) = и
for t e [to- t0 + 5] and such that и is nearly optimal on the remaining interval
[to + <5- T], we obtain a total cost J(to,#o,4) strictly smaller than V(to,^o)-
Indeed, by continuity (8.71) implies
(/?t(t, x) +	• V^(L x) <	- 0.	(8.72)
whenever
|t — t0| < <5,	|t- T01 <C8,	(8.73)
for some 8 > 0 small enough and C the constant in (8.52). Let T(t) =
t(Z; to,To,cu) be the solution of
i(t) =/(x(t),u>), rc(f0) = zo,
i.e. the trajectory corresponding to the constant control u(t) = cu. We then
have
V(to + d. x(t0 + <5)) - V(to,x0) < ¥?(to + 5, z(t0 + <*)) - <p(to, x0)
fto+S d
= Jt	dt
/•to+<5 x	x
= I	x(t)) + f(x(t),w) -Vip(t, £(<))} dt
/•<o+<5
< - /	/i(T(t),u)dt-^,	(8.74)
Jto
because of (8.72). On the other hand, the Dynamic Programming Principle
(8.58) yields
/»to+<5
V(t0 + 5, x(t0 + <5)) - V(to,xo) > - h(t, x(t)) dt. (8.75)
v/ to
Toget her. (8.74) and (8.75) yield a contradiction, hence (Pl) must hold.
2.	To prove (P2), we can assume that
V(fo^o) = ^o^o).	V(t,T) > <p(t,x) for all t,x.
8.7 The Hamilton-Jacobi-Bellman equation
191
If (P2) fails, then there exists 0 > 0 such that
Ptfaxo) + /(#o?tu) • V99(^0? ^o) + h(xo,u) > 6 for all uEU. (8.76)
In this case, we shall reach a contradiction by showing that no control function
u(-) is good enough. Namely, whatever control function u(-) we choose on the
initial interval [to? to 4- <5], even if during the remaining time [to + <S? T] our
control is optimal, the total cost will still be considerably larger than V(to, Xq).
Indeed, by continuity, (8.76) implies
<^t(t,x) +	• V<£>(t,x) > 0 — h(x.u) for all w e U, (8.77)
for all t,x close to to,#o? be. such that (8.73) holds. Choose an arbitrary
control function и : [to, to + 5] •—> A, and call t i—> x(t) = x(t,to, xq, u) the
corresponding trajectory. We now have
V(<o + <5, x(to + <5)) - V(t0, x0) > fp(t0 + 8, x(t0 + <9) - <p(to, x0)
fin+6 J
rt()+6
= /	x(t)) + f(x(t),u{t)) • x(t))dt
J to
rto+6
> /	0 — h(x(t), u(t)) dt,
J to
because of (8.77). Therefore, for every control function u(-) we have
/•to+<5
V(t0 + &, x(t0 + 6)) + h(x(t),u(t))dt>V(to,xo) + 89.	(8.78)
J to
Taking the infimum of the left hand side of (8.78) over all control functions
u, we see that this infimum is still > V(to,^o) + On the other hand, by
the Dynamic Programming principle (8.58), this infimum should be exactly
V(to, #o)- This contradiction shows that (P2) must hold, completing the proof.
One can combine Theorems 8.5.3 and 8.7.1, and obtain sufficient conditions
for the optimality of a control function. The usual setting is the following.
Consider the problem of minimizing the cost functional (8.54). Assume that,
for each initial condition (s, y). we can guess a “candidate” optimal control
us'y : [s,T] U. We then call
V(s,y) = J{s,y,us'y)
(8.79)
the corresponding cost. Typically, these control functions us'y are found by
applying the Pontryagin Maximum Principle, which provides a necessary con-
dition for optimality. On the other hand, consider the true value function
192	8 Viscosity solutions for Hamilton-Jacobi equations
V, defined at (8.57) as the infimum of the cost over all admissible control
functions u(-) G Ы. By Theorem 8.7.1, this function V provides a viscos-
ity solution to the Hamilton-Jacobi equation (8.66) with terminal condition
V(T, y) = g(y). If our function V at (8.79) also provides a viscosity solution
to the same equations (8.66)-(8.67), then by the uniqueness of the viscosity
solution stated in Theorem 8.5.3, we can conclude that V = V. Therefore, all
controls us,y are optimal.
8.8 Infinite horizon problems
This section is devoted to infinite horizon problems, where trajectories are
defined for all times t > 0.
Consider again a control system of the form
x = f(x, и) я:(0) = у,	(8.80)
with U C IR"' compact and f : IR " x U IR". For a given measurable control
function ?/(•) e U, we denote by t i—► x(t;y, u) the solution to the Cauchy
problem for (8.80). We seek an admissible control function u* : [0, oo[«-> U,
which minimizes the exponentially discounted cost
J(y, u) = f e~ath(x(t), u(t)) dt.	(8.81)
Jo
where a > 0 and t i—* x(t) = x(t; y. u) denotes the solution of the Cauchy
problem (8.80) corresponding to the control u(-).
To simplify the exposition, we assume that both f and h are uniformly
bounded and Lipschitz continuous, namely
\f(x.u)\<C,	\h(x,u)\<C,	(8.82)
|/(x, u) - f(y, u)\<L\x- y\,	|Л(.т, u) - h(.r, y)\< L\x-y\, (8.83)
for some constants C, L. The above assumptions on f imply that the trajectory
t и-> ;r(t; y. u) is well defined for all times t > 0. Call
U =	: IR IR'" measurable, u(t) G U for a.e.
the family of admissible control functions. Thanks to assumptions (8.82)-
(8.83). the cost J(y,u) is well defined for every admissible control function.
We define the value function by:
V(?;)= inf J(y,u).	(8.84)
u( ) ел/
As for the finite horizon case, one can now prove the uniform continuity
of the value function.
8.8 Infinite horizon problems
193
Lemma 8.8.1. Let the functions f,h satisfy the assumptions (8.82)-(8.83).
Then the value function V in (8.84) bounded and uniformly Holder contin-
uous, namely
|V(y)| <Co, 1Ш-	Vy.y'eIR", (8.85)
for some constants Cq.Ci and 0 < 7 < 1. If a > L, then V is actually
Lipschitz continuous.
Proof 1. Consider any control и : [0, oo[«—► U. The uniform bound (8.82)
yields
|J(t/,tt)I < I e~at\h(x(t,y,u),u)\dt < I Ce~at dt = — .
Jo	Jo	°
This yields the first inequality in (8.85), with Co = C/a.
2.	Consider any two initial values y, y', with \y-y' | < 1. In connection with the
same control function и : [0, oo[«—► U, by (8.83) the corresponding trajectories
satisfy
\x(t,y,u) -x(t,y',u)\ < eLt\y — y'\.
By the Lipschitz continuity of the cost function h. for every time T > 0 we
thus have the estimate
\J(y,u) — J(y',u)\ < /0T e~at\hfr(t,y,u),u) - h(x(t,y', u),u)| dt
-j-	e~at^\h(x(t,y.u),u)\ + \hfx(t,y',u),u)\^ dt
< fy e~Qt • LeLt\y — y'\dt + 2e~aTCo/a .
(8.86)
If L < a, letting T —► 00 we obtain
\J(y, u) - J(y',u)\ < —~—х\у~у'\-	(8.87)
a — L
If L > a, we choose
т = -7 ln|j/-y'|,-
JLj
When L = a, from (8.86) we deduce
\J(y, u) - J(y', U)| <-Lln|y~y/| \y - y'\ + Coly - y'l < G \y - y'\V (8.88)
Q
for 7 < 1 and a suitable constant C\. When L > o, the above choice of T
yields
|J(y,u) - J(y',u)\ < z^exp(^ln|y-y'||+2exp(f
<тЬ\У~ y'\(L~a)/L + 2C0|j/ - y'\“/L
^С^у-уГ-
(8.89)
194	8 Viscosity solutions for Hamilton-Jacobi equations
3. Consider again two distinct points у y', with \y — y'\ < 1. Fix any e > 0
and choose a near-optimal control tz(-) such that
J(y,u) < V(y) + e
If we use this same control in connection with the initial value z(0) = y', we
obtain a cost
< J(y,u) + Ci|y-y'|7 < V(y) +e + Ci|y-y'|7.
Therefore
V(y') < J(y’, u) < V(y) + e + G|y - y'R.
Since c > 0 can be chosen arbitrarily small, arid the points y, y' can be inter-
changed, this proves the Holder continuity of the value function.
In the case L < a, by (8.87) we can take 7 = 1, hence the value function
V is Lipschitz continuous.
Also in this case, the value function V can be characterized as the unique
viscosity solution to a Hamilton-Jacobi equation. First we prove a Dynamic
Programming Principle.
Theorem 8.8.2. (Dynamic Programming Principle - Infinite Hori-
zon). For every т > 0 and у € IR7’, one has
Vfy) = inf | [ e~ath(x(t:,y,u), u(t)) dt T e~aTV(xfr, y, ?z)) 1 . (8.90)
n(-) l./o	J
Proof. Call JT the right hand side of (8.90).
1. We first prove that JT < Vfy). For every control и € U. we have:
J(y, u) — J e~ath(<x(t;yiu)^ u(t))dt T j e~ath(x(t\y,u), u(tf) dt =
f e~at'h(x(t',y,u\ uft)) dt + e~ar I e~°lth(<x{T T t-,y,u), ufr + t)) dt.
Jo	Jo
Define the control u : [0, Too) i—> U by uft) = uft T т), then
Jfy,u) = f e~ath(x(t-, y,u), u(t)) dt T e~aT J(x(r, y, w), u) > JT.
Jo
Since и was arbitrary, this establishes our claim.
2. To prove that Vfy) < JT, fix e > 0. Then there exist a control и! e Id such
that
I e~ath(x(t-,y,u'), u'(t))dt T е~ат V(x(r; у, uf) < JT T г (8.91)
Jo
8.8 Infinite horizon problems 195
and a control u" e U such that
7(я(т; ?/, и'), и") < V(я(т; г/, г/)) + 5.	(8.92)
One can now define a new control й G U as the concatenation of uf up to time
т and u" time shifted:
1 u"(t — t) if te]r, +oo).
Then, by (8.91) and (8.92), we get:
V(y) <
= f e~ath(x(t\y,u), й(1)) dt + J e~ath(<x(t]y1u)1 u(t)) dt
= I e~ath(x(t;y,u,)y u'(t))dt + e~ar J(x(r; y, u'), u")
Jo
< JT + e(l + e"QT).
Since £ > 0 can be arbitrarily small, this second inequality is proved.
We are now ready to show that the value function is solution of a first
order P.D.E., in the viscosity sense.
Theorem 8.8.3. In connection with the control system (8.51), consider the
value function V = V(y) defined by (8.84) and (8.81). Then V is viscosity
solution of the Hamilton-Jacobi-Bellman equation
- [ - aV + H(x, VV)] =0 x e lRn,	(8.93)
and Hamiltonian function given by (8.68).
Proof. To show that V is a viscosity solution, let e C1 (lRn). Two separate
statements need to be proved:
(Pl) If V — kp attains a local maximum at a point xq G IRn, then
-aip(xo) 4- min {f(xo,ui) • V^(t0) + h(xo,u)} > 0.	(8.94)
(P2) If V — attains a local minimum at a point tq G IRn, then
—Q(/?(x0) + min {f(xo,aj) • V(£(a7o) + /г(жо,^) < 0-	(8.95)
1. To prove (Pl), we can assume that
V(zq) = (X#o)>	V(z) < <^Gr) for all x.
Fix (j€U and consider the trajectory x(t) = x(t; 0, Xq,iv\ solution of
196	8 Viscosity solutions for Hamilton-Jacobi equations
±(Z) = /(#(£), о?),	я(0) = Xq.
Then:
99(^0) - y>(x(t)) < V(z0) - V(x(t))
thus by the Dynamic Programming Principe (Theorem 8.8.2), we can write:
^(^0) - 9?(я(/)) < [ e~a8h(s,x(s)) ds + V(a;(^))(e“Qt - 1).
Jo
Letting t —> 0, we get:
-Dip(x0) •	< h(x0,u) - aV(xo).
Letting cj vary in U we get the desired conclusion.
2. To prove (P2), we can assume that
V(xo) = <^(#0),	V(#) ></>(#) for all x.
Fix e > 0. By the Dynamic Programming Principe (Theorem 8.8.2), for every
t > 0 there exists a control t?(-) and a corresponding trajectory xl such that
V(x0) > Г e“QS/i(^(s),u£(s)) ds + e-atV(x\t)) - te.
Jo
We can thus write
¥?(a?o) - <p(z‘(0) > v(xo) ~ ^(*‘(0) >
> [ e"QS/l(j;t(s),u‘(s))(/s + V(xt(Z))(e-at-l)-t£. (8.96)
Jo
On the other hand:
<p(x0) - ^(x‘(t)) = - [ W(s)) • /(®‘(s), u‘(s)) ds. (8.97)
Jo
Using (8.96) and (8.97), by continuity we can write:
— J [v^C^o) ’ /(^o, w*(s)) + ^(^o, W4S))] ds + / (1 — е-а5)/г(а?Отt££(s)) c?5
+V(a/(t))(l - e~at) > -te + o(Z),
where the Landau symbol o(t) denotes a quantity such that lim*—о о(0Л — 0-
The second integral is also o(t), thus we can write:
-t H(x0, V(f?(iro)) + ^(^£(0)(l ~ e-Qt) > — te + o(i).
Dividing by — t and passing to the limit, we get the conclusion since e is
arbitrary.
A comprehensive study of control problems in infinite horizon can be found
in [25].
8.8 Infinite horizon problems
197
Problems
8.1.	Consider the Cauchy problem:
ut + a(t,x)ux = 0,	u(0, a?) = uq(x),	(8.98)
where t > 0, x G IR. a and uq are smooth functions.
a)	Prove that there exists a smooth solution u(t, x) to (8.98).
b)	Find the solution to (8.98) for a(t,x) = x and uo(x') = sin(x).
8.2.	Let 12 = IR \ {0} and consider the equation:
(|uj - 1) (x2 - 1) = 0 Vrr e P, u(0) = 0.	(8.99)
Find all solutions и to (8.99) such that there exist a sequence sn, En —* 0,
and a sequence un, solution to (|uT| — 1) (x2 — 1) = en uxx, converging to
и uniformly on J?.
8.3.	Find £>±(0) for the following functions:
a) u(x) = x sin Q);
b)	u(x) = x2 sin Qj;
c)	u(x,y) = |x| -
8.4.	Consider the function
u(x) = min < |ж — z\ : z e
: n 6 IN
and compute the sets (defined in Lemma 8.2.3.)
8.5.	Write the Dynamic Programming Principle (Theorem 8.6.1) and equation
(8.66) for a Linear-Quadratic optimal control problem.
8.6.	Use Theorem 7.3.3 to derive sufficient conditions for a function U, solving
the HJB equation (8.66) almost everywhere, to be a viscosity solution.
8.7.	Consider the minimum time to origin problem for the linearized pendulum
with external force:
x + x = u,
x G IR, |u| < 1.
(a)	Compute the value function V as in Problem 6.5;
(b)	Show that V is a viscosity solution of the corresponding HJB equation
directly computing D±V at every point.
8.8.	Consider Example 6.5.
(a)	Compute the cost function V corresponding to the constructed syn-
thesis;
198	8 Viscosity solutions for Hamilton-Jacobi equations
(b)	Determine the curves along which V is not differentiable;
(c)	Show that the synthesis is optimal verifying that V is a viscosity
solution to the corresponding HJB equation.
8.9.	Consider the infinite horizon optimal control problem (8.51), (8.81). As-
sume that x G IR.
n(a?, u) = xe + x u,
with U = {?/ : |u| < 1}. For a = 2 is the value function Lipschitz contin-
uous?
9
Patchy Feedbacks
This chapter contains an introduction to the theory of patchy feedbacks and
its applications to problems of asymptotic stabilization and of optimal control.
We consider a nonlinear control system on IRn
x = /(z, u),	u(t) € U C Brn ,	(9.1)
assuming that the map f : IRn x U i—> IRn is smooth. We shall focus the
attention on two classical problems:
(I) Asymptotic Feedback Stabilization: Construct a feedback control
и = U(x} such that all trajectories of the resulting O.D.E.
i = g(X) = f(x, t7(x))	(9.2)
approach the origin as t oo.
(II) Optimal Control Problem: Let a terminal cost ф = and a
running cost function L = L(x, u) be assigned. For a given initial data
я(0) = г/,	(9.3)
find an optimal control ?/(•) and a terminal time т > 0 which minimize the
total cost	r
Ф = ^(ж(т)) + / L(rr(t), u(tY) dt.	(9.4)
Jo
In an ideal situation, the above problems would be solved by a C1 feedback
control x •—> U(x). In this case, for every initial state у e IRn, the Cauchy
problem (9.2)-(9.3) has a unique classical solution, continuously depending on
the initial data. In various cases, however, smooth feedback controls cannot
be used.
(I) For the problem of asymptotic stabilization, several examples of control
systems are known, where each initial state can be steered toward the origin
200	9 Patchy Feedbacks
as t —* oo by an open-loop control t u(t). However, no smooth (or even
continuous) feedback control function x »—> U(x) can accomplish the same
task, not even locally in a small neighborhood of the origin. See Examples 4.2
and 4.3 of Chapter 4. In all these examples, asymptotic stabilization can be
achieved only by a discontinuous feedback.
(II) For nonlinear optimization problems, it is well known that the optimal
feedback can be discontinuous, with a very complicated structure, wdiile op-
timal controls 11—► u(t) can have infinitely many switchings. Moreover, a dis-
continuous optimal feedback may not be robust w.r.t. perturbations. In other
words, one may find arbitrarily small functions ei,e2 such that the solution
to the perturbed equation
x = f(x, U(x + «1 (t))) + 62(t)
(9-5)
achieves a performance far worse than the optimal one. See [71].
It is worth noting that the use of a discontinuous feedback control x i—►
U(x) leads to the theoretical problem of how to interpret solutions for the
O.D.E. (9.2), when the right hand side is discontinuous w.r.t. the state variable
x. Indeed, the standard theory of O.D.E’s does not include any general result
in this direction. Different approaches can be followed:
•	Consider only solutions in a strong sense. We recall that a Caratheodory
solution of (9.2)-(9.3) is an absolutely continuous map 11—» x(t) which satisfies
(9.3) together with (9.2) at a.e. time t. Equivalently,
®(i) = У + / s(z(s)) ds.
Jo
(9-6)
In general, we expect that these strong solutions will have the asymptotic
convergence or optimality properties required by the original control problem.
However, if one does not impose any regularity assumption of the feedback
C7(x), there is no guarantee that any Caratheodory solution wall exist. The
simplest example is
if x < 0 ,
if x > 0 ,
z(0) = 0,
where no solution exists, forward in time.
•	Define a weaker concept of solution. For example, if g is measurable and
bounded, one can consider “Krasovsky solutions” (see for example [50]) of
the O.D.E. (9.2). These are, by definition, the trajectories of the differential
inclusion
x e G(x) = p) сб|д(у); \y - ят| < £ j .
e>0
9.1 Patchy vector fields
201
It is not difficult to show that, in this relaxed sense, at least one solution
always exists. In some cases, however, too many solutions are obtained. Not
all of them may have the desired properties.
In the present chapter we outline a recent approach, introduced in [2],
based on patchy feedbacks. These are piecewise constant controls, whose
discontinuities are sufficiently tame in order to guarantee the existence of
Caratheodory solutions forward in time. At the same time, this class is suffi-
ciently broad to solve a wide class of stabilization and optimization problems.
Section 9.1 contains the basic definitions of patchy vector fields and patchy
feedbacks, and a summary of their basic properties. In particular, we prove the
existence of forward solutions and the stability of the solution set under small
perturbations. In Section 9.2 we show how to solve the asymptotic stabilization
problem by means of a patchy feedback. Section 9.3 is concerned with the
robustness of patchy feedbacks, showing that they still perform well also in
the presence of small external perturbations, or small measurement errors in
the state of the system. Finally, in Section 9.4 we construct a patchy feedback
which is nearly optimal, for a general class of optimization problems with free
terminal time.
9.1 Patchy vector fields
We begin with some definitions. Throughout the following, the boundary and
the closure of a set 12 are denoted by dS2 and 12, respectively.
Definition 9.1 (single patch). By a patch we mean a pair (12, g} where
12 C IRn is an open domain with smooth boundary <912, and g is a Lipschitz
continuous vector field defined on a neighborhood of the closure 12, which
points strictly inward at each boundary point x E d(2. Calling п(ж) the outer
normal at the boundary point x, and denoting the inner product by a dot, we
thus require
sup n(rr) • g(x) < 0.	(9.7)
хЕдГ2
Definition 9.2 (Patchy vector field). We say that g : 12 i—» IRn is a
patchy vector field on the open domain 12 if there exists a family of patches
{(12a, gafi a e Л} such that
- A is a totally ordered set of indices,
- the open sets 12a form a locally finite covering of 12,
- the vector field g can be written in the form
= 9a(x) if
x €	\ [J fig .
fl>ot
(9-8)
202	9 Patchy Feedbacks
We shall occasionally adopt the longer notation (12, g, (12Q, <7а)а6Л) to in-
dicate a patchy vector field, specifying both the domain and the single patches.
By setting
o*(t) = max {a € A ; x E 12a},	(9.9)
we can write (9.8) in the equivalent form
g(x) = <7а.(ж)(я) for all x E £2.
(9.10)
Remark 9.1. It is important to observe that the patches (12Q, ga) are not
uniquely determined by a patchy vector field (12, g). Indeed, whenever a < /3,
by (9.8) the values of gn on the set 12Q A 12^ are irrelevant. Of course, the
values of ga for x outside the domain J? don’t matter either. Therefore, if the
open sets 12a form a locally finite covering of 12 and if for each a E Л the
vector field ga satisfies
na(x) • ga(x) < -6 < 0
for all x e n n dfia \ [J ,	(9.11)
/3>a
then the vector field g in (9.8) is still a patchy vector field. Indeed, without
changing the function 9, one can suitably redefine the values of each ga on
the set U/3>« or outside 12, and achieve the strict inequality
nQ(x)-ga(x) <	< 0
for all x E df2a .
Remark 9.2. For convenience, we are always assuming that the single patches
12tt are open, while the vector fields ga are defined on the closure 12Q . In
certain cases, it would be natural to choose patches of the form
12x = {# E 42; n • x < c} ,
1?2 = {x e 12; n • x > c} ,
for some unit vector n. In this way, however, the union 12i U12'2 does not cover
all of 12, because it does not contain the points where n • x = c. This situation
is easily fixed, replacing 121 by a slightly larger open set which contains also
these boundary points. The resulting vector field
0(z) =
f 51 (*)
if
if
n • X < c,
n • X > c,
can still be written in patchy form.
Remark 9.3. In some situations it is convenient to adopt a more general def-
inition of patchy vector field than the one formulated above. Indeed, one can
consider patches (12a, g(^ where the boundary of the domain 12Q is only piece-
wise smooth. For example, 12a could be the intersection of a ball and finitely
9.1 Patchy vector fields 203
many half-spaces. In this more general case, the inward-pointing condition
(9.7) can be reformulated by asking that, at each boundary point x 6 d!2a,
the vector ga(x) lies strictly in the the interior of the tangent cone to J?a at
the point x. Namely
ga(x) e int7bo(.-r),	(9.12)
where the tangent cone is defined by
T^(x) = ^elRn
lim inf
t|0
d(x + tv, 12Q)	1
t ~ T
All the results concerning patchy vector fields, stated in Theorem 9.1.1 below,
remain valid with this more general formulation. We also observe that, by
slightly modifying the domains 12a, we can always obtain a patchy vector
field where all boundaries dQa are smooth.
Definition 9.3 (Patchy feedback). Let (12, g, (12a, ра)а6Л) be a patchy
vector field. Assume that there exist control values va G U such that, for each
a € Д, there holds
^a(^) = f(x, va)	for all x G 12Q \ [J I?/?.	(9.13)
/3>a
Then the piecewise constant map
U(x) = va if x G 12Q \ U^.	(9.14)
(3>a
is called a patchy feedback control on J?.
Recalling (9.9), the patchy feedback control can thus be written on the
form
t7(j?) — va*^x) •
The next theorem collects the basic properties of Caratheodory solutions
for the O.D.E.
i = g(x) = go.(l)(x),	(9.15)
where g is a patchy vector field (see figure 9.1).
Theorem 9.1.1. (Trajectories of patchy vector fields).
Let (12, g, (1?Q, <7а)аел) be a patchy vector field. Then the following holds.
(i)	If 11—> x(t) is a Caratheodory solution to (9.15) on an open time interval
J, then its derivative t x(t) is piecewise and has a finite set of jumps
on any compact subset J' C J. The function 11—> a* (#(£)) defined at (9.9)
is non-decreasing and left-continuous. Moreover the one-sided limit of x
satisfies
*lim i(t) = g(x(r)) for every r G J. (9.16)
204
9 Patchy Feedbacks
Fig. 9.1. Trajectories of a patchy vector field.
(ii)	For each xq G 12, the Cauchy problem for (9.15) with initial condition
яг(0) = jtq has at least one local forward Caratheodory solution and at
most one backward Caratheodory solution.
(iii)	The set of Caratheodory solutions of (9.15) is closed. More precisely,
assume that xy :	i—> f2 is a sequence of solutions and, as v oo,
there holds
ay —> a,	by —> b, xy(t)x{t) for all t e]a,b[.	(9.17)
Then f(-) is itself a Caratheodory solution of (9.15).
Proof, (i) Since {f2a : a G Л} is a locally finite covering of 12, the function
t h-> a*(t) can take only finitely many values on each compact interval. In
particular, its lim-sup and lim-inf are well defined at each time r. To prove
that a* is left continuous, fix any r G J. Since f2Q.(r) is open, x(t) G
for all t sufficiently close to t. Therefore
lim inf o*(t) > Q*(r).	(9.18)
On the other hand, we now show that
a = lim sup a*(t) < а*(т) .	(9.19)
t-*r-
Indeed, since o* takes finitely many values on compact intervals, we can choose
8 > 0 such that o*(Z) < a. t G ]r—6, т] C J, then there exists s G ]r—6, t] with
a*(s) = a. The inward-pointing condition (9.7) now implies that x(t) G 12«
for every t G [s, г]. Hence o*(r) > a. Together, (9.18) and (9.19) show that the
map t и-> o*(i) is left-continuous, non -decreasing and thus piecewise constant
on compact sets. The identity (9.16) is now clear, because x =	and
the map t i—> «*(#(£)) is piecewise constant. Finally, x is piecewise C1.
(ii) To prove the local existence of solutions, forward in time, define
a = max{a; x$ G f2Q}. Because of the inward pointing condition (9.7),
the solution to the Cauchy problem
9.1 Patchy vector fields 205
x =	x(0)=xo,
remains inside for all t > 0. Moreover, by definition of the index ct, we
have x(t) Qp for all (3 > a and t € [0, J]. Hence, this function x(-) provides
also a solution of the original equation x = g(x), on the interval [0, <5].
To show backward uniqueness, let a?i(-), Я2С) be any two Caratheodory so-
lutions with jq(0) = £2(0). If they do not coincide for all times t < 0, by
continuity there will be a minimum time r such that jq(t) = X2 (t) for all
t e [t, 0] but xi(tk) / %2(tk) for an increasing sequence of times tk r-.
For i = l,2, set ct*(£) = max{a; Xi(t) e By (i), the maps t o*(i) are
piecewise constant and left continuous. Hence there exists 6 > 0 and indices
Qi, such that for every t E [t — 5, t], we have
q*(£) = &i, i = 1, 2.
But then di = oq(r) = a^C7) = ^2- Moreover, #i(-) and are solutions
of the same differential equation x — g^x) with same initial data. Since
is continuously differentiable, they coincide on an interval [r — 5, т].
This contradiction yields the uniqueness result.
(iii) Finally, let xy : [ay, by] »—> ]Rn be a sequence of solutions of (9.15),
satisfying (9.17). We need to prove that the limit x(-) is also a Caratheodory
solution. First, we observe that on any compact subinterval J C ]a, b[ the
functions xy are uniformly continuous and intersect a finite number of domains
say with indices cq < &2 < • ’ • < Qm. For each 1/, the function
o*(t) = max {a e A ; xu(t) e
is non-decreasing and left continuous, hence it can be written in the form
= if
By taking a subsequence we can assume that, as и —► 00, Vj —> tj for all j.
By a standard convergence result for smooth O.D.E’s, the function x provides
a solution to x = gQ . (x) on each open subinterval Ij = ]tj, tj+i [• Since the
domains are open, there holds
x(t) Qfj	for all /3 > aj, t E Ij.
On the other hand, since gQj is inward pointing, a limit of trajectories xy =
gaj(xy) taking values within 31aj must remain in the interior of I2aj. Hence
a*(x(f)) = aj for all t E Ij, completing the proof of (iii).
Example 9.1 Consider the covering of J? = IR2 consisting of
= IR2,
= {(^1,^2); xi < -x%},
= {(^1,^2); X!>X%},
206	9 Patchy Feedbacks
Fig. 9.2. The patchy vector field in Example 9.1.
and the family of inward-pointing vector fields
91 : f?i —► IR2,
92 '	—+ IR2,
9з :	-* IR2,
31(2:1,2:2) = (0,1),
32(2:1,2:2) = (-1,0),
33(2:1,2:2) = (1,0).
Then the vector field 3 on IR2 defined as
3(2:1,2:2) =
(0,1) if
(-1,0) if
(1,0) if
|2?i I < X%
.	2
Xi < — a?2
X1 > x%
is the patchy vector field associated with : a = 1,2,3} and {ga : a =
1,2,3}, see figure 9.2. Notice that the O.D.E. x = g(x) has three forward
solutions and one backward solution starting at the origin. For every initial
point of the form у = (£, у/?) with £ > 0, there exist two forward Caratheodory
solutions, but no solution backward in time.
Remark 9.4. As shown in the previous example, the set of all trajectories of
a patchy vector field starting from a given point is closed but not connected, in
general. For many applications, this is actually a desirable property. Indeed,
it avoids all the topological obstructions that can prevent the existence of
continuous stabilizing feedbacks. See Example 4.3 of Chapter 4.
9.2 Asymptotic feedback stabilization
In this section we describe a method for constructing discontinuous feedbacks
which stabilize nonlinear controllable systems. The basic idea can be explained
as follows. Assume that for each initial point у we can construct an open-loop
control t •—► uy(t) that steers the sytem from у into a small neighborhood
the origin. By continuity, this control uy will still perform the same task for
9.2 Asymptotic feedback stabilization 207
all initial points sufficiently close to y. By patching together a locally finite
family of these controls uy, we shall eventually obtain a feedback control on
the entire space IRn.
We begin by introducing a minimal set of assumptions, in order to have a
stabilizing feedback for the system (9.1).
Given a point у € lRn, and a control tz(-), we denote by ; г/, iz(-)), or briefly
x(-; u), the solution to the Cauchy problem
x = f(x(t),u(t)),	x(0) = y.
Definition 9.4 (Asymptotically controllable systems). The system (9.1)
is said to be globally asymptotically controllable (to the origin) if the following
holds.
1. Attractiveness: for each у e IRn there exists some admissible open-loop
control t i—> uy(t) 6 U such that the trajectory x(-;uy) starting at у is
defined for all t > 0 and satisfies x(t\uy) —> 0 as t —► oo.
2. Lyapunov stability: for each e > 0 there exists 6 > 0 such that for every
у e IRn with I?/1 < 6 there is an admissible control uy as in 1. such that
|a;(t;?iy)| < £ for all t > 0.
We will show that the above conditions are sufficient for the existence of
a stabilizing discontinuous feedback. The basic step in the construction of a
stabilizing patchy feedback is provided by the next lemma. This provides a
feedback that steers every point in the spherical shell Sr,s = {#; r < |x| < s}
into the inner ball Br = {x\ |rr | < 7~}.
Lemma 9.2.1. Let the system (9.1) be globally asymptotically controllable to
the origin. Then, for every 0 < r < s there exist T > 0y R > 0 and a patchy
feedback control U : f2 i—> U, defined on some open domain satisfying
{x ; r < |a?| < s} С 12 C {x ; |x| < R} ,
such that the following holds. For every initial state xq € 12 with |xq| > r, the
Cauchy problem for (9.13) admits at least one Caratheodory solution, forward
in time. Moreover, for every such solution 11—> 7(t), there exists ty <T such
that
|7(M<r.	(9.20)
Proof. 1. For every point у in the spherical shell Sr,s defined above, there
exists an open-loop control uy : [0,^] —> U that steers the system from у
inside the open ball Bs. The corresponding trajectory t ^(t) thus satisfies
я^(0) = у and xy(ty) e Br (see figure 9.3). By an approximation argument,
we can assume that the control uy is piecewise constant and right-continuous,
so that
Uy(fi) = Uj
te [tj, tj+ib j = 0,...,A-l,
208	9 Patchy Feedbacks
Fig. 9.3. Steering the system from у into the smaller ball Bs with an open-loop
control.
with to = 0, £дг = ty. It is clearly not restrictive to assume that the map
t i—> xy(t) is one-to-one. Otherwise we can simply delete closed loops and
obtain a trajectory without self-intersections. In particular	0.
Fig. 9.4. Construction of a section Zj of the flow tube.
2. We first construct a patchy feedback in a tube-like domain around the
trajectory {xy(t); t € [0, fj}. For each j = 0,1,... N - 1, given Cj > 0, we
construct a tube Fj around the portion of the trajectory
{*”(*) 5 t e [tj, ij+i]}.
Set Xj = xy(tj) and consider the (n — l)-dimensional ball B' with radius Sj,
centered at Xj and perpendicular to the vector f(xj, Uj)
B' = {z G IRn ; \x-Xj\<£j, (х-Xj, f(xj, Uj)) = o|.
9.2 Asymptotic feedback stabilization 209
For each b € B' and t G [tj — Ej , tj+i + call t i—► X(b, t) the solution to
the Cauchy problem
X = f(x, Uj) , x(tj) — b •
Observe that the tube (see figure 9.4)
Bj =	b 6 Bj? tj — Ej < t <	4~ Ej
together with the vector field f(x,Uj) does not yet constitute a patch, accord-
ing to our previous definition. Indeed, its boundary is not entirely smooth,
and the vector field is tangent to the lateral surface, not inward pointing. We
fix this problem by defining (see figure 9.5)
Qj = {%(£, 6); b e B', tj - Ej 4- Cj\b - Xj\2 < t < tj+i +Ej|,
where Cj is a constant large enough so that tj — Ej 4- Cj£2 > tj+i + Ej. This
guarantees that the lower boundary of , say
d~Q3; = |x(£,b); beBj, t = tj — Ej 4- Cj\b - xj2,
is smooth. Moreover, the vector field f(x,Uj) is strictly inward pointing at
every point of d~(lj .
Fig. 9.5. Replacing the flow tube by a patch Qj .
We now patch together the domains starting from and proceed-
ing by backward induction, as in figure 9.6. For every j = 0,1,...,7V — 1,
call
d+flj = |x(Z, 6); b e Bj, t>Cjlb — Xj\2, t =	4-
the upper boundary of Qj. We begin by choosing En-\ so small that all points
on the upper boundary d+ lie inside the ball Br. By induction, after Ej
has been determined, we choose Ej-i > 0 small enough so that all points on
the upper boundary d~ Qj-i are contained inside f2j.
Since the vector field f(x, Uj) points strictly inward on all the lower bound-
aries d~Qj, by Remark 9.1 the family |(f?j , /(•, Uj)); j = 0,1,..., N — 1 j
yields a patchy vector field.
210	9 Patchy Feedbacks
Fig. 9.6. Patching together the domains ,...,	.
3. For every initial point у in the spherical shell 5r,s, let Г2У be the open
set on which a patchy feedback is constructed, as in the previous two steps.
These sets form an open covering of the compact set Sr,s. We can thus
extract a finite subcovering, say Qyi,..., I2!Jm. For every i = 1,..., m, let
, /(•, Uij)), j = 1,..., rrii, be the patches defining the patchy vector field
on Qyi. We can then define a patchy vector field on the whole set D
by taking all patches (J?^-,/(•, w^)) and ordering them according to the
lexicographic order. Namely, (i, J) -< (г',/) if either i < i' or i = i' and j < j'.
Defining
{rrii
tyi + 2 eij
j=l
R — max sup < |rr|; x E ftyi
the proof is completed.
Using the above lemma, we can now prove:
Theorem 9.2.2. (Asymptotic stabilization with a patchy feedback).
If the system (9.1) is globally asymptotically controllable to the origin, then it
admits a stabilizing patchy feedback.
Proof. 1. We apply Proposition 9.2.1 to all spherical shells Sk =	; 2-/c“1 <
И < 2-fc}, where к ranges over all positive and negative integers. For every
k, this yields a patchy feedback (f4,£ , /(•, Uk,ef), t = 1, 2,..., Nk , defined
on the open domain
Nk
f4 = [J
c{x; 2-fe"2 < |l| <
(9.21)
This feedback steers all points of the spherical shell Sk into the smaller ball
В = {x; |x| < 2-fc“1}. Thanks to the property (ii) in the definition of
9.3 Robustness
211
asymptotic controllability, the construction can be performed so that the up-
per bounds Rk in (9.21) satisfy Rk —> 0 as к —> oo. Therefore, the family of all
patches ftk,£ constitutes a locally finite open covering of the space IRn \ {0}.
2.	The index set A = {(fc, £); к integer, f	can be totally
ordered lexicographically, i.e. by letting (fc,£) -< (V,f) if either к < к' or else
к = к' and £ < (!. The piecewise constant map I/* : IRn\{0} »—► U defined by
{/‘(я) = Uk,t if x^ilk^\ [J
(M)<(k',r)
provides the desired patchy feedback control.
9.3 Robustness
In practical applications, one has to take into account the presence of several
perturbations which may degrade the performance of a feedback control. For
example:
(i)	The model equation, described by the function f in (9.1), may not be
precisely known.
(ii)	The evolution of the system may be affected by random external pertur-
bations.
(iii)	While implementing the feedback, the state of the system may not be
accurately measured.
As a result, instead of the dynamics
x = f(x, C/(x)),
the controlled system will actually evolve according to
x = f(x, U(x + £i(*))) +e2(t),
(9.22)
(9.23)
for some small perturbations ei, e2. Here the ‘‘inner perturbation” ei accounts
for measurement errors, while the “outer perturbation” s2 models and external
disturbances.
In the above framework, it is important to design a feedback control which
is robust, i.e. it still accomplishes the desired task in the presence of (suffi-
ciently small) perturbations. As shown in this section, this important property
is actually achieved by all patchy feedbacks. We shall first prove some stabil-
ity results valid for patchy vector fields, then give an application to patchy
feedbacks.
212	9 Patchy Feedbacks
Together with the O.D.E. (9.15) determined by a patchy vector field g,
consider the perturbed equation
x = g(x) 4- w ,	(9.24)
where t »—> w(t) is a (possibly discontinuous) function with bounded variation.
By definition, a solution of (9.24) is a function t i—> x(t) such that
x(t) = a:(0) + / g(x(s)) ds 4- [w(t) - w(0)] .	(9.25)
Jo
for every t. Notice that the right hand side of (9.24) contains the derivative of
w. In particular, if the function w(-) has a jump at a time t, the same is true
for rr(-). We recall that a function ф : [0, T] i—> IRn has bounded variation if
N
Tot.Var.{0} = sup	l<№) “ 0(^-i)l < 00 •
o=to<ti< -<tN=T i=1
The total variation norm of ф is then defined as
HIlBV-Tot.Var.W + HIlL-	(9-26)
The next lemma extends the result in part (iii) of Theorem 9.1.1 to the
case where impulsive perturbations are also present.
Lemma 9.3.1. Let (12,g, (12a, да)аел) be a patchy vector field. Let x„ :
[0, T] i-* IRn be a sequence of solutions to the perturbed equations
xu = р(жр) 4- Wy ,	(9.27)
where Wy : [0, T] i—> IRn are BV functions such that
Tot. Var.{Wi,} —► 0	as у —> oo .	(9.28)
Assume that the solutions х„(-) take values inside a compact subset К C 12
and, converge pointwise to a function x : [0, T] i—> K. Then x(-) is a
Caratheodory solution of the unperturbed equation (9.15).
Proof. 1. To establish the lemma, it suffices to show that
lim [ \g(xy(t)) - g(x(tf)\dt = 0.	(9.29)
Jo
Indeed, (9.28) implies that |wi,(t) — w(0)| —> 0 uniformly for t G [0, Т]. If (9.29)
holds, we can thus conclude
x(t) = lim (яД0)4- / g(xy(t))dt + Iw^t) - wp(0)l )
\ Jo	/
= x(0) 4- I g(x(t))dt	(9.30)
Jo
9.3 Robustness 213
for every r G [0,T], proving the lemma.
For each t G [0, T], consider the indices
= max {a G Л; x(t) G	= max {a G A; xy(t) G .
By the definition of patchy vector fields, one has
p(x(t)) = pQ(t)(a;(t)),	= Pa„(z„(*)) •
Since each vector field ga is bounded and continuous, we know that xy(f) —>
x(i) uniformly for t G [0,Т]. To prove (9.29) it thus suffices to to show that
lim meas({£ G [0, T]; au(t) / a(t)}) = 0.	(9.31)
2.	Observing that f2Q(r) is open and ху(т) —> ж(т), for all v sufficiently large
one has xy(r) G . Therefore
lim inf ay(r) > a(r)	for all r E [0, T],
i/—»oo
and hence
Jnn^ meas({r G [0,T]; аДт) < q(t)}) =0.	(9.32)
The following steps will establish the identity
lim measure [0,T]; ou(r) > а(т)}) =0.	(9.33)
3.	Toward a proof of (9.33), we first show that, for every fixed time т G [0, T],
limsup meas( {t G [0,т]; оД^) > q(t)} 1 = 0.	(9.34)
v—>oc	'	'
Assume, on the contrary, that
limsup measf{t G [0,r]; ap(Z) > o(r)}") > 0.	(9.35)
P—>oo	'	'
A contradiction is obtained as follows. Since the patches which intersect
the compact set К are finitely many, there exists a unique index /3 >
such that
limsup measf{7 G [0,r]; ay(t) > /3}} = 0,	(9.36)
l/—->OO	'	'
while
limsup measf{£ G [0, t] ; ay(t) =/3}} = 7]q > 0 .	(9.37)
v—►oo	'
Consider the distance function from the complement of the set
Ф^(х) = dist (x ;
214
9 Patchy Feedbacks
Since the vector field gp is strictly inward pointing on the boundary сИЛз, we
can find 5 > 0, p > 0 such that, for every solution of the O.D.E.
y(0 = ^(уЮ),
one has
^Фз(у(<)) > <5 whenever 0 < 03(y(t)) < p. (9.38)
at
More generally, if t x(t) is a solution of the patchy O.D.E. (9.15), then
^Фр(х(1)) > 6 for a.e. t such that Фр(х(Ь)) < p and x(f) G ftp \
(9.39)
^Фр(х(1)) = 0	for a.e. t such that x(t) ftp ,	(9.40)
^<^(z(Z)) > -C	for a.e. t G [0, T],	(9.41)
with C = supx€/< |<7(#)| • We also observe that Фр, being a distance function,
satisfies
\Фр(х)-Ф0(у)\<\х-у\.	(9.42)
Next, consider the functions t Ф^(жг(^)), where x„ are the perturbed
solutions in (9.27). Observing that
^(r) = rCp(O) + [wr(r) - wI/(0)] + /	^(^p W) dt
Jt€[0,r],<*„(t)=p
+ [	g(xl/(t\)dt+ I	g^x^t^dt
Jte[0,r],a^(t)>P	Jt€[0,r],a,,(t)<P
and using the previous four estimates (9.39)-(9.42), we obtain (see Pb. 9.3 at
the end of this section for more details)
Фр(х»(тУ) > -Tot.Var.{wp} - C • measf{f G [0, r];
f	/	(9.43)
+ min ip , <5 • meas I {£ G [0, t] ; a„(t)=/3}]>.
Letting у —► oo, by (9.28) and (9.36) the first two terms on the right hand side
of (9.43) approach zero. Using (9.37) to estimate the third term, we conclude
limsup Фр(х„(тУ) > min<p, tfr/of >0.
This implies
0/j(x(r)) = lim фр(х^т)) > 0.
P—>oo
Hence x(r) G i2p and q(.t(t)) > /3, reaching a contradiction.
9.3 Robustness 215
4.	Next, assume that (9.33) fails. Then we can find an index /3 such that
limsup measf {t G [0,T]; ct(f) = (3 and ay(t) > (3}\ > 0.
i/—►oo	'	'
In particular, we can find a time т G [0, T] such that
a(r) = 0,
lim sup meas
0.
This provides a contradiction with (9.34). Hence (9.33) must hold.
Together, (9.32) and (9.33) yield (9.31), completing the proof.
We now consider the O.D.E. determined by patchy vector field g in the
presence of internal and external perturbations:
i = g(x + ei(t))+e2(t).	(9.44)
Assuming that the the perturbations ei, 62 are sufficiently small, the next
theorem shows that every solution of (9.44) remains close to some solution of
the unperturbed equation (9.15).
Theorem 9.3.2. (Robustness for patchy vector fields). Let g : J? i—> IRn
be a patchy vector field. Given any compact subset К C 12, and any T,e > 0,
there exists 6 > 0 such that the following holds. If у : [0, T] К is a solution
of the perturbed equation (9.44) and
||ei||sv<(5,	IIMl00 < <5	(9.45)
then there exists a solution x : [0, T] •—> 12 of the unperturbed equation (9.15)
such that
|a;(t) —	£	for all tG[0,T].	(9.46)
Proof. 1. By contradiction, assume that the conclusion was not true. Then
there exist a sequence of perturbations ei^y , 62,^ with
||ei,P||BV —* 0,	11^2,1/||l°°	0	as u—>oo,	(9-47)
and corresponding solutions xy : [0, T] > K. which do not approach any
solution of the original equation (9.15). Namely
H^-^IIl~([0,t]) > £	(9.48)
for some e > 0, every v > 1, and every solution x{-} of (9.15).
2.	Define the functions yy(fi} = xy(t) -h eijIZ(t). These functions satisfy
yv(t) = xv(f) + ei,„(t) =	+ e2>1/(t)) +ei,„(f).
216	9 Patchy Feedbacks
Setting
= ei?p(t) + / e2^(s) ds
Jo
we can thus write
M*) = 5(M0)+ «’„(<).
Notice that
Tot.Var.{w^} < Tot.Var.{ei?I/} +	► 0 as z/—>oo.
By the definition of solution, for every t e [0, T] one has
= 6/Д0) + [ g(y^s))ds\ + [w^t) - wp(0)].	(9.49)
\ Jo	/
The first terms on the right hand side of (9.49) are bounded and uniformly
Lipschitz continuous w.r.t. t. The last terms converge to zero uniformly on
[0,T]. Therefore, by Ascoli’s compactness theorem there exists a subsequence
(?/^)fc>i that converges to some function z(-), uniformly for t e [0, Т].
3.	By Lemma 9.3.1, this limit function z(-) provides a solution to the un-
perturbed equation (9.15). Notice that ||ei>IZ||BV —> 0 implies ||ei,p||l=» —* 0.
Therefore
lim inf ||xp - x||L« < hm (	+ \\Уик - z||l~ ) = 0-
i/—>oc	к—>oo X	/
This yields a contradiction with (9.48), thus proving the theorem.
Remark 9.5. According to the above theorem, every solution of the perturbed
equation is close to some solution of the original one. In general, however, these
solutions cannot be chosen with the same initial data. For example, fig. 9.7
shows the trajectories of a patchy vector field g, and a solution ?/(•) of the
perturbed equation (9.44). In this case, the solution z(-) to the initial value
problem
z = g(z),	z(0) = xQ = ?/(0),
is unique, but very different from ?/(•). In order to find a trajectory £(•) of
the O.D.E. (9.15) which remains always close to ?/(•), one has to start from a
different initial point.
Example 9.2. Consider the patchy vector field on IR2
(	(1,1) ifx2>0,
S(xi,a:2) -	ifa.2<o.
For each v > 1, let x„ : [0, T] i—> IR.2 be the solution to the perturbed Cauchy
problem
9.3 Robustness 217
Fig. 9.7. Two trajectories z(-) and rr(-) of the patchy vector field g, and a solution
y(-) of the perturbed equation (9.44).
ip(t) = glx^t) + ei,i,(t)) + e2>p(t),	жДО) = (0, 2 ").	(9.50)
Calling = [(& —1) 2-1/, k 2-I/[, the inner and outer perturbations are here
taken to be
° (t\-\	(°> °)	if Z	odd	o (t\=(\
- I (Oi _22-^)	jf tel^k, feeven,	e2,v(t)_0.
The corresponding solutions x„ are shown in figure 9.8. Observe that, as
и —* oo, one has the uniform convergence
xjj) —> x(t) = (t,o) t e [о, T].
However, the limit function rr(-) is not a solution of the unperturbed equation.
This does not contradict Theorem 9.3.2, because
lim ||ei p||l«> = 0 but lim inf Tot.Var. {ei ^} > 0 .
I/—»OO	’	V—ЮО
This example shows that the size of internal perturbations should be measured
in the total variation norm, not in the L°° norm.
Fig. 9.8. A solution xp(-) of the patchy O.D.E. (9.50) with internal perturbations.
218
9 Patchy Feedbacks
As a further application of Lemma 9.3.1, we now prove a robustness result
for patchy feedbacks, in connection with a stabilization problem. To avoid
technicalities, we shall assume that the patchy feedback is defined on the
entire space IRn, and that the function f in (9.1) satisfies the sub-linear growth
condition
|/(ж,и)| < cy(l + |x|)
for all и G U , x G IRn.
(9.51)
Theorem 9.3.3. (Robustness of stabilizing patchy feedbacks). Let the
system (9.1) satisfy the growth condition (9.51). As in (9.13)-(9.14), let
U(x) = be a patchy feedback defined on lRn. Given a compact set K$,
assume that every solution of
i = g(x) =
(9.52)
starting inside Kq reaches the ball Br centered at the origin with radius r,
within time T. Then, given e > 0. there exists 6 > 0 such that, if the pertur-
bations Ei, E2 ‘ [0,T] i—> IRn satisfy
Ikillsv < 6,

(9.53)
then any solution of (9.23) with initial data x(0) G enters the ball Br+e
within time T.
Proof. 1. Assume, on the contrary, that the conclusion does not hold. Then
one can find e > 0 and a sequence of perturbations E\>y ,Е2^ч with
||£i,i/||bv	o,,	||£2,^||l«>-> о	as	(9.54)
and corresponding solutions t •—» x„(t) of
iy(t) = f(xv(t), U(x^(t) + Ei>lz(t))) + e2, </(i),	(9.55)
such that the following holds. For every v > 1,
гГр(О) G Kq ,	|xp(£)| > r + £ for all t G [0,T].	(9.56)
2. By assumption, g(x) = f(x, U(x)) is a patchy vector field. We can write
(9.55) in the equivalent form
iy = g(x +	+ e2jP(t),	(9.57)
with
ei,p(<) = £i,p(t)>
e2,„(0 -	+ /(^(i), U(xv(t) + ei,p(t)))
+ ei,p(t), U(xv{t} + £i,p(t))) •
(9.58)
9.4 Nearly optimal patchy feedbacks 219
By (9.54), we also have
ll^i,p||bv	o,,
11^2,1/||ь°° “* 0
as и —> oo .	(9.59)
3. By the sublinear growth condition, all trajectories : [0,T] * IRn remain
uniformly bounded. Repeating the steps 2 -3 in the proof of Theorem 9.3.2,
we can choose a subsequence x„k which converges to a limit function jr(-)
uniformly for t G [0,T]. Using Lemma 9.3.1 we again conclude that rr(-) is a
solution of the unperturbed equation (9.52). However, by (9.56) it follows
z(0) G ,
|x(t)| > r + г for all t G [0,T],	(9.60)
contradicting the main assumption of the theorem. This achieves the proof.
9.4 Nearly optimal patchy feedbacks
Consider a general optimization problem with running cost L, terminal cost
and free terminal time:
min < '0(я:(Т))+ I L(x(t), u(t)) dt I ,	(9.61)
T»w() I	Jo
for the nonlinear control system
x = f(x, u)	u(t) G U,	(9.62)
with fixed initial datum. The minimum is sought over all times T > 0 and all
measurable control functions и : [0,T] i—► U. Aim of this section is to show
that this problem can be approximately solved by a patchy feedback, with an
arbitrary degree of accuracy.
For convenience, we list here all the basic assumptions.
(H) The set of admissible control values U C lRm is a compact, the function
f : JR" x U i—> IRn is continuous w.r.t. both variables, and twice continuously
differentiable w.r.t. x. In addition, f satisfies the sub-linear growth condi-
tion (9.51) for some constant Cf . Both the terminal cost	i—> IR and
the running cost L : IRn x U IR are continuous and non-negative. More
precisely
V>(x) > 0,
Л(Ж, Ii) > Q(] > 0
for all x G IRn, uGU.
(9.63)
Throughout the following, V denotes the value function for the optimiza-
tion problem (9.61)-(9.62), namely
220	9 Patchy Feedbacks
V(y) =
inf
T, x(’,u)
(9.64)
where the minimization is taken over all T > 0 and all solutions of t i—> x(t, u),
z(0, u) = y, corresponding to a measurable control и : [0, T] i—> U. The main
result can be stated as follows.
Theorem 9.4.1. Existence of a nearly optimal patchy feedback). Let
the functions 'ip.L.f in (9.61)-(9.62) satisfy the assumptions (H). Let e > 0
and a compact set К C IRn be given. Then there exist a closed terminal set
S C HVZ and a patchy feedback и = U(я) defined on the complement IRn \ S
such that the following holds. For each у e К, every Caratheodory solution of
x = f(xy t/(rr)) ,
z(0) = у
(9.65)
reaches the set S within finite time. Calling т = inf {t; x(t) € S'} the first
time where the trajectory reaches S, we have
V>(z(t)) + I L(x(t), dt <V(y)+e.	(9.66)
Jo
We recall that, by well known properties of patchy vector fields, for every ini-
tial point у e IRZ'\S the O.D.E. (9.65) has at least one forward Caratheodory
solution. According to (9.66), all of the solutions starting from the compact
set К are nearly optimal, for the cost (9.61).
Proof. The proof of Theorem 9.4.1 will be given in several steps.
1.	Various reductions can be performed. Taking a smooth approximation,
we can assume that € C°°. Moreover, approximating the cost function L
by a more regular function, it is not restrictive to assume that L is twice
continuously differentiable w.r.t. x. Recalling that L(x. u) > Qq > 0, we can
now replace f(x,u) by
g(x,u) =	(9.67)
L[x. u)
and consider the equivalent problem
inf
T,u(-)
(9.68)
with dynamics
x = g(x,u),
x(0) = у.
Notice that the function g in (9.67) is continuous w.r.t. both variables x,u,
and twice continuously differentiable w.r.t. x. Moreover it satisfies the growth
condition
9.4 Nearly optimal patchy feedbacks
221
|5(z,u)| <	(1 + |x|)
for all и e U .
In the following, we thus assume without loss of generality that the running
cost is simply L(x,tz) = 1, so that the minimization problem (9.61) reduces
to (9.68).
2.	Choose a constant M such that
M > 1,
M > max 'ф(х).
хек v
To fix the ideas, throughout the following we assume that 0 < s < 1/8 and
that the compact set К is contained in the open ball Вp centered at the origin
with radius p. Because of the sub-linear growth condition (9.51), the a priori
bound (2.23) of Corollary 2.1.6 holds. In particular, during the time interval
t e [О, 2M] every trajectory of (9.65) starting form a point у 6 К C Bp will
remain inside the open ball Bp, with
p = eCf2M (p + 1).
(9.69)
3.	Let V = V(y) be the value function for the optimization problem (9.68),
with dynamics (9.62). We claim that V is semi-concave. More precisely, there
exists a constant к such that, for any y, y' e Bp, one has
v(y') < v(y) + w • (у’ - у) + к -
(9.70)
for some vector w e D+V(y) in the super differential of V at the point y, see
Section 8.2.
Indeed, from the theory of optimal control [6] it is well known that the
optimization problem (9.68), (9.62) with initial data з:(0) = у has at least one
solution, within the class of chattering controls. Let t i—> x(t) = x(t; y,u,0)
be an optimal chattering trajectory, with
x(0) =?/,	x(f) =	/(#(£), 14(f))	t e [0, r] ,
i=0
for some measurable functions (it, 0) = (tto, • • •,	, 0n) satisfying
щ : [0, г] U,
^:[0,t]h^[0,1],
n
£>(*) = 1.	(9.71)
i=0
For any other initial data y'. we can consider the same chattering control
(й, 0), always stopping at the same terminal time t = r. This yields the cost
Vu'\y') = T +	У,й,0У) .
(9.72)
222
9 Patchy Feedbacks
The regularity assumptions on /, ф w.r.t. the variable x imply that, as y' varies
in the ball Bp, the map y' yu’<9’T(y) is twice continuously differentiable.
Moreover, its C2 norm remains bounded:
11^й’*,Т|1са(Вд) — K-
(9.73)
Since r e [0, Tmax] while both й and 0 in (9.71) range over compact sets, this
bound is uniform, i.e. in (9.73) we can take a constant к > 1 which does not
depend on the particular chattering control, or on the time r. Observing that
V(y) = V^T(y),
V(y’) < V“’^’T(y/)
for all y' e Bp,
the inequality (9.70) follows from (9.73), choosing
w = VV^’^y).
(9-74)
4.	As shown in the previous step, the value function
V(y) = min Уй’ё’т(у)
й,0,т
is Lipschitz continuous on the ball Bp. By (9.74), the constant к > 1 in (9.73)
also provides a Lipschitz constant for V, namely
|У(д:) — V(?/)| < к |x — y\ for all x,y£Bp. (9.75)
By Rademacher’s theorem, sec Theorem A.6.1, V is differentiable almost
everywhere. At each point x E Bp where the gradient W(x) exists, if
V(j:) < ф(х) then one has the relation, see Theorem 7.3.2:
min {W(x) • /(.t,tz)} + 1 = 0.	(9.76)
Consider the open set
V = {# ; V(x) < 'ф(х)} .
Given 6 > 0, we can choose finitely many points y\,... ym G Bp П T) such that
W(?;? ) is well defined for each i = 1,..., m, and moreover
m
ВрПЪС \jB(yi,8).	(9.77)
1=1
Define the approximate value function
IT(x) = min {ф(х), fVi(x), ... , VLmGr)},	(9.78)
where
9.4 Nearly optimal patchy feedbacks 223
Wi(x) = V(j/j) + VV(yi) • (x - yi) + k|x -yi\2.	(9.79)
We claim that, by choosing 8 > 0 sufficiently small, for all x G Bp the
following relations hold.
V(z) < W(x) < V(x) + e,
(9.80)
min {Vl4\(x) •/(x, ?/)}-h l
< £
whenever кИДгг) = W(a?),	(9.81)
Indeed, the first inequality in (9.80) follows from (9.70). Next, since f is
continuous and U is compact, we can find Ji e]0,1] such that the following
conditions hold. If x 6 Bp , w = WQ/) exists and
min {w • f(y, n)} -h 1 = 0,
izGU
|w' - w| < 2k<5i ,	|ж — y\ <	,
then
| min {w' • f(x,u)} + 1| < £.	(9.82)
We now choose 8 > 0 such that
Given any x G Bp, if j is an index such that |rr — yj | < <5, recalling the Lipschitz
condition (9.75) we find
W(x) < V(j/j)+|VV(%)| |ж-у,|+ф-у,|2 < У(ж)+2ф-у,|+к|ж-г/;|2,
/ к8^“ i
Ж(ж)-У(ж) < min |г,	(9.83)
This already yields (9.80). Comparing (9.70) with (9.79) we notice that
W) - V(x) > к |Z ~ У<|2 •
Hence from (9.83) it follows
|x — yi\ < <51 whenever Ж(^) = W(x).	(9.84)
Observing that
|v^(x)-vw^(%)|
< 2/< |x — yj\ < 2k(5i ,
from (9.82) we deduce the inequality (9.81). This establishes our claim.
5.	By the definition of PK, it is clear that all level sets where Wi is constant
are spheres. Indeed, for any given constant c we can write
224
9 Patchy Feedbacks
{x; Wi(x)=c} = {rr; |x-#i|=r},
with Xi = yi —	/2к and a suitable radius r.
For each i = 1,..., m, consider the set
^ = {хеВ-р- Wi(x) = Ж(х)}.
(9.85)
In this step we show that there exists a minimum radius rmin > 0 and a
maximum radius rmaa, such that, if x E Pi, then the level set where Wz —
Wj\x) is a sphere of radius r with
'min _	max •
(9.86)
Indeed, since e< 1/2, by (3.17) it follows
(9.87)
Calling
Mf = max
|x|<p,uEU
from (9.87) we deduce
1
2M~f ’
Therefore
|VWi(x)|
2k
1
4к Mf
On the other hand, by (9.79) and (9.84) we have
min •
|VlV,(x)| < IVWiG/Jl +2K\x-Vi\ < /с + 2к<51 < Зк.
Hence
|V^(x)| < 3
2k “2
T'max •
(9.88)
1
6.	We are now ready to construct the near-optimal patchy feedback on the
open set
Г2 = {z e Bp; W(x) <	.	(9.89)
The terminal set S will then be defined as
S = EV1 \P.
Given r) > 0 small, for each point x G T>i consider the point (see figure 9.9)
Pi
2 1
r+3
X Xi
9.4 Nearly optimal patchy feedbacks
225
and the ball Bx — B(px, |ж — xJ/3) centered at px with radius r = |x — Xf|/3.
By (9.81), there exists a nearly-optimal control value и = ux e U such that

(9.90)
Consider the lens-shaped region
ri = B(Pi, \ B(Xi ’ Iх “	“ *0-	(9'91)
Its upper boundary will be denoted as
д+Ггх = дГх \ Bfa , |z -	- V) •	(9.92)
Moreover, for z € д+ Гх, we write п^(г) for the outer unit normal at the point
z.
W.=W.(x)
Fig. 9.9. Construction of a lens-shaped patch.
We claim that, by choosing p > 0 sufficiently small, the following holds:
VU'(z)-/(z,^)<-l + 2e
for all z e Гх ,	(9.93)
n<(z) •	< -7)
for all z&d+rtx.
(9.94)
Moreover, the constant p > 0 can be chosen uniformly valid for all i — 1,..., m
and all x e Bi.
For fixed z, x this is clear because, as p 0, the diameter of the set Гх
approaches zero. Moreover, as z varies on the upper boundary д+Гх , all the
unit normals n^z) approach the vector VWi(^)/| VWi(x)|. Therefore, both
inequalities (9.93)-(9.94) follow from (9.90).
We now observe that f = /(a;, u) is uniformly continuous on the compact
domain Bp x U. Moreover, on each set 7?г, the gradient VWi(j:) is uniformly
226	9 Patchy Feedbacks
Lipschitz continuous and bounded away from zero. Finally, the radius of each
level set, where Wi is constant, by (9.86) is bounded above and below. This
allows us to choose a constant rj > 0 uniformly valid for all i,x.
7.	To achieve a nearly optimal feedback, we would need the inequality
VW(z) • /(г, u?) < -1 + 4s	for all z G Ггх .	(9.95)
If W(z) = Wi(z) for all z e Гх, this is a trivial consequence of (9.93). However,
we must also consider the case where some of the points z G Г? lie in a region
where W(z) = Hj(z) < W2(z). for some different index j. For this purpose,
we observe that the set where Wi = Wj is always a hyperplane, say
= Wi(x) = WjCr)} = {я; nij-x = Cij}. (9.96)
for a suitable constant Cij and a unit normal vector nZJ . The orientation of
riij will be chosen so that
{.t ; Wi(z) < Wj(x)} = {a?; nij - x < Cij] .
We claim that, by choosing 77 > 0 sufficiently small, uniformly w.r.t. г,х, one
of the following two cases occurs (see figure 9.10).
CASE 1: At every point z G Гх П Hij one has
nij-/(z,O < -т/.	(9.97)
CASE 2: At every point z G Гх one has
VW^z) -f(z,ux) < — l+4e.	(9.98)
Indeed, assume that (9.97) fails. Then there exists a point z* G Гх D'Hij
such that
n0-/(z*,uf) > -77.	(9.99)
By (9.96) and the orientation of the unit vector n4- , we can write
VW/z*) = V^(z*) -(3nij	(9.100)
for some constant (3 > 0. Together, (9.93) and (9.99) now imply
VHA(z‘)-/(^,uf)= VHA(z*)-/(z‘,7zn-/?no-/(^,«?)	/qwn
< -l + 2e + /3r] < — 1+Зг,	1	'
provided that we choose t] > 0 sufficiently small. Since f and VWj are uni-
formly Lipschitz continuous, from (9.101) it follows that (9.98) is valid for all
9.4 Nearly optimal patchy feedbacks 227
z sufficiently close to 2*. By reducing the size of r] > 0, we can make the di-
ameter of the lens-shaped domain Г? as small as we like. Hence the inequality
(9.98) will hold for all z G .
To prove or claim, it remains to observe that the functions f and are
uniformly continuous, and that the constant /3 in (9.100) remains uniformly
bounded. Hence the constant 7/ > 0 can be chosen uniformly valid for all
г,
We now define the smaller domain
Pf = r?\ U {ze]Rn;	(9.102)
jeli
where Ц C {1,... , m} is the set of indices j i for which CASE 1 applies.
By the previous analysis, for each j such that Wj(z) = W}(z) for some
z G Г®, two cases can occur. If CASE 1 applies, then the vector field
is strictly inward-pointing along the portion of the boundary dQ? where Wi =
Wj. On the other hand, if CASE 2 applies, then (9.98) holds on the entire
domain Г? . Notice that 12? always contain a ball centered in x.
Fig. 9.10. If the domain Г? intersects the half-space where Wj < W/, two cases
must be considered. Left: in Case 1, the vector field	points toward the set
where Wt < Wj. As a patch we then take the shaded region Pf G Г*. Right: In
Case 2,	points toward the set where Wj < VK.We can now take Г2? = Г?,
because the control u* is nearly optimal on this whole region.
8.	Consider the family of all domains J??, as i G {1,..., m} and x ranges over
the closure of the set Q = {.r G Bp ; W(x) < Since all these domains
are defined by the same 77 > 0, they are “uniformly large”. More precisely, for
each x G 12, consider the union U« >taken over all i G {1,..., m} such that
x ET>V Then this set contains the ball with radius t] centered at x.
It now remains to select finitely many domains 12? which cover the compact
set J2. This last step, however, must be done with some care because on the
lower portion of the boundary
d~Q* = 012? П Bfa , |x -	- 7/)	(9.103)
the vector field	may not be inward-pointing. To cope with this prob-
lem, we first observe that there exists a uniform constant h > 0 such that
228	9 Patchy Feedbacks
W^z) < W^-th,
(9.104)
for every i, x and every z E d (7? . Wc now set M* = max { IF(x) ; x E Bp },
and split the domain 1? in sub-domains of the form
= {ж E (2; NB - (£ + l)h < W(x) < M* - th} .
(9.105)
For each £, we cover the compact set with finitely many patches
constructed as in step 7. choosing x E . After a relabelling, this yields the
patches (see figure 9.11)
(fytt,/(>4<>))> a	(9.106)
On the collection of all patches (9.106) we define the lexicographic order:
(Ла) -< (Ла')
iff either ( < f! or f — and a < a'.
Fig. 9.11. The domain f? = U is covered by a family of patches .
We claim that the above construction yields a patchy vector field:
g(x) = f(x,u^Q)
iff
X E \
(ЬХ'.а')
(9.107)
Indeed, according to Remark 9.1, it suffices to check that, for each patch
the vector field /('Ль) =	is inward pointing at every
point of the set
Q n dQt,a \ IJ Рг,а/ .
(€,»)<«',а')
In the present case, this is clear, because the only boundary points where
is not inward pointing are those on the lower boundary d~ i2f. Since
x E we have IV(rr) < Af* — £/?,, and hence by (9.104)
W(z) < M* -(€ + 2)Л
for all z E d f2f .
Therefore, given any point z E d , either W(z) = 'ф(г) and z 12, or else
z is contained in a patch f2^yOt> with as required in Remark 9.1.
9.4 Nearly optimal patchy feedbacks 229
9.	To complete the proof, we now check that the patchy feedback that we
have constructed is nearly optimal. We recall that, by the analysis in step 7,
for every г, x we have
VWi(z))-/(z, u?)<-l + 4e
for all zef2*. (9.108)
Now take any initial point у 6 К and let t •—> x(t) be any Caratheodory
solution of the Cauchy problem
x = g(x) x(0) = у ,
with g defined at (9.107). Call г > 0 the first time at which x(t) reaches the
boundary of the set C = {x* E Bp ; W(x) < 'ф(х)} . By (9.108) we have
dt
< (-1+4s)t.
Since 0 < s < 1/8, the above implies
W(y) — W(x(r)}
T	l-4g ~ 2W^ ~	~2M’
By the definition of p at (9.69), it follows that the trajectory 11—> x(t) remains
inside the open ball Bp for all t E [0,т]. Therefore, at time t = r we must
have W(j;(t)) = VJ(^(T))- Stopping at time r, since W(a?) > t^(x’) > 0 and
V(t/) < M, the total cost can be estimated as
, ,7 / n . W(y) - Vy(x(T))	,
т + ^(ж(т)) < ---------
1 — 46
, V(w) + e , ... . ,	+ 1)
S TT £1,i'	1—4e
Since 6 > 0 was arbitrary, this completes the proof.
Problems
9.1.	Consider the covering of IR2 consisting of T2i = IR2, J?2 = {(#i, ^2); ^2 >
Xi}, 1^3 = {(^1,^2);	< 0, x\ < —^2}, and the family of inward-
pointing vector fields g± = (cos(a), sin(o)), g2 = (0? 1), Рз = (—1, —1)- Let
g be the associated patchy vector field, see figure 9.12.
For every a E [0, 2тг] compute the set of forward and of backward solutions
from the origin.
9.2.	Let (f?#,^) be a single patch, and assume that the open set is
bounded, with smooth boundary. Using (9.7), give a detailed proof of
the inequality (9.38).
230
9 Patchy Feedbacks
Fig. 9.12. The patchy vector field in Problem 9.1.
Hint: since the boundary dQp is smooth, there exists p > 0 such that
the following holds (see figure 9.13). If a? € I? is any point such that
dist^x; < p, then there exists a unique closest point 7r(rr) E
with
In this case, one has
/ X	X ~
where n7r(T) is the unit outer normal to the set at the boundary point
7t(j:).
Fig. 9.13. The distance from the boundary is strictly increasing, along the trajec-
tories of the inward-pointing vector field gp.
9.3.	Give a detailed proof of the estimate (9.43).
Hint: consider first the case where	< p for all t E [0,т]. Prove
9.4 Nearly optimal patchy feedbacks 231
that (9.43) holds if wy = 0. Then argue that, since Фр is Lipschitz con-
tinuous with constant one, the presence of the perturbation wy can de-
crease the terminal value Фр(яг(т)) by an amount < Tot.Var{wzy}. In
the case where supfe[0 Фр(хр(£)) > p, define the time r' = sup{t G
[0, г]; Ф@(ху(1)) > p} and study the function t i—► Фр(жм(^)) on the
interval [г', т].
9.4.	Consider the control system (4.13). Assume attractiveness, Lyapunov sta-
bility, and let the assumptions of Theorem 4.3.1 hold. Prove that, on every
ball B(0, R) centered at the origin with radius R > 0, there exists a patchy
feedback stabilizing the system to the origin defined by a finite number of
patches.
Hint: Considering a level curve of a (local) Lyapunov function, one can
use a single patch in a neighborhood of the origin.
9.5.	Consider the control system (4.13) and assume that there exists a C1
function V such that V (0) = 0, V(x) > 0 for x 0 and V(x) oo when
|#| —> oo. Moreover, assume that
inf W(x) • /(.r,?z) < 0 for all x.
u£U
Prove that there exists a stabilizing patchy feedback.

10
Impulsive Control Systems
This last chapter contains an introduction to the theory of impulsive control
systems , described by equations such as
x = x, u, u).	(10.1)
Here x 6 IRn is the state variable, the control и ranges in a set U C IRm and
we assume that Ф is continuously differentiable w.r.t. all variables. When the
control и is absolutely continuous, its derivative й is an integrable function, de-
fined almost everywhere. A solution of (10.1) can thus be defined in the usual
Caratheodory sense, i.e. as an absolutely continuous function which satisfies
the differential equation at a.e. time t. On the other hand, when the control
и is discontinuous, its derivative must be interpreted as a distribution. This
gives to the system (10.1) an impulsive character, because the corresponding
trajectory can then be discontinuous as well. In this case, the previous concept
of Caratheodory solution is no longer applicable, and an alternative definition
is needed.
We shall focus on two main cases. First, we shall consider systems with
vector-valued controls, where the derivative of the controls enters linearly in
the equations:
m
X = f(t, X, 'll) +	•	(10.2)
1=1
Later, we study the case where the derivatives also enters quadratically:
771	ГП
x = f(t,x,u)-\-^^gi(t,x,u)Ui-\- hij(t,x,u)uiUj .	(10.3)
i=l	i,j = l
As a motivation for the above models, in Section 1 we introduce a class of con-
trolled Lagrangian systems, whose equations naturally have impulsive char-
acter. The theory of control of mechanical systems by means of moving con-
straints was initiated independently by Aldo Bressan and by Charles-Michel
234
10 Impulsive Control Systems
Marie, around 1980. The memoir [20] was motivated by problems of optimal
control for the ski or the swing, later studied in [21]. In [66] one can find a more
general geometric approach, also including some mechanical applications. The
connections between the two approaches were clarified in [24].
In order to define a generalized concept of solution for (10.1), which is
consistent with the classical one when и is absolutely continuous, a natu-
ral approach is to approximate the measurable control u(-) by a sequence
of smooth control functions	and study the limits of the correspond-
ing trajectories a;^(*) as p —> oo. For a given initial datum t(0) = x, two
possibilities may arise.
CASE 1: As v —► oo, the sequence of Caratheodory solutions x^y\-) converges
to a unique limit x(-) which does not depend on the choice of the approxi-
mating sequence. It is thus appropriate to define this limit x = x(^u) as the
generalized solution to the Cauchy problem, corresponding to the control u.
CASE 2: As и —» oo, the sequence x^ may diverge, or converge to different
limits depending on the choice of the approximating sequence , In this
case, additional information is needed in order to determine a well defined
solution.
The main results presented in this chapter can be summarized as follows:
•	For the system (10.2), if the vector fields gi satisfy a crucial commutativity
assumption, then CASE 1 occurs. Discontinuous trajectories, corresponding to
controls having jumps, can be uniquely defined as limits of smooth solutions.
By a suitable change of variables, the presence of the time derivative й can be
entirely eliminated from the equations. This reduces the system to a standard
form, without impulsive character. All the classical results of control theory
can then be applied to this equivalent system.
•	Still for the system (10.2), if the vector fields gi do not commute, there is no
canonical way to define the trajectory determined by a general discontinuous
control t I—> u(t). A unique trajectory 11—* x(t, u) can be still defined under ad-
ditional conditions, namely: (i) The control function tz(-) should have bounded
variation, (ii) At each time r where и has a jump, a continuous path joining
the left and right limits u(r—), 'u(r-h) of the control, should be specified. This
leads to the concept of graph completion of the control function tz(-), first
introduced in [16].
• In cases where the equation contains the square of the derivative of the
control, trajectories corresponding to discontinuous controls typically blow
up instantly. One thus needs to restrict the analysis to absolutely continuous
controls with square integrable derivative. The set of trajectories of (10.3) can
be described in terms of an auxiliary differential inclusion.
10.1 Mechanical systems controlled by moving constraints 235
The final section of this chapter is concerned with optimization problems
for impulsive systems. When the commutativity assumptions hold, we show
that a problem in Mayer form can be reduced to a standard optimization
problem for a suitable non-impulsive control system. This auxiliary optimiza-
tion problem can then be analyzed by well established techniques, such as the
Pontryagin Maximum Principle or the PDE of dynamic programming.
10.1 Mechanical systems controlled by moving
constraints
Consider a mechanical system described by N Lagrangian variables Qi,..., q^.
As usual, upper dots will denote derivatives w.r.t. time. Let the kinetic energy
be given by the quadratic form
1 N
T(q,q) = - ^2	(Ю.4)
t,J=l
and assume that the system is affected by external forces having components
Qz = Qi(t,q,q). The motion of the (uncontrolled) system is thus determined
by the equations
1=1.....N- d»-5»
There are two distinct ways in which an external controller can influence
the system, both physically meaningful, leading to substantially different sets
of equations.
•	The controller can apply additional forces, whose components (f)i(q. u) de-
pend continuously on the state q of the system. In this case, one obtains the
system of equations
d дТ ЭТ
ла? = а? + <Ш’’’, + л('!’“) i-ь.(юс)
This leads to a standard control system, where the right hand side depends
continuously on the control u(-).
•	The controller can directly prescribe the values of some of the coordinates as
functions of time. Namely, let N = n + m and assign the values qi(t) = Uj(t),
j = n + l,...,n + mof the last m coordinates as functions of time. Then the
evolution of the first n = N — m free coordinates will be determined by an
impulsive system of equations, linear or quadratic w.r.t. the time derivatives
of the control functions.
236	10 Impulsive Control Systems
Example 10.1 Consider a small child riding on a swing, pushed by his
mother. His motion can be described as a forced pendulum, say of length
p and mass m (see fig. 10.1, left). In addition to the gravity acceleration, the
child is subject to a force и = u(t) exerted by the mother who is pushing.
Denote by 0 the angle formed by the swing with a vertical line. The motion
is then described by the equation
mp0 — — mgp sin 0 + и .
(10-7)
Calling w = 0, we obtain a control system in standard (non-impulsive) form:
lu = — g sin 0 4—— и.
mp
Fig.	10.1. Left: a child pushed on a swing. Right: a boy on a swing, standing up
and down.
Example 10	.2 Next, consider an older boy riding on the same swing. By
standing up or kneeling down, he can change at will the radius of oscillation
(see fig. 10.1, right). We describe this new system in terms of two variables:
the angle 0 and the radius of oscillation r. The kinetic energy is given by
T(r, 0, r, 0) = — (r2 + r2#2),
(10.8)
while the potential energy is
V(r, 0) — mgr cos 0.
The control implemented by the boy amounts to assigning the radius of oscil-
lation as a function of time, i.e. r = u(t), for some control function u. Calling
L = T — V the associated Lagrangian function, the evolution of the remaining
coordinate 0 = 0(t) is now determined by the equation
10.1 Mechanical systems controlled by moving constraints 237
dfrL _ dL
dt дв dO ’
which in this case yields
2mr0r + mr20 = — mgr sin# .	(10.9)
Calling cu = 0 the angular velocity, we obtain an impulsive control system of
the form (10.2), namely
•	. g sint* 2w .	,inim
# = iv ,	v =-----------------u.	(10.10)
и и
Observe that in the above equation, the derivative of the control enters only
linearly.
Example 10.3 Consider a bead of mass m that can slide freely along a rigid
bar of negligible mass. The bar can rotate, with one end fixed at the origin
(fig. 10.2, left). Calling 6 the angle formed by the bar with a vertical line,
and with r the distance of the bead from the origin, the kinetic energy of
the system is again given by (10.8). However, assume that now we assign the
angle в = u(t) as a function of time, and regard the radius r as a free variable.
The motion is now governed by the equation
d_ dL _ dL
dt dr dr
r = rd2 + g cos в .
Introducing the radial velocity v = r, we thus obtain the impulsive control
system
r = v ,	v = g cos v 4- ru2 .
Notice that in this case, the equation contains the square of the derivative of
the control u.
We now describe a general framework for the impulsive control of La-
grangian systems. Consider a system described by N = n + m Lagrangian
coordinates, say Qi,...,^n, Qn+ь • •  Qn+m- Let (10.4) represent its kinetic
energy and and assume that the system is affected by external forces having
components Qi = Qi(t,q,q)- The motion of the (uncontrolled) system with
m + n degrees of freedom is thus determined by the equations (10.5). As-
sume now that a controller prescribes the values of the last m coordinates
gn+i,... ,gn+m as functions of time. This will be achieved by implementing
m frictionless constraints. Here frictionless means that forces produced by
the constraints make zero work in connection with any virtual displacement
of the remaining free coordinates	More precisely, call ФД£) the
components of the additional applied forces, needed in order to achieve the
equalities
238	10 Impulsive Control Systems
Fig. 10.2. Left: a bead sliding on a rotating bar. Right: two masses joined by a
rigid bar, with the first mass contrained on the t/-axis.
qn+i(t) = Ui(t) i =	(10.11)
The motion is now determined by the equations
= 7Ti +	+ #»(*)	i = l,...,n + m. (10.12)
dt dql dql
The assumption that the constraints are frictionless means that the following
identities hold:
ф1(^ = ... = фп(^) = 0.	(10.13)
Remarkably, there is no need to compute the remaining forces Фп+1 ,..., Фп+т
in order to completely determine the evolution of the system. Indeed, the
coordinates gn+1,..., qn+m are already assigned by (10.11). Of course, their
time derivatives
9n+1=ui(f), ... ,qn+m = um(t)
are determined as well. We now show that the evolution of components
g1,..., qn can be derived from the first, n equations in (10.12), taking (10.13)
into account. This is done in two steps.
STEP 1: In connection with the quadratic form (10.4), introduce the conjugate
moments
Pi = Pi(q,q) = qt- = 52Aj(9)qj-	(10.14)
i=l
Moreover, define the Hamiltonian function
- n+m
H(q,p) = -j BlJ (q) pipj ,	(10.15)
10.1 Mechanical systems controlled by moving constraints 239
where BIJ are the components of the (n+m) x (n+m) inverse matrix В = A x.
In other words,
1
0
£ BljAjk =
J=1
if i = k,
if i 7^ k.
STEP 2: Solve the system of Hamiltonian equations for the first n variables
9*
Pi
^p)
-	+ Qi(t,q,q)
i = l,...,n.	(10.16)
Notice that (10.16) is a system of 2n equations for q},.... qn,pi,... ,pn, where
the right hand sides also depends on the remaining components pJ? j =
n + 1,... , n + m. We can remove this explicit dependence by inserting the
values
qn+t=iii(t)	z = l,...,m,
<
Pj = Pj(Pl- •• ,Pn, Qn+1, . • • ,Qn+m)	j = n + l,...,n + m.
(10.17)
In (10.17), to express Pj as a linear combination of pi,... ,pn, gn+1,..., qn+my
we proceed as follows. Let C = (Cij) be the inverse of the rn x m submatrix
(B'^)i,j=n+l,...,n+m> SO that
if j = /с,
if j ± k,
j,k e {n + 1,.., ,n +m} . (10.18)
Recalling that p = Aq, q = Bp, we multiply by Cji both sides of the identity
n	n+m
6f = £^Pfc + £ в'кРк,
fc=l	Jc=n+1
then we sum over z = n + l,...,n + m. By (10.18), this yields
n+m	n+m n
Pj — £ Cjiq1 - £ ^2CjiBlkpk j = n+l,...,n + m. (10.19)
i=n+l	i=n+lfc=l
Inserting in (10.16) the values pn+i? • • • ,Pn+m given at (10.19), we obtain a
closed system of 2n equations for the 2n variables (71,...,gn,pi,... ,pn.
We now take a closer look at the equation of motion derived at (10.16)-
(10.17). For simplicity, we shall first assume that there are no external forces,
i.e. Qi(t,q,q) = 0. The extension to the general case is straightforward.
Fix an index i G {l,...,n}. Inserting the values (10.19) for the last m
components in (10.16) and recalling the definition of the Hamiltonian function
at (10.15), we obtain
240	10 Impulsive Control Systems
= E"., B'> p, + £”i", в» (£;_+”, cj( «< - Ей”, E*”., с„в'Ч)
(10.20)
Next, using again (10.15) and (10.19) we compute
(1	।	у^п-|-тп . 1	\ ()B^k
2	' 2^j=l 2_-rfc=n4-l '2 2^j,fc=n+l ) dqi PjPk
= -lY^J^Pipk
- e;=1 eSi (z:in+i ckhqh - e^:+1 e?=1 ckhB^
— lvn+m dBjk (\^n+m .h y^n+m v^n TDhf^ A
2^j,bn+l dqi \2^h=n+l ^jh4 Z^h=n+1 2^=1°jh^> 14)
x (ер:г+1 ckrqr - е:г+1 e;=i скгв^Ре).
(10.21)
Recalling that дп+г = щ, and that the matrices C(q) = (О) (7)) are
invertible, a direct inspection of the above equations reveals that:
•	The right hand side of (10.20) is always an affine function of the derivatives
fti,..., um .
•	The right hand side of (10.21) is an affine function of the derivatives
ill ,..., um if and only if
d (a}
———=0	for all i e {1,..., n} , j, к e {n + 1,..., n 4- m} .
uq1
(10.22)
Following [20], systems whose equations of motion are affine w.r.t. the
time derivatives of the control will be called fit for jumps. If there exists
a coordinate system for which the derivatives щ do not appear at all in the
equations, we say that the system is strongly fit for jumps. From the above
analysis it thus follows
Theorem 10.	1.1. The system described by (10.11)-(10.13) is “fit for jumps”
if and only if the external forces Qi are affine functions of the derivatives qJ,
j = n 4- 1,..., n 4- rn, and the identities (10.22) hold.
Theorem 10.	1.2. The system (10.11) (10.13) is “strongly fit for jumps” pro-
vided that the external forces Qi depend only on the variables t,ql, and more-
over the identities (10.22) hold, together with
B'\q) = 0
i e {1,..., n} , j € {n 4- 1,..., n 4- rn} .	(10.23)
10.2 Generalized trajectories for commuting vector fields
241
Example 10.4 Consider again the swing with variable radius of oscillation,
described by a Lagrangian system having kinetic energy (10.8). Assigning the
radial coordinate r = ufj) as a function of time, we obtain the control system
(10.10), which is linear w.r.t. u. This would follow from the above theorem,
observing that the gravity force Q = —gm sin в does not depend on r and that
the matrices A, В = A-1 here have the form
On the other hand, if we assign the angular coordinate 0 = u(t) as a function
of time, the remaining radial coordinate is determined by the equation
r = g cos и + r u2 .
Here the right hand side contains the square of the derivative of the control
function u. Of course, the assumptions of the Theorem 10.1.1 in this case are
not satisfied.
10.2 Generalized trajectories for commuting vector fields
The aim of this section is to provide a definition of generalized solution to an
impulsive control system, where the right hand side depends linearly on the
derivative of the control. As a preliminary, we observe that, by introducing
additional variables Xq = t and ;rn+i = tii,..., xn+m = um with equations
•ГО = 1 ,	^n+1 =	,	• • •	? *Гп4-т =	>	(10.24)
the system (10.2) can be put in the simpler form
x = F(x) -F Gi(x)ui.	(10.25)
г=1
The new vector fields F, Gi on 1RA (N — 1 + n + m) do not depend explicitly
on the variables t, u. For simplicity, we shall thus consider the Cauchy problem
determined by (10.25), together with the initial data
z(0) = x € JRn.	(10.26)
Our construction rely on a crucial commutativity assumption on the vector
fields Gi. Precisely, we assume that all their Lie brackets vanish identically:
[Gi,Gj] = 0	i,j = l,...,m.	(10.27)
We recall that the Lie bracket of two vector fields f, g is defined as
242
10 Impulsive Control Systems
lf,g] = (Dxg)f-(Dxf)g.
This is the directional derivative of g in the direction of /, minus the di-
rectional derivative of f in the direction of g. It is worth noting that the
assumption (10.27) is trivially satisfied if m = 1.
As usual, we shall impose some smoothness and sublinear growth condi-
tions on the vector fields F,Gi, which guarantee that the Cauchy problems
have a unique, globally defined solution.
(Л) The vector fields F, Gi are twice continuously differentiable. Moreover,
there exists a constant C such that, for all x € IRA and i = 1,.... m,
|F(x)|<C(l + |x|),
Gi(z)| <C(1 + H).
In the following, we shall use the notation
т t—> х(т) = (ехрт/)(гг).
to indicate the solution to the Cauchy problem
^-x(r) = /(x(r)),	x(0)=x.	(10.28)
ат
For clarity of exposition, let us first consider the case where no drift is
present, i.e. F(x) = 0. Our impulsive control system thus reduces to
х = ^С{(х)щ.	(10.29)
i=l
According to Frobenius’ Theorem A. 10.3 in the Appendix, there exists a
unique map
и (—> Ф(и) = (exp UiGi j (a?)	(10.30)
\ i=i /
from HV” into 1RA such that
Ф(0)=х,	^-Ф(и) = СДФ(и)), for all w = (ub. ,.,ит) e IRm.
(10.31)
We claim that, for every continuously differentiable control function и =
(ui,...,um) : [0, T] •—> lRm, the formula
x(t, u) = 0(u(f) - u(0))	(10.32)
provides a solution to the impulsive Cauchy problem (10.29), (10.26). Indeed,
this follows at once from the properties of the function Ф at (10.31):
x(0) = Ф(0) = x,
10.2 Generalized trajectories for commuting vector fields 243
I	'll' Q
= 12 аГф(и(0- u(0))ui(<) =
dt
= £^(ф(и(0-«(о)))м*) =
i=l	i=l
It is important to observe that, while the equation (10.29) involves the time
derivative of u, the solution formula (10.32) does not. Indeed, (10.32) makes
perfectly good sense for an arbitrary measurable function 1i—> u(t) G IRm. We
can thus use this formula as a definition of generalized solution to the Cauchy
problem (10.29), (10.26), for arbitrary measurable controls u(-). The previous
analysis shows that this reduces to the usual concept of solution when и e C1.
Next, we wish to extend the above construction to the more general equa-
tion (10.25), where a drift is present. We shall always rely on the commuta-
tivity assumptions (10.27), but we do not make any assumption on the Lie
brackets [F, G$].
Consider any C1 control function t i—> u(t), and let t i-► x(t, u) be the
corresponding solution of the Cauchy problem (10.25), (10.26). We seek a
representation formula which does not explicitly involve the time derivative
of u, and hence remains meaningful also for controls which are discontinuous.
Fix an arbitrary value u* = (u*,..., e IR"1 and consider the auxiliary
trajectory
t > £(£, u) = I exp ^2(tz* — Ui(tY)Gi j (a?(Z, u\).
\ i=l	/
(10.33)
We shall derive a differential equation satisfied by £. Writing Z)x(exp 52 ViGi)
for the differential of the map x h-> (exp 52 viGi)(x), consider the function
m
Dx ( exp ^(u* - Ui)G^ (a?)
t=i
(10.34)
where the differential is computed at the point x — (exp 52 (^г — u* )GJ(£).
In other words, given £ e IRA and и e IRm, the value of F*(£,n) is
obtained by
(i) computing the vector F at the point x = (exp 52 (u* —
(ii) pulling back this vector from the point x to the point £, using the
differential of the map
x и->
(10.35)
ПЫ =
Щ ~ ui
Theorem 10.2.1. Let the vector fields F,Gi be continuously differentiable,
satisfy (Jit) and the commutativity assumptions (10.27). Let и : [0, T] h-> IR™
244	10 Impulsive Control Systems
be a C1 control function, and let x(-,u) be the corresponding solution of the
Cauchy problem (10.25), (10.26). Then the function £ defined at (10.33) pro-
vides a solution to the Cauchy problem
4	=	(Ю-36)
C(0) = [eXpJ2« -иД0))сЛ (®).	(10.37)
X i=l	/
Proof. Since x(0) = x, the initial condition (10.37) is clear.
To prove (10.36), we shall use the identities
d
Ovt
m	\
exp^UjGj I (x) = Gi
j=i	/
(10.38)
Computing the time derivative of (10.33), one obtains
C = - E(=i Gi ((exPE£=l(«J - Ч?)^)^)) йг
+ [^(ехрЦ™,^ -u^Gj jer)] x
= ЕГ1 Gi ((exp£™ j(uJ - Uj)Gj)(x)) щ
+ ^(exp^jl^wj - Uj)Gj)(x) F(x)
+ ^(exp ^^(uy-Uj)Gjj(x) • £’"i Gi(x)tii
= [£>x(exp£”Lj(u* -u>(t))G>)(a:)] • F ((exp£™	-Wj)Gj)(£))
(10.39)
Indeed, the two summations involving the time derivatives щ cancel out each
other, according to Lemma A. 10.4 in the Appendix.
Remarkably, the differential equation (10.36) does not involve any of the
derivatives йъ. Indeed, as a result of the transformation (10.33), the contri-
bution of the terms G/Uz has been cancelled out. A Caratheodory solution
of (10.36) is thus well defined even when the control и is only measurable.
Moreover, as soon as £(t,u) is known, the value of x(t,u) can be recovered by
inverting (10.33):
m
x(t,u) = (exp^2(uj(t) -u*)Gi)(£(t,u)).	(10.40)
i=l
Motivated by the previous analysis, a concept of generalized solution can
now be introduced.
Definition 10.1 (Generalized solution for a commutative, impulsive
control system). Let the vector fields F, Gi satisfy the assumptions in The-
orem 10.2.1. Let и : [0, T] i—> IRm be a measurable control function. Then we
10.2 Generalized trajectories for commuting vector fields 245
say that x : [0, T] i—> IRл is a generalized solution of the impulsive Cauchy
problem (10.25)-(10.26) if, for some и* e IR™ the function £(•) in (10.33) is a
Caratheodory solution of (10.36)-(10.37).
Remark 10.1. If x(-) satisfies the above definition for one particular choice
of the constant w* € IR™, then the same is true with any other choice. In
other words, the above definition of generalized solution does not depend on
u*.
In the commutative case, the impulsive Cauchy problem (10.25)-(10.26)
can thus be solved in two steps.
(i) Choose any u* € IRm, for example u* = u(0), and compute the solution
t h-> £(t, u) to the standard control problem (10.36)-(10.37).
(ii) Recover x(t, u) from £(£,?/), using the formula (10.40).
The existence and uniqueness of generalized solutions is an immediate
consequence of this construction. If the vector fields Gi are C2 (i.e. twice
continuously differentiable), then the same is true of the exponential map
(10.35). The differential Dx(exp UiGi)(x) is thus a C1, and the function
F* in (10.34) is continuously differentiable w.r.t. both £ and u. Applying the
standard O.D.E. theory to the Cauchy problem (10.36)-(eq 10.315) we obtain
Theorem 10.2.2. Assume that the vector fields F. Gi satisfy (J^J together
with the commutativity assumption (10.27). Let и : [0, T] i—> IR™ be any
bounded, measurable control function. Then the impulsive Cauchy problem
(10.25), (10.26) has a unique generalized solution x(-,u), pointwise defined
forte [0,T].
Remark 10.2 As a special case, assume that the control zz(-) is piecewise
continuously differentiable, with jumps occurring at finitely many times 0 <
Ti < T2 < • • • < Tfc < T. To fix the ideas, let и be right continuous, so that
'u(tj) = lim^Tj_|_ u(t). In this case, a piecewise continuous map t i—> x(t),
continuous from the right, is a generalized solution of (10.25) provided that
(i) x(-) is a classical solution of (10.25) inside each open subinterval
Tj[ where и is smooth.
(ii) At each time т where и has a jump, the left and right limits of x(f) as
t —> т satisfy
m
®(r+) = (ехр^2(иДт+) - иг(т-))Сг) (ге(т-)).
i=l
(10.41)
Indeed, the function t £(t) remains continuous even at times r = Tj where
?i(-) has a jump. The representation formula (10.40) now yields
246
10 Impulsive Control Systems
m
х(т+) = (ехр^2(иДт+) -<)G,)(C(t))
2=1
m
= (ехр]Г(и,(т+) -Ui(r-))Gi^
2=1
m
•(expJ2(Ui(T-) -u*)G^(C(r))
= ( exp52(ui(r+) - Ui(r-))Gi) (x(r-)).
2=1
The next result shows that the concept of generalized solution is robust:
a generalized solution x(-, u) is obtained as the unique limit of Caratheodory
solutions x(-,tz^^), as the discontinuous control function и is approximated
by a sequence of more regular functions u^.
Theorem 10.2.3. Under the same assumptions on F Gi and и as in Theorem
10.2.2, let u^ : [0, T] »-> IRm, у > 17 be a sequence of absolutely continuous,
uniformly bounded control functions such that, as v oo,
u(">(0) ->u(0),	и^(Т) -> u(T), ||UM - u||L1 _ o.	(10.42)
Call t *—> Xy(t) the Caratheodory solutions to
xv = F(xI/) + Gi(xy)u^ ,
2=1
x(0) = X e IR* (10.43)
and let #(•) be the generalized solution of (10.25)-(10.26). Then, as у oo
one has
Xy(T) t(T) ,	[ \xy(t) -x(t)\dt -> 0.	(10.44)
Jo
Indeed, for a fixed u* G IRm, the solutions of
tW = F*(C,(t), uy^),
in
C(0) = (exp ^2«
2=1
(0))ф),
converge to the unique solution of the Cauchy problem (10.36)-( 10.37), uni-
formly on [0,Т]. The limits in (10.44) are thus an immediate consequence of
(10.42) and of the representation formula (10.40).
Example 10.5. Adding the variable х’з = u, the impulsive system (10.10)
takes the standard form
(х1,х2,хз) = 6^2, —	,0^-i-fo, _i^ = F(x) + G(x)u . (10.45)
\	хз /	\ хз /
10.3 The non-commutative case: graph completions 247
Since m = 1, in this case the commutativity assumptions are automatically
satisfied. Solving the differential equation x — G(x) we find
2
(expuG)(xliX2Jx3) = xi, 7-------—x3 + и .
\	\x3 4- uy J
Choosing u* = 1 as reference value for the control, from (10.33) we obtain
2
(Ci,6,6) = fci, ^'<3— 72, X3 + 1 ~u).	(10-46)
At this stage, it is useful to recall the physical meaning of our variables:
x\ = Ci =0 is the angle formed by the swing with the vertical direction,
X2 = 6 is the angular velocity, while x3 = и = r is the radius of oscillation. In
(10.46) we therefore have £3 = 1, while £2 = Or2 is the angular momentum.
The differential equation satisfied by £ can of course be recovered using the
general formula (10.34). However, it is more convenient to derive it directly
from (10.9). Observing that
^-(0r2) = вг2 4- 20rr = (-~—П— -	| r2 4- 20rr = -grsinO , (10.47)
at	\ r r I
we obtain for £ the non-impulsive system
(41,4г,4з) = (^, -pusin6, 0) = F*(6«)-	(10.48)
Notice how the terms involving r = й cancel each other out in (10.47). As
soon as the solution £(•) is computed, the evolution of the original variables
Xi can be recovered using the identity
(Ж1,®2,^з)(«) = (б(0,	(Ю-49)
x	CL I	/
10.3 The non-commutative case: graph completions
In this section we consider again the impulsive control system
rn
x = F(rr) + СУх)щ ,
i=l
x(0) = X ,
(10.50)
but we drop the crucial assumption (10.27) on the commutativity of the vector
fields Gi. In this case, Frobenius’ Theorem A. 10.3 cannot be applied and one
cannot construct any map и Ф(и) having the properties (10.31).
If u(-) is a discontinuous control, we can still approximate и by a sequence
of Lipschitz controls u^\ in L1 and pointwise almost everywhere. However,
248	10 Impulsive Control Systems
the corresponding trajectories will now heavily depend on the approxi-
mating sequence.
Example 10.6 Consider the impulsive system on IR2
= (1,0)^i + (0,^i)w2 = G](x)ui + G2(a:)w2 ,	(10.51)
with initial condition
(a?i,x2)(0) = (0,0).
(10.52)
Observe that in this case the vector fields Gi,G2 do not commute. Indeed,
their Lie bracket is [Gi,G2] = (0,1). Consider the discontinuous control func-
tion
(M2)(0=	(Ю.53)
One way to approximate the discontinuous control и by more regular con-
trol functions is as follows.
(u
(0,0)
(0, 1 + v(t - 1))
(Ki-1), 1)
(1,1)
if t e [o, i - i/i/],
if t € [1 - 1/p, 1],
if t € [1, 1 + l/И ,
if t € [1 + 1/p, 2].
(10.54)
The corresponding Caratheodory solutions of the Cauchy problem (10.51)-
(10.52) are computed as
(0,0)
(i/(f- 1), 0)
(1,0)
if t e [o, 1],
if t e [1, 1 + i/H,
if t e [1 + l/p, 2].
(10.55)
As v —> oo, the above sequence of trajectories converges (pointwise and in L1
to the limit trajectory
/	f (0,0)
if t e [o, 1],
if t e [1, 2].
(10.56)
Next, consider a second approximating sequence
	f	(0,0)	if	t e	[o, i - i/И,	
c)"1.4p))(<) = <	(1 +i/(t- 1), 0) (1, Hi-l))	if if	t e t e	[i -Ш i], [i, i + i/H,	(10.57)
	I	(1,1)	if	t €	[1 + 1/^, 2].	
The corresponding solutions (10.51)-(10.52) are now
1
«2И)(0 = *
(a?i,a;2)(f,u(‘/)) =
10.3 The non-commutative case: graph completions
249
(xi, Xz)(t, U^) = <
(0,0)
(l + i/(Z- 1), 0)
(1, -1))
(1,1)
if
if
if
if
t e [0, 1 - 1/p],
t e [i - i/p, i],
te [1, i + 1/И,
t e [1 + i/p, 2].
(10.58)
As » oc, in this second case the limit trajectory
( J(o>o)
(xl,X2)(t) = j L’jj
if t e [o, 1],
if t € [1, 2].
(10.59)
is still well defined, but different from (10.56).
The above example shows that, in the non-commutative case, the limit of
the approximating trajectories depends not only on the pointwise values of
u, but also on the way we approximate и by more regular controls. Observe
that in the first case the values of change from (0,0) to (0,1), and then to
(1,1). In the second case, the values of vary from (0,0) to (1,0), and then
to (1,1). This suggests that, in the noncommutative case, a unique trajectory
can be determined only if, at every time r where и has a jump, we specify
along which path the transition from u(r—) to u(t+) takes place. The next
definition, introduced in [16], makes this more precise.
Definition 10.2 (Graph completion). Consider any function и : [0,T] i—>
lRm. A Lipschitz continuous path 7 = (70,71, • • •, 7m) • [0, S'] ► [0, T] x IR™
is a graph-completion of и if
•	7(0) = (0,7i(0)), 7(5) = (T, ii(T)),
•	7o($i) < 70(^2) for all 0 < $i < s2 < S',
•	for each t € [0,T] there exists some s such that 7(5) = (£,
The path 7 thus provides a continuous parametrization of the graph of и in
the (t, u) space. In the case where и is piecewise continuous, at a time т where
и has a jump the path 7 must include an arc joining the left and right points
(r,u(r—)), (t,?i(t+)).
Lemma 10.3.1. A graph completion of и exists if and only if и has bounded
total variation.
Proof. l.Let 7 = (70,7i ? • • • ,7n) be a graph-completion of u. For any finite
sequence 0 = to < t\ < • • • < tk = T we can choose parameter values
0 = So<3i<---<Sfc = 5 such that 70 (sj) = tj. We then have
^2 MM -	MM “
3	3
|7(s)|ds-
Taking the supremum over all increasing sequences to < ti < • • • < tk, к > 1,
we obtain
250	10 Impulsive Control Systems
Fig. 10.3. Two different graph-completion of the same control function 11—> u(t).
Tot.Var.
|?(s)|c/s
ОС .
because 7 is Lipschitz continuous.
2.	Viceversa, assume that the control function u(-) has bounded variation. As
a consequence, и is a.e. continuous, with at most countably many points of
jump. Moreover, it admits left and right limits u(r—), u(r+) at every time
t. We shall construct a graph-completion of и by bridging each of its jumps
with a straight segment. For each т G [0, T], consider the total variation of и
restricted to the half-open subinterval [0, r]
V(r) = sup MM “ ufe-i)l •
0<tO<tl<...<<N<T
Set S = T + V(T) and define the path 7 : [0, S] [0, T] x HT as follows.
Observing that the map 11—> t + V(t) is strictly increasing, given s G [0, S]
there exists exactly one т G [0, T] such that т + V(т—) < s < т 4- V(т-h). We
consider various cases:
(i)	If s = г + V(r), we set 7(5) = (r, w(r)). This happens, in particular, if и
is continuous at r.
(ii)	If т + V(t—) < s < т + V(r), say s = 0[r + V(r)] + (1 — 0)[t + V(r—)]
for some 0 G [0,1], we set
7(5) = (t, 0u(r) + (1 — 0)u(r—)) .
(iii)	If т + V(r) < s < r + V(t+), say s = 0[r 4- V(t+)] + (1 — 0)[r + V(r)]
for some 0 G [0,1], we set
7(5) = (7-, ^z(r-h) 4- (1 — 0)u(r)^ .
10.3 The non-commutative case: graph completions 251
It is now easy to check that the above construction satisfies all conditions
required by the definition of graph completion. In particular, the map s i—> 7(5)
is absolutely continuous, being Lipschitz continuous with constant L = 1.
Remark 10.3. The graph completion constructed above is in a sense “canon-
ical”, assuming that the control function u(-) takes values in the Euclidean
space IRm. For certain applications, however, it is more natural to consider
controls taking values in an m-dimensional manifold AL In this case, per-
forming the above construction w.r.t. different charts will give rise to different
graph-completions.
Using graph-completions, we can now uniquely determine solutions to
the impulsive Cauchy problem (10.50). Let u(-) be a control function with
bounded variation, and let 7 = (70,71, • • • ,7m) be a graph-completion of u.
Consider the related Cauchy problem
d	ш
—y(s) = F(2/(s))7o(s) + 52Gi(j/(s))7i(s),	y(0) = x.	(10.60)
i=l
Since 7 is Lipschitz continuous, its derivative 7 = ^7(5) is a bounded mea-
surable function, defined for a.e. s e [0,5]. Therefore, by Theorems 2.1.1
and 2.1.3 the Cauchy problem (10.60) has a unique Caratheodory solution
s i—► ?/(s,7). Following [16] we now introduce
Definition 10.3 (Trajectory determined by a graph-completion). Let
£/(-,7) be the unique Caratheodory solution of (10.60). Then the (possibly
multivalued) function
t	= {?/(s,7); 7o(s) = t}	(10.61)
is called the generalized trajectory of (10.50) determined by the graph-
completion 7 of u.
Remark 10.4. It can be shown that the trajectory #(-,7) depends on the
path 7 itself, but not on the way it is parametrized. In particular, let 7 :
[0, 5]	[0, T] x be another graph-completion of и such that
7(0(s)) = 7(3)
s e [0,5]
for some absolutely continuous, strictly increasing ф : [0,5]	[0,5]. Then
the generalized trajectories ^(-,7) and #(-,7) coincide.
Example 10.6 (continued). For the discontinuous function и in (10.53),
consider the graph-completion 7 : [0,4] i-> [0, 2] x IR2 defined as
252
10 Impulsive Control Systems
(s, 0, 0)
(1, 0, s - 1)
(1, s-2, 1)
Дз-3, 1, 1)
if t € [0, 1],
if <€[1,2],
if t € [2,3],
if t € [3,4].
(10.62)
The generalized trajectory t h-» x(t, 7) is
*(<,?) =
f (0,0)
1(1,0)
if
if
*e[o, i][,
t e]i, 2].
(10.63)
Notice that this coincides with (10.56) for all t e [0, 2], t 1, while ж(1,7) is
multivalued. Observe that the curve 7 in this case is precisely the limit of the
graphs of the approximating functions u^.
A different graph-completion, following the construction in Lemma 10.3.1,
is achieved by bridging the jump at time r = 1 with one single straight
segment. This yields the path
7(s) = <
(s, 0, 0)
(1, s — 1, s- 1)
(s-1, 1, 1)
if t € [0, 1],
if <€[1,2],
if < e [2,3].
(10.64)
The corresponding trajectory of (10.50) is given by
(0,0) if
~ ((1, 1/2) if
< € [0. 1],
<€ [1,2].
(10.65)
Using graph-completions, trajectories of the impulsive system (10.50) can
be computed by solving a Cauchy problem for the O.D.E. (10.60). For detailed
results on the dependence of generalized solutions on the path 7, and on
approximations with smooth control functions, we refer to [16].
10.4 Systems with quadratic impulses
We consider here a control system where the right hand side contains also the
square of the derivative of the control function:
m	m
x = f(x) + '^gl(x)ul + У2 hij(x) Uitij .	(10.66)
i=l	i,j=l
Here the state of the system is described by the variable x € IRn, while u(t) G
IRm is the value of control. The upper dot denotes a derivative w.r.t. time. We
assume that the functions /, and hij = hji are at least twice continuously
differentiable and have sub-linear growth, so that
m	m
l/(x)| + £ |Pi(x)| + £ IM*)I < c(1 + И).
i=l	i»J=l
10.4 Systems with quadratic impulses 253
We remark that the case where these vector fields depend also on time and
on the control и can be easily rewritten in the form (10.66). Indeed, if
m	m
x = f(t,x, u) 4-	и) щ + hij(t, x, u) iiiUj ,
t=i	ij=i
it suffices to introduce the additional state variables xq = t and xn+j = Uj,
with equations (10.24). This yields a new control system of the form (10.66),
on the extended state space x 6
Notice that now we can no longer use discontinuous functions as controls.
Indeed, if some components of и have a jump at some time t, the right hand
side of (10.66) would formally contain the square of a Dirac delta distribution,
which is not bounded.
Example 10.7 Consider a ring sliding without friction along a rotating bar,
as in fig. 10.2. Assume that we want to assign the angle 0 = v(t) as a discon-
tinuous function of time, say
f в~	if 0 < f < 1,
- |0+	if i < t < 2.	(10.67)
Consider any sequence of smooth approximations t	^(t), with \\0y —0||ц
0 as z/ —> oo. The radial coordinate satisfies
r = g cosv + rv2.
Hence, if r(l—) > 0, as у —> oo we have
and hence
r(t, Vy) —* oo,	r(t,Vy) —> oo
for every t > 1.
As admissible controls, we shall thus consider a family of absolutely con-
tinuous functions ti(-) with derivative in L2. For example
L( = [и : [О, T] w IRm ; и absolutely continuous, /* |ti(£)|2 dt < k\ .
Jo	J
(10.68)
Given the initial condition
х(0) = ^,	(10.69)
a natural problem is to describe the set of all possible trajectories. The main
goal of the following analysis is to provide a characterization of the closure of
this set of trajectories, in terms of an auxiliary differential inclusion.
254	10 Impulsive Control Systems
It will be convenient to work in an extended state space, and use the
variable x = [ 0 | e lR1+n. For a given x. consider the set
\rx
where co(S) denotes the closed convex hull of a set S C JR1-1"71. Notice that
у i—> F(y) is a convex, compact valued multifunction on IR1+n, Lipschitz
continuous w.r.t. the Hausdorff metric, see Section A.7 and [5]. For a given
interval [0, S'], the set of trajectories of the differential inclusion
€ F(£(s)),
as
^(°) = Qt)
(10.71)
is a non-empty, closed, bounded subset of C([0, S]; IR1+n). Consider one par-
ticular solution, say s i—► x(s) =	defined for s e [0, S']- Assume
that T = Xq(S) > 0. Since the map s a?o(s) is non-decreasing, it admits a
generalized inverse
$ = $(£) iff xo(s)=t.	(10.72)
Indeed, for all but countably many times t e [0, T] there exists a unique value
of the parameter s such that the identity on the right of (10.72) holds. We
can thus define a corresponding trajectory
t i—► x(t) = x(s(t)) € IRn.
(10.73)
This map is well defined for almost all times t 6 [0,T].
To establish a connection between the original control system (10.66) and
the differential inclusion (10.71), consider first a smooth control function u(-).
Define a reparametrized time variable by setting
(10.74)
Notice that the map t s(t) is strictly increasing. The inverse map s i-> t($)
is uniformly Lipschitz continuous and satisfies
dt
ds
10.4 Systems with quadratic impulses 255
Let now x : [0, T] f-* IRn be a solution of (10.66) corresponding to the smooth
control и : [0, T] IRm. We claim that the map s i-> f(s) = I Л \ 1 is a
\ x\f\s)) /
solution to the differential inclusion (10.71). Indeed, setting
(10.75)
/	«o(s)	\
\/(x(s))vg(s) + E™ i S«(a:(s))«o(s)vi(s) + £™=1 hij(x(s))vj(s)vj(s)/
(10.76)
Hence (10.71) holds, because by (10.75)
m
ZX?<s) = i.
i=0
The following theorem shows that every solution of the differential inclu-
sion (10.71) can be approximated by smooth solutions of the original control
system (10.66).
Theorem 10.4.1. (Reduction of a quadratic impulsive system to a
differential inclusion). Let the vector fields f,gi,hij in (10.66) be Lipschitz
continuous. Let x : [0, S’] i—> H14~n be a solution to the multivalued Cauchy
problem at (10.70)-(10.71). Let the first component satisfy xq(S) = T > 0.
Then there exists a sequence of smooth control functions и” : [0, T] ь-> IRm
such that the corresponding solutions
s i—> ^(s) =
r(S) \
of the equations (10.75)-(10.76) converge to the map s > x(s) uniformly on
[0, S]. Moreover, defining the function x(if) = as in (10.73), one has
lim I \x(t) - xu(t)| dt = 0.	(10.77)
p-^oo Jo
Proof. By assumptions, the extended vector fields
9i =
256
10 Impulsive Control Systems
are Lipschitz continuous. Consider the set of trajectories of the control system
d	rn	m
£(«) = /vo + 52 ’-Wi + 52 ViV3 ’
i=l	M=1
(10.78)
where the controls Vi satisfy the constraints
^o(s) C [0,1],
m
52 = 1
s € [0, S'].	(10.79)
According to Theorem 3.4.2, the set of trajectories
s i-> £($) =	. ,^n)(s)
of (10.78)-(10.79) is dense on the set of solutions to the differential inclusion
defined by (10.70)-( 10.71). Hence there exists a sequence of control functions
s i—> v%s) = (l?q, ..., v^) (s), v > 1, such that the corresponding solutions
s tp(s) of (10.78) converge to £(•) uniformly for s e [0, S']. In particular,
this implies the convergence of the first components:
^(S) = / [^«</s^r0(S) = T.
Jo
(10.80)
We now observe that the “input-output map” v(-) •—> i(-,^) from controls
to trajectories is uniformly continuous as a map from L1 ([0, S]; R1+m) into
C([0,S]; IR.1 ^n). By slightly modifying the controls vy in L1, we can replace
the sequence vy by a new sequence of smooth control functions wy : [0, S] h->
IR1+m with the following properties:
Wq(s) > 0	for all s e [0, S], v > 1,	(10.81)
Cs
/ [wo (s)]2 cLs = T for all v > 1,	(10.82)
Jo
lim / |wI/(s) — vl/(s)| ds = 0.	(10.83)
Jo
This implies the uniform convergence
£m ||i(-,w1')-i(-)||C((0,S];]R—) = 0-	(10.84)
By (10.81), for each v > 1 the map
s^^(s)= / [w£(s)]2ds
Jo
is strictly increasing. Therefore it has a smooth inverse s = sy(t) We now
define the sequence of smooth control functions uy : [0, T] >—> IR™ by setting
10.5 Optimization problems for commutative impulsive systems 257

«ИО) .
®оИО
(10.85)
By construction, the solutions t xy(t\ uy) of the original system (10.66)
for the controls uy coincide with the trajectories t i—>	, xy)(sy (t)),
where xy = (л-q,	..., rr^) is the solution of (10.78) with control wy =
To prove the last statement in the theorem, define the increasing functions
*00 = [ [w(r)]2
Jo
t^s)= [S[w^r)]2dr
Jo
dr,
and let t и-> s(t), t >—> sy(t) be their respective inverses. Notice that each sy is
smooth. Moreover we have
d . .
ds
lim f |s(t) - s"^)! dt = lim f ' \t(s)
p-oo Jo
< 1,
ty(s)\ds = 0.
(10.86)
(10.87)
Using (10.86), we obtain the estimate
f \x(t) — #P(£)| dt = /* |a:(s(t)) — j:Z7(.s(/))| dt +
о	Jo
[ \xy(s(f))-xy(sy(t))\dt
S J°	fT
\x[s)-x^s)\ds + C- / |s(t)-Z(0|dt.
Jo
Here the constant C denotes an upper bound for the derivative w.r.t. s,
for example
(10.88)
where the supremum is taken over a compact set containing the graphs of all
functions tp(-). By (10.84) and (10.87), the right hand side of (10.88) vanishes
in the limit у oo. This completes the proof of the theorem.
10.5 Optimization problems for commutative impulsive
systems
This last section is concerned with optimization problems for the impulsive
system (10.25), assuming that the vector fields F, Gi on 1RA satisfy the
258	10 Impulsive Control Systems
sub-linear growth condition (♦) and the commutativity assumptions (10.27).
Assume that the control values u(t) are constrained within a compact set
U C IRm, and let the initial values
x(0)=x,	г/(0)=й	(10.89)
be assigned. The family of admissible controls will be denoted by
U = {и : [0, T] U, и measurable, tt(O) = й, T > 0} .
Given a continuous cost function J = J(t,x) and a closed target set S C
IR x IR77, we consider an optimization problem in Mayer form, with a terminal
cost, terminal constrains and variable terminal time:
min J(T, x(T, u)),	subject to (T, x(T))eS. (10.90)
Fix an arbitrary value if =	..., G IR m. According to Theorem
10.2.1, the trajectory t x(t,u) is then provided by the representation for-
mula (10.40), where £ is the solution to the Cauchy problem (10.36)-(10.37).
As we now show, the minimization problem (10.90) can be reformulated
as a standard Mayer problem for the auxiliary variable £ G IRA . We begin by
defining the functions
v '	( +oo if (t,x) f S.
= min Js (t, (.	(10.91)
\	i=l	/
Observe that the minimum in (10.91) is always attained (unless it equals 4-oc),
because Js is lower semicontinuous and U is compact. Next, we consider the
optimization problem
min J*(T, £(!»)	(10.92)
u£U
for the control system (10.36)-(10.37). As a consequence of the representation
formula (10.40), we obtain
Theorem 10.5.1. Let the vector fields F,Gi be continuously differentiable
and let the commutativity assumptions (10.27) hold. Assume that there ex-
ists at least one admissible trajectory of (10.60) that satisfies the constraint
(T, x(T\u)) G S. Then, the following are equivalent:
(i) The control function uEU is optimal for the Mayer problem (10.90), in
connection with the impulsive system (10.60).
(ii) The control u(-) is optimal for the optimization problem (10.92), in con-
nection with the control system (10.36)-(10.37). Moreover, the assignment
w — u(T) yields the minimum value of J$(T. •) in (10.91).
10.5 Optimization problems for commutative impulsive systems 259
If the function J* is sufficiently regular, the new optimization problem for
the variable £ can now be studied by standard techniques. For the case of a
swing, described in Example 10.2, optimization problems were discussed in
[61].
Problems
10.1.	A girl riding a swing is standing when the angle of the swing with a ver-
tical line is increasing and kneeling down in the opposite case. Hence if
0 < rmm < r < rmax then и = rmax when 9 • 9 < 0 and и = rmin when
9 • 9 > 0. Assume that she starts with zero velocity and an angle of тг/4
with respect to the vertical position.
(a)	Compute the position of the swing in time for 0 < t < тг.
(b)	Is the swing going to stop as t —► oo?
Hint: Refer to Example 10.2 and compute the control r(t). For simplicity,
use the normalization g = 1 in (10.10).
10.2.	Consider a mechanical system consisting of two point masses on a vertical
x-y plane (see figure 10.2, right). The mass at A is constrained to move
vertically on the i/-axis. The mass at В is connected to A by a rigid bar of
length p. The system is thus described in terms of two lagrangian variables:
the height h and the angle 9. Call	the two masses, and g the
gravity acceleration. Assuming that h(t) = u(t) is assigned as a function
of time (by frictionless constraints), write the equation determining the
motion of the remaining free variable 9(t}. Is this sytem fit for jumps ?
10.3.	For i = 1,2, consider functions gi(t, x,u) from IR x IRn x IRm into IRn.
Inserting additional variables as in (10.24), define the corresponding func-
tions C?i, G2 : IRA •-> RN, with N = 1 + n + m. Find conditions on <71,
which guarantee that the Lie Bracket [Gi,G2] : IRA >—► IRA vanishes
identically.
10.4.	Consider the impulsive control system:
±1 = #1 + axi ui
X2 = 1 + (—2^2 + ж2) щ + (я?? 4- ж2) W2,
Find conditions on a for the commutativity condition (10.27) to hold.
10.5.	Consider the impulsive control system:
±1 = й1 + X2
±2 = Щ + U2)
260
10 Impulsive Control Systems
and the control given by (10.53) for t 6 [0, 2]. Define the graph completion:
<^(s) = <
(S,(0,0))
(l,((s-l),(S
(«-1, (1,1))
0 < s < 1
1) + a(s — l)(s — 2))) 1 <s<2 .
2<s<3
For every a, compute the corresponding generalized trajectory xa.
10.6.	Consider a bead sliding along a bar as in Example 10.3 and consider the
control function:
u(t) = (l-t)°.
(a)	Determine the values of a for which a solution is defined;
(b)	Compute the trajectory for a = 3/4.
10.7.	Consider the impulsive control system (10.10) with u(f) 6 U = [1,3].
Let the initial conditions be 0 = 0, ш = — 1. Consider the optimization
problem
max cj(T) subject to 0(T) = 0 and 0 < T < 2тг.
Find the optimal control.
10.8.	Give a detailed proof of Remark 10.4. In other words, show that the
trajectory determined by a graph completion 7(-) is independent of the
way the path 7 is parametrized.
10.9.	Write a representation formula for the solution of the Cauchy problem
(10.25)-(10.26) assuming that all vector fields commute, i.e. [F, Gi] = 0,
[Gi, Gj] = 0 for all z, j = 1,..., m.
10.10.	Consider the impulsive control system (10.25), assuming that FGZ are
smooth and globally Lipschitz continuous, so that
rn
|F(x) - F(y)| + £ |СДх) - Gi(y)| < L |x - y\.
i=l
Consider a set of uniformly Lipschitz continuous control functions
A = {a : [0,T] IR”1; lu«W - «i(s)| < C - s| for all t,s € [0, T]} .
1=1
Call t x(t, u) the solution of the Cauchy problem (10.25)-(10.26). Prove
that the map tz(-) (—> ж(-,и) is Lipschitz continuous w.r.t. the C° norm:
max |rr(^,u) — rr(t,v)| < C• maX] |u(t)—v(t)|	for all u,v^A.
Hint: Using an equivalent norm
10.5 Optimization problems for commutative impulsive systems 261
lk(-)llt =
check that the Picard operator
о
[f(z(s)) +	Gi(#(s)) йДз)] ds
i=l
satisfies the assumption of the Contraction Mapping Theorem A.2.1. given
in the Appendix. Use an integration by parts to prove the Lipschitz con-
tinuity of Ф w.r.t. u.

Appendices
We collect here various definitions and basic results of mathematical analysis,
which constitute the main background material used throughout the book.
A.l Normed spaces
Let X be a vector space. A norm || • || on X is a mapping X i—► IR+ with the
following properties. For every vectors x,y € X and every real number A e IR
one has
•	Non degeneracy: ||.t|| > 0 whenever x 0,
•	Homogeneity: ||Aj:|| = |A| ||x||,
•	Sub-additivity: ||rr + y\\ < ||x|| + \\y\\.
Roughly speaking, the norm ||j:|| provides a measure of how big is the
vector x G X.
A subset А С X is convex if, given any two points x,x' € A and A 6 [0,1],
the convex combination A.r+(1 — X)x' also lies in A. From the above properties
it follows that, for every r > 0, the ball
Br = {x e X \ ||ж|| < r}
centered at the origin with radius r is convex. Indeed, if ||z|| < r, ||t/|| < r and
0 € [0,1], then the norm of the convex combination satisfies
||0ar+(l-0)y|| < ||0ж|| + Н(1-ЭД = 6*lkll + (1-0)lly|l < 0r+(l-0)r = r.
We recall that a sequence of points	is called a Cauchy sequence
if, for every e > 0 there exists an integer N large enough so that
||xm — яп|| < 8 whenever m,n > N .
264 A Appendices
Intuitively, this condition means that the elements of the sequence are getting
closer and closer to each other. We say that the normed space X is complete
if every Cauchy sequence converges to some limit point in X. A complete
normed space is called a Banach space.
Example A.l. Let X be the space of all polynomial functions defined on the
interval [0,1], with norm
IM -	11.(01
This is a normed space but not a Banach space. To see this, consider the
sequence of polynomials
Pn(o = E Ь 
k=0
This is a Cauchy sequence, but it has no limit in the space X. Indeed, as
n —> oo, the polynomials pn converge to the function et uniformly on the
interval [0,1]. However, this exponential function is not a polynomial and
does not lie in the space X. Therefore, our space X is not complete.
Several well known Banach spaces will be used throughout this book. We
recall here the main examples.
•	The finite dimensional space IR” with the Euclidean norm
M - y.r'f +.T^ + ••• + .7(2.
•	The space C°([a, 6]) of all continuous functions on the closed interval [a, 6],
with norm
ll/llc" = max |/(t)|.
tG[a,b]
•	The space C^Qa, 6[) of all continuously differentiable functions on the open
interval ]a, b[, with norm
ll/lld = sup |/(t)| + sup 0)1.
a<t<b	a<t<ib
•	The space Ьх([а, 6]) of Lebesgue integrable functions on the interval [a, 6],
with norm
J a
•	The space L°°([a, 6]) of Lebesgue measurable, essentially bounded functions
on the interval [a, b], with norm
A.2 Banach’s contraction mapping theorem 265
||/||l~ = ess- sup |/(t)| = inf { r > 0; meas{t e [a, 6]; | f(t)| > r} = 0
t€[a,6]
Given a subset J? C lRm, we say that a map f : J? i—► IRn is Lipschitz
continuous if there exists a constant L such that
l/(®) - /(y)l < L\x- y|
(A.2)
Vt, у 6 SI.
The space Lip(f?; IRn) of all these Lipschitz continuous mappings is a Banach
space with norm
II/IIlip = sup |/(x)| + sup /(y)l
x£.Q	1*^ У\
A.2 Banach’s contraction mapping theorem
One of the main interests of mathematicians is to solve equations. In many
cases, an explicit formula for the solution cannot be achieved. Still, one might
be able to prove that a unique solution exists, depending continuously on the
parameters that describe the problem. According to Banach’s theorem, this
is possible if the equation can be written in the form
z = <Z>(z,A)	(A.3)
and the map x i—> Ф(А, x) is a strict contraction for each given value of the pa-
rameter A. Indeed, the fixed point of this mapping can be found by a standard
iterative procedure.
Theorem A.2.1. (Contraction Mapping Theorem). Let X be a Banach
space, Л a metric space, and let Ф : А x X i—> X be a continuous mapping
such that, for some к < 1,
||Ф(А,Ж) - Ф(А, г/)|| < к ||х - у\\ ЧХ,х,у.	(А.4)
Then, for each А € Л there exists a unique fixed point x(A) € X such that
х(А)=Ф(А,х(А)).	(A.5)
The map A—>x(A) is continuous. Moreover, for any A 6 А, у € X one has
||у-х(А)||<-1- ||y-^(A,y)||.	(A.6)
1 Av
266
A Appendices
Fig. A.l. The approximating sequence, obtained by iteration.
Proof. Fix any point у € X. For each fixed A € A, consider the sequence (see
figure A.l)
Уо=У, У1 = Ф(А,?/о),	", y^+i = Ф(Х,уО, "•
By induction, for every 2/ > 0 one checks that
b+i - Ы < lli/1 - 2/01| =	||2/ - Ф(Х, y)||.	(A.7)
Since к < 1, the sequence y„ is Cauchy and converges to some limit point,
which we call x(A). By the continuity of Ф we now have
x(X) = lim y„ = lim Ф(Х,уи-х) = Ф (A, lim y^-x) = Ф(А,х(А)),
P —>OO	P—>ОО	\ P —>00	/
hence (A.5) holds. The uniqueness of a?(A) is proved observing that, if
= Ф(А,я?1), x2 = Ф(Х,х2),
by (A.4) it follows
||tfi -Я2Ц = ||Ф(А,Х1) - Ф(А, ж2)|| < к||Я1 -#2||-
The assumption к < 1 thus implies aq = x2.
Next, observe that (A.7) yields
V	V	1
11^+1 —2/11 <52ll%+i — 2/jll <	Пз/-ф(А’2/)11	||y-#(A,2/)ll •
j=0	j=0
(A.8)
Letting i/—>00 in (A.8) we obtain (A.6). To show that the fixed point x depends
continuously on A, let (An)n>i be a sequence of parameters converging to A*.
Using (A.6) with A = An, у = x(A*), we obtain
||x(A*) - x(An)|| < -T_ ||x(A*) - Ф(АП, x(A*))|| =
1 — К
= гЧ ИA*’ *(0 - ф(А-	A*))II •	(A-9)
1 rv
Since Ф is continuous also w.r.t. the variable A, the right hand side of (A.9)
tends to zero as n^oc. Hence rr(An)—*x(A*).
А.З Brouwer’s fixed point theorem 267
A.3 Brouwer’s fixed point theorem
This section contains a short proof of the classical fixed point theorem of
Brouwer: a continuous mapping from a closed ball В C IR71 into itself has at
least one fixed point. A corollary of this result plays a key role in the proof of
the Pontryagin Maximum Principle.
Brouwer’s theorem will first be proved in case of a continuously differen-
tiable map. Later, the regularity assumption will be removed by an approxi-
mation argument. With this in mind, we recall here a standard mollification
technique, to approximate an arbitrary continuous function by a smooth one.
As usual, we say that a function f is smooth, or equivalently f G C°°, if f is
к times continuously differentiable for every integer к > 1.
Lemma A.3.1. Let f : IR71 > IR/71, be a continuous map and consider a C°°
function ф : IR71 i—► [0,1] with compact support, such that
For each e > 0 define the mollified approximation
dy.
Then each f£ is smooth. As e —> 0, the functions f£ converge to f uniformly
on bounded sets.
We are now ready to prove the main result of this section. Here and in the
sequel (•, •) denotes the inner product of two vectors in IR/1. For simplicity, we
prove the theorem for the unit ball: the proof for any ball is obtained in the
same way.
Theorem A.3.2. (Brouwer). Let f be a continuous map from the closed
unit ball В C IRn into itself. Then there exists a point x* G В such that
X* = f(x*) 
(A.10)
Proof. 1. We first prove the theorem under the additional assumption that
f G C1 is a continuously differentiable mapping defined on the whole space
IR77, taking values strictly within the interior of the ball B.
Assuming that no fixed point of f exists, we will derive a contradiction.
For any x e IR/1, consider the ray originating from f(x) and containing x,
as in figure A.2. By assumptions, x f(x) and |/(z)| < 1. Therefore, the ray
crosses the unit sphere at exactly one point, which we call g(x). In other
words,
з(х) = /(x) + А(ж)(ж - /(a:)),
where A = A(or) > 0 is implicitly defined by the equation
268 A Appendices
Fig. A.2. A continuous map x g(x) whose values lie on the surface of the unit
ball.
1 = lffk)|2 = l/(z)|2 + 2A(/(z), ;г-/(я)> + Л2|я-/(ге)|2.	(A.11)
The assumption |/(#)| < 1 for all x implies that (A.11) has two distinct
roots, with opposite signs. Therefore, the map x X(x) is continuously dif-
ferentiable, and hence g G C1 as well. Moreover, we notice that
|a?| = 1
= ж.
Fig. A.3. The map and the wrould-be polynomial P.
2.	We now consider a family of maps, depending on an additional parameter
t e [0,1].
V’(t)k) = (1 — t)x + tg(x).	(A.12)
Clearly, is the identity mapping, while = g. Since f takes values
strictly inside the unit ball B, for each t G [0,1[ the construction of g implies
И < 1 => |</5(t)Cr)| < 1,
И = 1	=►	<f^(x)=X,
kl >1 => kt)WI>i.
(A.13)
А.З Brouwer’s fixed point theorem 269
3.	Let L be a Lipschitz constant for g, so that
\g(x) - g(y)\ < L\x - y\ 4x,y € B,
and choose т G]0,1[ such that
t G [0, t]
Lt 1
1-^2*
(A.14)
We claim that, for t G [0,r] the map is one-to-one C1 mapping from В
onto itself, whose inverse is also C1. Indeed, by the first condition in (A.13)
each maps В into itself. To show that is one-to-one and onto, let any
point у G В be given. By (A.14), for t G [0,t] the map
X » Ф^х) =	p(x)
(A.15)
is Lipschitz continuous with constant < |. Hence, by the Contraction Mapping
Theorem A.2.1, it has a unique fixed point x = x(y). The third condition in
(A. 13) implies that |я?(?/)| < 1. From (A. 12) and (A. 15) it now follows
<£(f)(z) = (1 - t)x + tg(x) = y,
proving that is bijection. The smoothness of the inverse function ip^ is
proved by checking that the Jacobian matrix V<^(t)(a?) = (1 - f)I — tVg(x') is
invertible when t G [0, т].
4.	For t G [0,1], consider the function
P(t) = / det(V(£>(t)(a:)) dx = / det ((1 — t)I + tS7g(x)) dx.	(A.16)
Jb	J в '	'
For each x G B, the determinant of an n x n matrix whose entries depend
linearly on a parameter t is a polynomial of degree < n. Therefore, the integral
P(t) itself must be a polynomial of degree < n.
This leads to a contradiction. Indeed, for t G [0, t], the transformation
is a C1 one-to-one map of В C IRn onto itself. By the formula for a change of
variables in a multiple integral, we compute
P(t) = / det(V^(t)(x))drr
J в
= vol (^(t)(B)) = vol(B).
For t > 0 small, the polynomial P(t) is thus constantly equal to the volume
of the unit ball В C IRn.
On the other hand, since |<P(i)(a?) | = |^(x)| = 1 for all x, when t = 1 we
find
P(l) = j det(V</(x)) dx = I Odx — 0.
J в	J в
270 A Appendices
No polynomial having such behavior exists (see figure A.3). This contradiction
proves the theorem, under the additional assumption that f is a C1 mapping
defined on the whole space lRn.
5.	To prove the theorem in the general case where f : В i—► В is only assumed
to be continuous, we first extend f to a map f defined on the entire space
IRn, by setting
7(x) = /(тг(а;)).
Here 7Г : IR” >—> В is the perpendicular projection on the unit ball:
if
'	[ ж/|ж| if |.r| > 1.
Next, we construct the approximations
A(x) = (l-e)£	dy'	(A17)
J1R7'	£	\ £ /
where ф is a smooth mollifier, as in Lemma A.3.1. For each e > 0, the smooth
function f£ maps the entire space IRn strictly into the interior of the unit ball
В because of the additional factor 1 — e present in (A. 17).
By the previous arguments, this map has at least one fixed point, say
Xe = /(ж£) e В .
By the compactness of the closed ball B, we can find a subsequence eu —> 0+
such that the corresponding fixed points converge: x£v x*. Since f£ f
uniformly on B, this implies
/(®*) = lim f£ (x£ ) = lim ix£u = x*,
eu—*0	е„ —>0
proving the theorem in the general case.
The above theorem has several far-reaching extensions. A few are described
below.
Corollary A.3.3. Let v:B» IR” be a continuous vector field defined on the
n-dimensional unit ball which points outward at the boundary:
(v(x),x) > 0 whenever |x'| = 1.
(A.18)
Then there exists a point x* e В with v(x*) — 0.
From a geometric point of view (see Figure A.4), the condition (A. 18)
means that the angle 0 between a unit vector x on the surface of the ball
В and its image f(x) satisfies |0| < 7r/2. In particular, this implies that
|x + v(x)| > 1 whenever |ar| = 1.
А.З Brouwer’s fixed point theorem
271
Fig. A.4. The outward-pointing condition.
Proof. Consider the perpendicular projection 7Г : IRn—>B, and define
f{x) = 7t(z) - t;(7r(a;)).
Since v is bounded, we can assume |v(a:)| < M for all x € B. Hence f maps the
ball Вм+i centered at the origin with radius M + 1 into itself. By Brouwer’s
theorem, f has a fixed point ж*. If |ж*| > 1, then (A.18) implies
(x*, x*) = (тг(а;*) — -и(тг(а;*)), x*) < (г—г,	< (я*,#*).
\М /
This contradiction implies |ж*| < 1, hence
v(x*) = x* — /(/) = 0.
Next, we show that in Brouwer’s theorem the unit ball can be replaced by
any bounded, closed convex set К C IRn.
Corollary A.3.4. Let К be any compact convex subset of IRn. Then every
continuous map f : К i—» К has a fixed point.
Proof. Choose a > 0 so large that К is contained inside the closed ball aB
centered at the origin with radius a. Let тг : НС h-► К be the perpendicular
projection onto К. This is characterized by the properties
7г(гг) € К, k(x) — ж| = min \y — x|
уек
for every x E IRn. By Brouwer’s theorem, the composed map g : aB i—► aB
defined by g(x) = /(тг(д:)) has a fixed point x*. Clearly, x* E K, hence
In our last application, we consider a mapping f from a compact set К
into lRn. Given a point wq 6 K, from information on how f behaves on the
boundary Ж we deduce the existence of a solution to the equation
= w0.
The key assumption here requires that each boundary point x E dK remains
sufficiently close to its image.
272
A Appendices
Corollary A.3.5. Let К be a compact, convex neighborhood of a point w G
IRn, with boundary dK. Assume that the continuous map f :K^ IRn satisfies
\f(y)-y\ < |y-w| forallyedK.	(A.19)
Then there exists x$ & К such that /(^o) — w.
Fig. A.5. A map whose range contains the origin.
Proof. By possibly performing a translation of coordinates, we can assume
w = 0. Choose a constant a > 0 so large that К is contained in the closed
ball aB centered at the origin with radius a. Let ir : IRn i—> К be the radial
projection on К. This projection (not to be confused with the perpendicular
projection!, see figure A.6) is defined by setting 7r(x) = x if x G К, while
7г(х) = Xx, Л = max{A' > 0; X'x G K}
in the case x^K. We claim that the continuous function
v(x) = /(тг(х)) x G aBn
points outward at each point x on the boundary of the ball aBn. Indeed,
assume |x| = a and у = тг(х) = Xx for some A G (0,1]. Since у G dK, from
(A.19) it follows
X{v(x),x) = X{f(Xx),x) = (f(y),y)
= (y,y) +	-y,y} > Ы2 -l№) - y\ • li/l > 0,
proving our claim. By Corollary A.3.3, there exists a point x* such that
v(x*) = 0. The result thus holds with Xq = 7r(x*).
A.4 A compactness theorem 273
Fig. A.6. The perpendicular projection and the radial projection on the set K.
A.4 A compactness theorem
This section contains a simple version of Ascoli’s compactness theorem, which
is used in several existence proofs. We say that a sequence /„(•) is uniformly
Lipschitz continuous if all f„ are Lipschitz continuous with the same constant
L, i.e.
|/p(t) — /p(s)l < L\t ~ sl for s G [a, b], v > 1.
Theorem A.4.1. Let (fv)v>\ be a bounded, uniformly Lipschitz continuous
sequence of functions from a compact interval [a,b] into IRn. Then there ex-
ists a subsequence fu> converging to some Lipschitz continuous function f,
uniformly on [a, b].
Proof 1. Let be a dense sequence of points in [a, b]. From the original
sequence /p, we extract a subsequence (/i,m)m>i such that /1,тп(£1) con-
verges. This is possible because f„ is bounded. Assume now that a sequence
(A,m)m>i has been constructed, which converges at the points ii,---
From this sequence we can extract from this further subsequence (A+i,m)m>i
which converges also at the point tk+i- By induction on к, we thus obtain a
double sequence of functions fk,m> k,m > 1, such that each subsequence
Лд, A,2, А,з> • • • converges at the points <1,^2, • • •,tk-
2. By the previous step, the diagonal sequence of functions /1,1,/2,2, /3,3, • • •
converges to some value f(tj) at each point tj of our sequence. Moreover, the
uniform Lipschitz continuity of all functions f„ implies
\f(ti) -	< L\ti - tj\ for all ij.
Therefore, the map f can be uniquely extended by continuity to a Lipschitz
continuous map, still called /, defined on the entire interval [a, b].
3. We claim that fmim—*f uniformly on [u,b]. Indeed, fix any e > 0. Choose
p so large that
274
A Appendices
ж	(a-2°)
2=1
where B(£,p) = [t — p,t + p\ denotes the ball centered at t with radius p.
Notice that (A.20) is possible because the points tj are dense on [a, b\. Choose
N so large that
f for all m > N,
о
For every m > N and every t e [a, b], since t e B(ti,e/(3L)) for some i < p,
we have the estimate
|/тп,т(*) - /(t)| < |/m,m(*) “ /m,m(^)| + |/m,m(^) “ /(Ml + |/(M “ /(01
This establishes the uniform convergence of the subsequence fm,m, completing
the proof.
A.5 Review of Lebesgue measure theory
This section collects some of the main definitions and results of Lebesgue
measure theory. For the proofs, see Folland [F] or Rudin [Ru 2].
As a preliminary, we recall that a function / : [a, b] i—► IRn is continuous if
at each point r there holds
f(r) = lim/(f) -
We say that the restriction of / to a subset J C [a, b] is continuous if, for
every т 6 J, one has
/(t) = t /(t).
A function / : [a, b] *—> IR, is lower sernicontinuous if at every point r it satifies
/(r) < liminf /(f) •
These concepts are illustrated in figure A.7.
1. (Measurable sets) A bounded set A C IRn is measurable iff, for every
e > 0, there exists V open and К compact such that К С А С V and
meas(V\K) < e. Here V \ К = {ж; x e V, x £ K} denotes the set-
theoretic difference of the two sets.
2. (Measurable functions) A function / : [a, b] IRn is Lebesgue measur-
able if, for every open set V C IRn, the preimage f~} (V) = {t 6 [a, b]; /(£) G
A.5 Review of Lebesgue measure theory 275
Fig. A.7. The function f is lower semicontinuous. Moreover, it is continuous re-
stricted to the compact set J.
У} is a Lebesgue measurable set. Every continuous function and every lower
semicontinuous function is measurable.
3.	(Properties valid almost everywhere) We say that a property P holds
almost everywhere (a.e.) on a set A if there exists a null set N such that
meas(A) = 0 and every point of A\N has the property P.
4.	(Pointwise converging sequences) Consider a sequence (fy)u>i of mea-
surable functions. If one has the pointwise convergence /Дх) —> f(x) for
a.e. point ж, then the limit function f is measurable.
5.	(Integrable functions) If f is measurable and |/(t)| < </>(t) for all t, for
some integrable scalar function </>, then f itself is integrable. Functions /, f
which differ only on a set of measure zero are identified. With this equivalence
relation, the space of all Lebesgue integrable functions f : [a, b] i—> IRn is
written L^a, b\; IRn). This is a Banach space with norm

6.	(Lebesgue Dominated Convergence Theorem) Consider a sequence
of integrable functions fy which converge to f at a.e. point in [a, b]. Moreover,
assume that there exists an integrable function such that and |/i/(t)| <
for all I/ > 1 and t G [a, b\. Then
>b	rb
f(t)dt= lim / fy(t)dt.
>	J a
7.	(Severini - Egoroff’s Theorem) Let fy : [a, 6] ]Rn be a sequence of
measurable functions which converges a.e. to f. Then, for every e > 0, there
exists a set J such that meas(J) < в and fy^f uniformly on [a, 6]\J.
276 A Appendices
8.	(Characterization of measurable functions) For any function f :
[a, b] i—> lRn, the following statements are equivalent:
(i)	/is measurable.
(ii)	For every e > 0, there exists a compact set J C [a, b] with meas([a,6] \
J) < £, and such that the restriction of / to J is continuous.
(iii)	There exists a sequence of disjoint compact subsets Jk C [a, 6] with
meas ( [a, b] \ Jk I = 0
\	fc=i /
(A.21)
and such that the restriction of / to each set Jk is continuous.
(iv)	There exists a sequence of disjoint compact subsets Jk C [a, b] satisfying
(A.21), such that the restriction of / to each set Jk is measurable.
9.	(Absolutely continuous functions) A function / : \a,b] »—> III7' is ab-
solutely continuous if, for every s > 0, one can choose <5 > 0 such that the
following holds. If ]$$, i = 1,... N is any finite collection of disjoint intervals
with total length
52 “ S<l S’
then

10.	(Differentiation vs. integration) If / is absolutely continuous on [a, b],
then its derivative /' is defined almost everywhere. Moreover, it satisfies the
Fundamental Theorem of Calculus:
/W = /(*o)+/ f'(s)ds for all t,toe[a,b].
Every Lipschitz continuous / defined on an interval [a, 6] is absolutely contin-
uous.
12.	(Lebesgue points) A point т € [a, b] is a Lebesgue point for an integrable
function f if

When this happens, we also say that / is quasi-continuous at r. In this case,
the integral function
A.6 Differentiability of Lipschitz continuous functions 277
F(t) = f f(s) ds
J a
is differentiable at the point r and one has the identity F'(r) = f(r)-
We say that a point £ is a Lebesgue point for the set A C IR if
lim — meas(A П \t — e, t + el) = 1.
2e
13.	(Lebesgue Theorem) Let f : [a, b] i—> IRn be integrable. Then almost
every point t E [a, b] is a Lebesgue point for f. If A is a measurable set, then
almost every t E A is a Lebesgue point of A.
14.	(Lyapunov’s Theorem on convex combinations) Let •• • ,	€
Lx([a, b];JRn) be integrable vector valued functions. Let 0or" ,	• [a, b] «—>
[0,1] be measurable weight functions such that	= f°r everY
Then there exist a partition of [a, b] into disjoint measurable subsets Jo, • • •, J к
such that
/•b i к	\	к л
/ b>w/(i)wU=£	(A-22)
\i=0	/ i=0Jji
Observe that, for each t E [a, 6], the integrand on the left hand side of (A.22)
is a convex combination of the vectors	., f^k\t) with coefficients
0O(*),..., Ok(t) € [0,1]. The right hand side can also be interpreted as a convex
combination, but where the coefficients are allowed to take only the two values
0 or 1.
A.6 Differentiability of Lipschitz continuous functions
We recall that a function f : IRn i—► IR™ is differentiable at a point x if there
exists a linear map Df(x) : IRn »—> IR™ such that
f(x + Л) - /(x) - D/(x) • h =
h™	|/i|
where | • | indicates the Euclidean norm in Rn.
We say that f is locally Lipschitz continuous if, for every compact set К C IRn,
there exists a costant Lk > 0 such that
|/(s) - /(у) I < LK |x - 7/1 for all X, у e K.
It is well known that a Lipschitz continuous function of a single real variable
is absolutely continuous, hence differentiable almost everywhere, see 10. in
Section A.5. The following theorem shows that the same is true for functions
of several variables.
278 A Appendices
Theorem A.6.1. (Rademacher) Let f : IRn »—> IRm be locally Lipschitz
continuous. Then f is differentiable almost everywhere.
Proof. Without loss of generality we may assume that f is Lipschitz contin-
uous, with a uniform constant L > 0. Moreover, considering separately each
component, it suffices to prove the theorem in the case m = 1.
1.	Let v € IRn with |v| = 1. For every x e IRn the map t »-> f(x+tv) of a single
real variable is Lipschitz continuous, hence differentiable almost everywhere.
Therefore, the directional derivative of f along v, written Dvf(x), exists for
a.e. x e IRn. In particular, letting v vary among the standard basis of IRA we
see that the vector
grad /(ar) =
is well defined for a.e. x E RA
2.	For every smooth function ф with compact support in Rn, we can write
x 4- tv) — f(x)

dx.
(A.23)
Letting t —► 0+ in (A.23), by the dominated convergence theorem (see 6. in
Section A.5) we obtain
t
=	[ ^~{х)ф{х)Фх
JjR." dxi
/ (v • grad /(х))ф(х) dx.
JlRn
Since in the above equality ф is arbitrary, for every v e IRn it follows that
Dvf(x) = v - grad f(x) for a.e. x € RA	(A.24)
3.	Let {гр}^>1 be a countable dense subset of the unit sphere of IRA Define
By = {x € IR'1; Dv„f(x) =	• grad f(x)}, В = Q By.
l/=l
By the previous step, one has
meas(lRn \ B) < meas(IRn \ By) = 0.	(A.25)
17=1
We claim that f is differentiable at every point x € B, and its differential is
Df(x) = grad /(x). Indeed, for every unitary vector v E IRn, define:
A.7 Multifunctions 279
Q(x,v,t) = № +	~ /M - V . grad Ц*).
Our claim will be proved by showing that
^lim Q(x, v,t) = 0
(A.26)
for every x € В uniformly w.r.t. v.
Now fix x 6 В and 5 > 0. Choose N so large that, for every unit vector v,
one can find z/ = L/(t>) G {1,..., N} such that
- ',“l *= 2(1 + ^)L 
(A.27)
Then by (A.27) we get:
|Q(x, v,t) - Q(x,v,,, t)| <
/(x + tv) - f(x + tv„)
t
+ |(v-Vp) - grad /(x)|
< L\v — v„\ + |grad /(x)| |г> - vv\
< (1 + Vn)L|u-u„| <
(A.28)
Notice that, by definition of В, (A.26) holds for every vu. Hence there exists
6 > 0 such that
p
|Q(x,v^,t)| < -
Vte]0,5], v =	(A.29)
Consider now w G IRn \ {0}, |w| < 6. Set v = w/|w|, so that w = tv with
t G]0, <5]. Using (A.28) and (A.29), we get
f (ж + w) - f(x) - w • grad f (a?)
|w|
= |Q(x,v,t)| < |Q(x,u,t) - Q(x,vt/,i)| + |Q(x,vv,t)| < | + j = e.
Since e > 0 was arbitrary the theorem is proved.
f{x + tv) - f(x)
—-----------------v • grad f(x)
A.7 Multifunctions
In the following, X is a Banach space with norm || • ||. The distance between
a point x and set А С X is defined as the smallest distance between x and
points in A, i.e.
d{x, A) = inf H# - a||.
a€A
The open E-neighborhood around the set A is denoted by
B(A,e) = {x € X : d(x,A) < e}.
280 A Appendices
The Hausdorff distance between two (nonempty) compact sets А, А! С X is
defined as
df^A, A') = max {d(x, A'),d(xf, A); x G A, x' G A'} .
Equivalently, c?h(A, A') can be defined as the infimum of all radii p > 0 such
that A is contained in the p-neighborhood around A' while A' is contained in
the p-neighborhood around A (see figure A.8).
dH(A,A') = infb>0; AcB(A',p) and A'cB(A,p'
Fig. A.8. The Hausdorff distance between the two sets A, A' is rnax{p, p'}.
A multifunction F from X to Y is a map that associates to each point
x G X a set F(x) C Y. We say that F is compact valued if F(x) is a non-
empty compact subset of У, for every x G X. The multifunction F is bounded
if all its values are contained inside a fixed ball В C Y. A multifunction F
with compact values is said to be Hausdorff continuous if
lim dH(E(y),F(x)) = 0 for every x G X.
y^x
We say that the multifunction F : X i—> Y has closed graph if its graph
Graph(F) = ((x, у); у G F(x)}
is a closed subset of X x Y. This condition means that, whenever x„ —► x,
у и —> у and у у е F(xjf) for every и > 1, then we also have the inclusion
У € F(x).
A. 7 Multifunctions 281
In the Introduction, we remarked that a general control system can alter-
natively be written in the form of a differential inclusion. The next lemma
shows that, under a natural assumption, the continuity of the function f in
(1.3) implies the continuity of the corresponding multifunction F in (1.5).
Lemma A.7.1. Let U be a compact subset o/IRm and let f : Rn x U i—> IRn
be continuous. Then the multifunction F defined by F(x) = {f(x,u) : utU]
is Hausdorff continuous.
Proof. 1. Consider any closed ball В C lRn. On the compact set В xU the
continuous function f is uniformly continuous. Therefore, for any e > 0, there
exists S£ > 0 such that
|ж — x'\<5£, \u —	=>	|/(x, u) — /(з/,г/)| < e	(A.30)
whenever x, x' € B, u,u' 6 U.
2. We now show that
dH(F(x), F(x'))<e	(A.31)
whenever x,x' e B, \x — x'| < 6£. Indeed, consider any point у 6 F(x). Then
у = f(x, u) for some и e U. By (A.30), the point y' = f(x', u) G F(x') satisfies
\y' — y\e. Hence F(x) is contained in the ^-neighborhood B(F(x'), s) around
the set F(x'). Inverting the roles of x, x', we obtain F(x') C B(F(x),s). This
proves (A.31). Since e > 0 and the ball В were arbitrary, the theorem is
proved.
Given a multifunction 11—» F(t) with non-empty values, a natural problem
is to construct a selection, i.e. a single-valued, function f such that f(t) € F(t)
for every t. If the sets F(t) are non-empty, then the existence of some selection
would follow simply from the axiom of choice in abstract set theory. However,
one would like to have a selection with some additional properties, such as
continuity or at least measurability. The following analysis will establish the
existence of a measurable selection, for a wide class of multifunctions.
On IRn, the lexicographical ordering is defined as follows. For any two
distinct points x = (a?i,..., xn) and у = (т/i,..., i/n), we write x -< у if one of
the following alternatives holds.
xi <yi,
Zi = У1
and X2 < У2 5
Я1 = У2 , ^2 = У2 , and x3 < y3 ,
ж i — y\,..., xn—i — Уп—i, and xn < yn .
282 A Appendices
Notice the analogy with the ordering of words in a dictionary.
We now show that every compact set К C IRn has one first point £ =
(aril,... , £n), w.r.t. the lexicographical order. Indeed, its coordinates can be
determined inductively as follows: Let {ei, ©2, ... , en} be the standard basis
of unit vectors in IRn. Define
£1 — min (ei , x) ,	K\ = {# e К , (ei, x) = £1}
£2 = min (©2 , x) ,
x^Ki
K-2 = {x G Kx , (e2, x) = £>}
Cn — niin , x) ,	— {x G Kn—i , (en , x) — Cn} •
xEKn-i
By induction, it is clear that all sets К D K\ D К 2 2 • • • D Kn are compact,
hence the components £i,...,£n are well defined. The set Kn contains the
single point £, which precedes all other points of К in the lexicographical
order.
If t t—> F(t) C HV? is a multifunction with compact values, we can now
define the selection t •—►£(£) € F(£), where £(£) is the first point of the
compact set F(t) w.r.t. the lexicographical order. Our next goal is to show
that this selection is measurable. A preliminary result is needed.
Lemma A.7.2. Let t 1—> F(t) C IRn be a bounded multifunction with closed
graph, defined for t in a closed set J C R. Then for each vector v € IRn, the
function
= min (v, y)
is lower semicontinuous.
Proof Fix any r 6 J and consider a sequence of points tk 6 J with tk r,
such that
lim inf = lim <p(tk).
t—►T, teJ	k—^OQ
Choose yk G F(tk) such that = P • У к- Since this sequence of vectors is
uniformly bounded, we can extract a convergent subsequence, say —> У-
Clearly у e F(t) because F has closed graph. We now have
— min (v, y) < (v, y) = lim tp(tk) — liminfcp(Z),
yeF(t) ’	k->oo	t-^T
proving the lemma.
Theorem A.7.3. Let t 1—> F(t) be a bounded multifunction with closed, graph,
defined for t € [a, b]. For each t, let £(t) € F(t) be the lexicographic selection.
Then the map t»—* £(t) is measurable.
A.8 Convex sets
283
Fig. A.9. Three examples of lexicographic selections.
Proof. Let £ = (£i,...,£n). To prove measurability, we need to check that
each component 11—> £Д<) is measurable. This is clear for the first component,
because
£i = min (ei , x),
xeF(t)
and the result follows from Lemma A.7.2.
By induction, assume that we already proved that the maps £i(-)> • • • ? £>()
are all measurable. Then there exist countably many disjoint compact subsets
Jk as in (A.21) such that the maps £i,..., are all continuous when restricted
to each J*.. Consider the multifunction t i—> Fj(t) defined as
Fj(t) = {x € F(t); (ee,x)=&(t),	€ = 1,2 ... j} .
By construction, each set Fj (t) is compact and non-empty. Restricted to each
set Jfc, the multifunction t Fj(t) has closed graph. Hence by Lemma A.7.2
the function
Cj+i(i)= min (ee,x)
is lower semicontinuous. Since the sets Jk cover the entire interval [a, b] except
a set of measure zero, this proves that £j+i is also measurable.
By induction on J, we conclude that all components of £(•) are measurable
functions.
A.8 Convex sets
Let A be any set contained in a vector space. We say that A is convex if, for
any two points x, xf e A, the segment that joins them is entirely contained in
A. Otherwise stated, A is convex if
Ox + (1 — 0)x' € A for every x, x' e A, 0 e [0,1].
284 A Appendices
The convex hull of a set A is the smallest convex set which contains A. It can
be represented as the set of all convex combinations of finitely many elements
of A, namely
{N	N	>
52 Ar > 1, Xi&A, Ai€[0,l], 5ZAi = 1f' (A-32)
i=l	i=l	J
An well known result of Caratheodory states that, for subsets of IRn, the
convex hull can be obtained by taking convex combinations of only n + 1
points.
Theorem A.8.1. (Caratheodory). If A C IRn, then
{n+l	n+1
У^ XiXi; Xi € A, Xi e [0,1], Xi = 1 > .	(A.33)
i=l	i=l	J
Proof. Fix any element x G co A. Let N be the smallest integer such that we
can write x as a convex combination of N points of A. We need to show that
N < n + 1.
1.	Assume, on the contrary, that N > n + 1. Consider the set
N
< (0 = (#i,..., On) ; У2 @ixi — x у
k 1=1
0i e [o, i],
Notice that (9 yields all possible ways of obtaining x as a convex combination
of the Xi. Since О is a compact subset of IR1V, it contains a first point 0 =
(#i,..., On) w.r.t. the lexicographical ordering.
We claim that 0j = 0 for some index j. This will imply that x is the convex
combination of N — 1 points, proving the theorem.
2.	If 0 < Oi < 1 for all i = 1,..., TV, consider the equations
N
OiXi = 0 ,
1=1
N
2> = °-
1=1
(A.34)
We regard (A.34) as a linear homogeneous system of n + 1 equations in the
N variables 0\,... ,0n- If N > n + l, this system has a nontrivial solution,
say A = (Ai,..., Xn) / (0,..., 0). Choosing s > 0 sufficiently small, the two
vectors 0 + eX and 0 — eX both lie in 3. Since 0 is the mid-point of the above
two vectors, it cannot be the first element of 3 in w.r.t. the lexicografical
order. We thus reach a contradiction, and the theorem is proved.
The above theorem has some important consequences. In particular:
Corollary A.8.2. The convex hull of a compact set К C IR77 is compact.
A.8 Convex sets
285
Proof. Let An be the standard n-dimensional simplex, defined as
An =	= (0q, • • * , вп); 0j = l, Gi > 0 for every i j C IRn+1.
i=0
(A.35)
Then co К is compact, being the image of the compact set К x • • • x К x An
through the continuous mapping
n
(•ГО? ’ * ' 1	I > GiXi .
i=0
A useful technique, that we already encountered in the proof of Brouwer’s
theorem, is the continuous projection of points in IRn onto a convex subset.
The following theorem collects the basic facts about perpendicular projections.
Theorem A.8.3. Let A C IRn be a closed, convex set. Then, for every x E IRn
there exists a unique point 7r(x) E A such that
|тг(ж) — ж| = inf |a — ж|.	(A.36)
a£A
The perpendicular projection x 1—> 7r(a;) has the following additional properties:
(x — тг(х), a — 7г(ж)) < 0 for all x 6 IRn, a E A.	(A.37)
|тг(д?) — 7г(я')| < |z — a/| for all j?,j?'ElRn.	(A.38)
Proof. 1. To construct the projection, for any given x E lRn consider a se-
quence of points ym E A such that
lim \ym — x\ = inf |a — ж|.
тп—-ос	aCA
This sequence is clearly bounded, hence by compactness we can extract a
subsequence converging to a limit point y. We then set у = 7г(я). Clearly
7г(а?) E A because the set A is closed.
2.	To prove the uniqueness, assume that y\,y2 E A are distinct points such
that
- ж| = |т/2 - ж| = inf |а - х|.
а^А
Then, taking the convex combination у =	and observing that у— x
and y2 —yi are perpendicular to each other (see figure A. 15), using Pythagoras’
theorem we obtain
I |2 I 12	|У2 ~ У112	. p 1 j
\y - Я = \У1 - я----------- я
4 aeA
286 A Appendices
3.	To prove (A.37), consider the segment joining 7r(z) with a. Points on this
segments can be parametrized as y(t) = тг(ж) + (a — тг(ж)), for t G [0,1]. We
now compute
-*i2
dt
= 2 (а — 7г(яг), 7г(я:) — x) .
t=o
(A.39)
By convexity, y(t) G A for all t G [0,1]. Since the distance ||?/(t) — ж|| attains a
minimum at t = 0, the right hand side of (A.39) must be non-negative. Hence
(A.37) holds.
4.	Finally, we establish the contraction property (A.38). By the previous step
we have
(x — 7r(z) , 7г(я/) — 7г(я)) < 0 ,
(x' — тг(х') , 7г(х) — 7г(я/)) < 0 .
Therefore
(х — х', X — х') — (тг(х) - 7г(я/) , 7г(х) — 7г(я/)) =
=	— xf) + (тг(ж) — 7г(д/)) ?	— х') ~ W37) “ 7Г(379)^ — 0 •
A point х G A is an extreme point of a convex set A if it cannot be
obtained as a strict convex combination of two distinct points of A. More
precisely, x G A is an extreme point if
x = Xxi T (1 — A)#2 for some , a?2 € A 0 < A < 1 implies Xi = x% = x.
Theorem A.8.4. (Krein-Milman). Let E be a Banach space such that its
dual E* separates points. Then every compact convex set К С E contains an
extreme point.
Fig. A. 10. The extreme points of a set A and the separation of two convex sets
К, K' by hyperplanes.
A.8 Convex sets 287
Remark A.l (row and column vectors). Up to now, by a vector x e lRn
we usually meant a column vector. In particular, the state x(t) G IRn of a
system and the value u(t) G IRm of a control are always regarded as column
vectors. However, in some cases it is useful to work also with row vectors.
A linear mapping IRn i—► IR can be conveniently identified with a row vector
p = (p1?...,pn), writing
p-x = (pi,...,pn)
n
Pi^i •
2=1
Gradients of scalar functions ф : IRn IR will always be regarded as row
vectors, having the form
дф дф \
dxi ’	’ dxnJ
The next lemma is concerned with the separation of two compact convex
sets by means of a hyperplane.
Lemma A.8.5. (Separation of convex sets) Let K,K' C IRn be disjoint,
closed, convex sets, with К compact. Then they can be strictly separated by a
hyperplane. More precisely, there exists e > 0 and a unit row-vector p € IRn
such that
min p • v > sup p • v' + e.	(A.40)
Proof. 1. Choose two sequences of points x„ G К, x'y G K' such that
lim \xy - x'y\ = inf {\y - y'\- у G K, y' G K'}.
I/—>OC
Since К is compact, the sequence xy is bounded; hence the sequence x'y
is bounded as well. By taking a subsequence, we can assume xy-+x € K,
xy^x' G K'. This of course implies
|jr — xf| = min{|2/ - t/'|; у € К, у G К'}.	(A.41)
2. We claim that the conclusion of the theorem is satisfied by
Indeed, by (A.41), x' is the perpendicular projection of x on Kf, while x is
the perpendicular projection of x' on K. Therefore, (A.37) implies
p • x = min p • v,	p’x' = max p • v'.	(A.42)
vGK	v'EK1
Our claim is now an immediate consequence of (A.42).
288 A Appendices
Lemma A.8.6. (Support hyperplanes). Let К C IRn be closed, convex,
with boundary dK. Then every boundary point w G Ж admits a support
hyperplane. More precisely, there exists a unit vector p such that
p • w — max p • y.
(A.43)
Proof. Choose a sequence of points xv К with xu—+y. Let у у = 7r(xp) be
the perpendicular projection of Xy on K. Observe that y^^w as i/—>oo. By
possibly taking a subsequence, we can assume that there exists a unit vector
p such that
.	- у у
Ру = ;--------;—*p as V~>0C.
- Уу\
Recalling (A.37), we have
Pv • Уу > Ру ’ У for all у e К, и > 1.
For every у € К, this yields
p • w = lim py • w = lim pu • yu > lim py • у = p • y,
y—>OO	У—ЮО	y—>OQ
proving the theorem.
A.9 Convex cones
In this section we study a special class of convex sets, namely convex cones.
We recall that a set Г C lRn is a cone if
Л x e Г whenever x 6 Г, A > 0 .
Let V C IRn be any set of vectors. The span and the positive span of V are
defined respectively as
{N
У A^;
г=1
{N
57 Аг^;
i=l
Now consider a closed set S C lRn
tangent cone to S at the point £ as
N > 1, Vi e V, Xi e IR
N > U eV, Xi > 0
and fix any point £ 6 S. We define the
Ts(O = {<> =
lim e 1dU + ev, S) = 0
e—o+	v	7
A.9 Convex cones 289
Example A.2. Consider the set (see figure A.11)
S = {(#1,^2) C IR2 ; sq > 0, X2 > 0, xj +	— 1}
Then the tangent cone to S at the point £1 = (1,0) is
^s(Ci) = {(2/ь?/2) C IR2 ; yi < 0, У2 > 0} .
The tangent cone to S at the point £2 = (1/a/2 , l/\/2) is
^>(£2) — {(У1,Уз) E IR2; yi + У2 < 0} .
Fig. A.11. Examples of tangent cones.
We shall be mainly interested in the case where the set S is defined in
terms of finitely many equations or inequalities (see figure A. 11)
S = {.т; ^(ж) = 0, i =	k},	(A.44)
S+ = {x\ фо(х) > фо (£), ФЛХ>) =	* • • ,	(A.45)
where </>o, • • • , Фк are C1 functions, £ G S'. The next lemma is an immediate
consequence of the implicit function theorem.
Lemma A.9.1. (Representation of tangent cones). Let фо,’",фк •
IRn и-> IR. be continuously differentiable. Assume that, at the point £ G S,
the gradients V</>o(£)>”’ ^Фк(Я) are linearly independent. Then the tangent
cones at £ to the sets (A.44), (A-4$) are, respectively:
Ts($ = {v, V0i(£)-v = O, i =	(A.46)
Ts+^) = {v; V</>0(£) • v > 0, Wf(£H = 0, г = !,•••,&},	(A.47)
290 A Appendices
Lemma A.9.2. Under the same assumptions of Lemma A.9.1, a vector p
satisfies
p • v > 0 for all v e Ts+ (£)	(A.48)
if and only if p can be written as a linear combination
к
р = ^Х^ф^)	(A.49)
i=O
with Aq > 0.
Proof. Indeed, if (A.49) holds with Aq > 0, by (A.47) we have
p • v = AqV0o(£) • v > 0 for all v G (£).
To prove the converse, assume that (A.48) holds and choose additional smooth
functions 0^+1, • • • , фп-i •	•—> IR such that the set of gradients
Wo(£), V0i(e), ••• ,
forms a basis	in IRn.	Let {do,	• • • , bn-i}	be the dual basis, so	that
_ ,	, fl	if i	= j ,
 ь, - 10	,f . .
Observe that,	by (A.47), the vectors do, ibfc+ь • • •	, ±dn_i	all lie inside Ts+ (£).
Write the vector p as a linear combination
n—i
i=0
where Xi = p • bi, i = 0,1, • • • , n — 1. If Aq <0 then p • do < 0, in contrast with
(A.48). If Xi 0 for some i > k, then either p • bi < 0 or p • (—dj < 0. In any
case, this contradicts (A.48).
We say that two cones Г, Г' are weakly separated if there exists a unit
vector p such that
p • v < 0, p • v' > 0, for all v G Г, v' G Г'.
The next two lemmas on the separation of tangent cones play a key role in
the proof of Pontryagin’s Maximum Principle.
Lemma A.9.3. Let Г = span+(V) for some set V C IRn, let Г' = Ts+(£),
with Т$+,£ as in Lemma A.9.1. Assume that the cones Г,Г' are not weakly
separated. Then there exist finitely many vectors Vo,--- ,vn € V such that
span* {t>o, • • • , тдг} is not weakly separated from Г'. Moreover, there exists
e > 0 such that, if
\wi - vj < e i = 0, • • • ,N,
then the cones span+{wo, • • • , w^} and Г' still cannot be weakly separated.
A.9 Convex cones 291
Proof. 1. Let {vj}j>o be a sequence everywhere dense in Г. Assume that, for
all v > 1, there exists a unit vector py such that
• Vj < 0 j = 1, • • ♦ , y, pu • v' > 0 for all v' G Г'.	(A.50)
By taking a subsequence, we can assume py-^p for some unit vector p. By
(A.50) and the density of the sequence {vj} it follows
p • v < 0 for all v G Г, p • v' > 0 for all vr G Г',
a contradiction. Hence, for some A, span+{vo, • • • ,vn} is not weakly sepa-
rated from Г'.
2. The proof of the second statement is similar. If no e > 0 with the desired
property exists, then one can construct sequences (v(0>l/), • • • , v^yf) and unit
vectors py such that for every у > 1, v’ G Г' and i = 0, • • • ,7V, one has:
|^i,p -	< p	pv-v' > 0.	(A.51)
Taking a subsequence, we can assume Pv—tp. Then (A.51) implies
p • Vi < 0 p • v1 > 0	for all v1 G Г', i = 0, • • • , N,
showing that the cones span+{t?o? • • • ,vn} and Г' are weakly separated, a
contradiction.
Lemma A.9.4. Let Г = span+{vo, • • • , v^} C IRn and let Г' = T$+(£), with
S+,£ as in Lemma A.9.1. Define the unit simplex An as in (A.35) and let
X : [0, e] x An i—> IRn be a continuous map such that
in,
г—0
uniformly for 9 G An- If Г and Г' are not weakly separated, then there exists
some £, 9 such that
фв(Х(ё, 0)) > 0o(^),	0)) = 0, i = 1, • • • , к. (A.53)
Remark A.2.
In connection with the constrained optimization problem
max |</>o(X(e,0)); (e,0) e [0,e] x 4N, X(e,0) 6 S’} ,	(A.54)
the conclusion of the lemma states that the optimal value for (A.54) is strictly
larger than <fo(£) (see Figure A.12).
292 A Appendices
Fig. A. 12. If the cones Г and Г' are not weakly separated, then £ is not an optimal
solution to the constrained optimization problem (A.54).
Proof, of Lemma A.9.4- It is clearly not restrictive to assume that </>o(£) = 0.
Assume that the cones Г and Г' are not weakly separated. Consider the
vector-valued function ф = (0o,	• • •, <^fc) ’	Its (fcfl)xn
Jacobian matrix will be denoted by \?ф = (V0o, • • • , V</>&). Moreover, define
the cone
Г* =span+{V^ Vi; i = 0, ••• ,JV} C IRfe+1.
1.	We claim that the vector
w = (1,0,-•• ,0) e lRfc+1
lies in the interior of the cone Г*. Indeed, if w is not in the interior of Г*, by
Lemma A.8.6 there would exist some unit vector p = (po, • •  ,р&) such that
pw=maxpp.	(A.55)
yer*
Since Г* is a cone containing the origin, both quantities in (A.55) must be
zero, i.e. po = 0. Now consider the linear functional on IRn
к
V p • V0 • V =	‘ V'
i=l
By (A.47) and the definition of Г*, this functional is identically zero on Г'.
On the other hand, by (A.55) it is non-positive on Г. The two cones Г, Г' are
thus weakly separated. This contradiction proves our claim.
2.	By the previous step, there exist к + 2 vectors, say Wq, • • • , Wk+i 6
such that
A.9 Convex cones 293
w E int co{wq, • • • , Wfc+i},	(A.56)
TV
Wi — CijV0 • Vj	г = 0, • • • ,fc-h 1,	(A.57)
j=0
for some nonnegative coefficients Cij. Observe that it is not restrictive to
assume that > 0 for all i,j. Define the map h : (e, $0,..., $fc+i) «—>
(б,0о,- • • ,#tv) by setting
(\ -1
\	fc+i
57 I	= - 57 ^iCij 
г,£	J	i=0
3.	Define a new map (б,$) i—> У (б,??) from [0,f] x Ak+i hito IRfc+1 by setting
Y(e,,..., i?fc+1) = ^(X(A(6, tf0, • • •, tffc+i))) •
To prove the lemma, it now suffices to construct a point (б, $□,..., tfk+i), with
e > 0, such that
Г(ё, t?o, • • •, dk+1) = (c, 0,..., 0) € IRfc+1.	(A.58)
Observe that, by (A.52) and (A.57), we have
lim	(A.59)
€	г=0
uniformly for 0 6 Дк+i • By (A.56), the map
fc+i
0 : ($o, • ' , tffc+i) ^2 ^iWi
i=0
is a linear bijection between the unit simplex Дк+i and a convex neighborhood
К of w, namely К = co {wq, ..., Wk+i} • Call 0-1 : К i—> Дк+i its inverse
mapping. For б > 0, define the map f(: К Rfc+1 by setting
As б —> 0+, we have
lim /c(w) = w
uniformly for w € K. In particular, this holds for points w € dK on the
boundary of К. Since w = (1,0,..., 0) € int К, we can apply Corollary A.3.5
and deduce the existence of б > 0 sufficiently small and w € К such that
/?(w) = w.
Hence (A.58) holds with d = 0-1(w). This completes the proof.
294 A Appendices
A. 10 Lie brackets and Frobenius’ theorem
In the following, we shall use the exponential notation 0 i—> (exp0/)(a:) to
denote the solution of the Cauchy problem
dW	,	ZAK
— = /(w),	w(0) = x.	(A.60)
(1U
Moreover, for a fixed 0, we shall denote by (exp#/)* = Z)x(exp^/) the Jaco-
bian differential of the map
x (expOf)(x).
(A.61)
Given two smooth vector fields f and g on IRn, their Lie bracket is the
vector field defined as
[f,5] = Dxg  f - Dxf-g,
In other words, the Lie bracket [/, <?] is the directional derivative of g in the
direction of /, minus the derivative of f in the direction of g. The following
lemma provides various equivalent constructions of Lie brackets.
Lemma A. 10.1. (Characterization of Lie brackets). The Lie bracket can
be equivalently characterized as
[/, 5] = lim	[(exp eg)(exp e/) - (expE/)(expsp)j,	(A.62)
[/,g] = liml [(exp(-eff))(exp(-£/))(expE5)(exp£/)],	(A.63)
[f,g] = lim |[(exp(-e/))tp(exps/) - p],	(A.64)
Proof. In the following estimates, the Landau symbol o(s2) indicates a higher
order infinitesimal w.r.t. б2, i.e. a quantity such that o(£2)/e2 —> 0 as e —> 0.
1.	To prove (A.62), we observe that
(exps/)(x) = x + ef(x) + — (Dxf)  f(x) + o(e2),
(expc5)(expE/)(x) = x + ef(x) + ^-(Dxf)  f(x)
+eg(x) + e2(Dxg)  f(x) + у(Dxg) • g(x) + o(e2).
(A.65)
Interchanging f and g in (A.65) and subtracting the results we obtain
(ехре<?)(ехрг/)(х) - (exp ef) (exp eg) (x) = e\(Dxg)  f - (Dxf) • д') + o(e2).
Hence (A.62) follows.
A. 10 Lie brackets and Frobenius’ theorem 295
2.	Concerning (A.63), we have
lim£_o e~2 [(exp(-Eg)) (exp(-e/))(expep)(expe/)(x)j
= lime_0 (ехр(-£^))Дехр(-£/))ф-
•lime_o e~2 |\ехрез)(ехр£/)(а:) - (exp e/) (exp Eg) (a:)^
= [/,<?] (я:)-
3.	Finally, to prove (A.64), we observe that the value at time t = 0 of the
solution of the linear Cauchy problem
i>(£) = £>x/((expt/)(a;)) • v{t),	v(e) = p((expe/)(a;)),	(A.66)
is precisely
v(0) = (exp(—e/))*  g(expef)(x).	(A.67)
From (A.66)-(A.67) it follows
v(0) — v(e) - eDxf(x)  v(e) + o(e)
= g(x) + eDxg(x)  f(x) - eDxf(x)  g(x) + o(e)
(A.68)
Letting e —> 0 in (A.68) we obtain (A.64).
One can think of (A.63) as a recepy for constructing the Lie bracket.
Starting from any point ж, let us move along the vector field f for a time E,
then along g for time e, then along — f for time e, and finally along — g for time
e. As shown in figure A. 13, in the end we reach a point ж + e2 [/, </] (ж) + o(e2).
Fig. A.13. Two constructions of the Lie bracket [/,<?].
Example 3.8. Consider again the car steering problem (1.16) discussed in
the Introduction. This control system is described by the three variables P =
(Ж1,Ж2,0). Here (ж1,жг) yield the position of the barycenter of the car, while
the angle в determines the orientation. If the car advances with unit speed
steering to the right, its motion satisfies
296
A Appendices
-T7 — f(P) = (cos#, sin#, -1).
at
On the other hand, if the car steers to the left, its motion satisfies
dP
= g(P) = (cos 0, sin0, 1).
In a typical parking problem, one needs to shift the car on one side, without
changing its orientation, see figure 1.6 in Chapter 1. This is obtained by the
following maneuver:
1.	right-forward,
2.	left-forward,
3.	right-backward,
4.	left-backward.
Indeed, according to (A.62) the sequence of these four actions generates the
Lie bracket
[/, g] - (Dg)  f - (Df) • g — (2 sin #, - 2 cos #, 0)
which is precisely the desired direction.
According to (A.62), the Lie bracket [/, g] vanishes if the flows generated
by the vector fields /, g commute. Roughly speaking, [/, g] measures the non-
commutativity of these flows, at an infinitesimal level. The next result provides
a converse to Lemma A. 10.1. To avoid technicalities, we assume here that the
vector fields /, g are globally defined on IRn and satisfy a sub-linear growth
condition such as
|Ж)|<с(1 + И).
(A.69)
Fig. A.14. Proof of the commutativity property.
A. 10 Lie brackets and Frobenius’ theorem 297
Lemma A. 10.2. (Commuting vector fields). Let fig be smooth vector
fields on IRn, satisfying the sub-linear growth condition (A.69). Assume that
their Lie bracket vanishes identically: [fig] = 0. Then they commute, namely
(exp sffiexptgfix) = (exp tg) (exp sfi) (ж)	(A.70)
for every x E lRn and every s.t 6 IR.
Proof. For each 9 E [0, t], consider the points (see figure A. 14)
Q(0) = (exp(t - 9)gfiexpsfifiexp9gfix),	P(9) = (exp sfifiexp9gfix).
We claim that
^W) = o.	(A.71)
au
This is clearly true, provided that we can show
b(0)=ff(P(0)).	(A.72)
av
A proof of (A.72) goes as follows. Fix 9, and call
T] ~ Xе (tj) = (expTj/)(exp^)(f)
the solution to the Cauchy problem
= ЛаЛт?)),	®0(O) = (ехр0р)(ж).
ar/
Moreover, let g i—► ve(g) be the solution of the linearized equation
= Dxf(xe^) • veM,	V*(O) = <7(^(0)).	(A.73)
dg
According to Theorem 2.3.1, for every g E [0, s] one has the identities
(exp tj/) (exp 0g)(x) = t/(»j) •
In particular, when g = s we have
-^P(0)=tr’(s).
at/
To prove (A.72), it thus suffices to show that
^(tj) =	f°r all 9 € [0,t] •	(A.74)
Using the assumption [/, g] = 0, we can write
fg(xe^) = Dg(xe(7))).f(xe(Tj)) = Df(xe(ri))-g(xe(r/)).	(A.75)
298 A Appendices
Together, (A.73) and (A.75) imply
-Р(^(т?))| < |Дг/(^(7?)) • (ve(rj) -5(^(7/))) j
< C  |ve(?7) -<z(x0(t/))|
When T] = 0, by (A.73) we have г/0(О) — д(ж0(О)) = 0. An application of
Gronwall’s Lemina 2.1.2 now yields (A.74). Hence (A.72) and (A.71) also hold.
In turn, (A.71) yields 6^(0) = Q(t). By the definition of Q, this is precisely
the conclusion of the lemma.
We now consider a family of vector fields Gi,..., Gm on IR5. assuming
that all their Lie brackets vanish identically:
[G^GjCrHO
for all г, j € {1,..., m} , x G IR 5 .	(A.76)
The following theorem shows that, in this case, the equations x = Gi(x) can
be simultaneously integrated.
Theorem A.10.3. (Frobenius). Let Gi,... , Gm be smooth vector fields on
IR5, satisfying a sub-linear growth condition (A.69) and the commutativity
assumptions (A. 76). Then, for any given x G IR 5, there exists a unique C1
map и = (щ,... , zzm) i—► Ф(и) from IRm into IR5 such that
Ф(0) = x,
дФ
(и) = СкЩи»
duk
for all fcG{l,...,m}, и G IRm.
(A.77)
Proof. The sub-linear growth condition guarantees that, for every Cauchy
problem of the form
at
x(0) = y.
the corresponding solution t x(t) = (exp tGifiy) is well defined for all t G IR
and does not blow up in finite time.
1. To prove uniqueness we observe that, by (A.77) the map t i—> x(t} =
Ф(£,0,... ,0) satisfies
= Gi(x(t)),
z(0) = x.
Therefore, for every щ G IR, when t — ui we must have
Ф(гг1,0,... ,0) = х(щ) = (expuiGi)(rr).
Similarly, the map t i—+ x(fi) = Ф(и\, t, 0,.... 0) satisfies
A. 10 Lie brackets and Frobenius’ theorem 299
-^-x(t) = G2(ic(t)),	ж(0) =	0,..., 0) = (exp tiiGi)(я).
dt
Therefore, for every U2 6 IR, when t = ?i2 we must have
0(t£i, tz2,..., 0) = a:(u2) = (expu2G2)(expiAiGi)(z).
Repeating the above argument, we conclude that the map Ф : lRzn > IRa
must be given by
Ф(и) = (expwmGm) ••• (expuiGi)(£).	(A.78)
In particular, Ф is unique.
2. It remains to show that the map Ф defined by (A.78) has the required
properties (A.77).
The condition Ф(0) = x is clearly satisfied. To compute the partial deriva-
tives of Ф, we observe that, by Lemma A. 10.2, we can arbitrarily permute the
order of the terms in (A.78). In particular, for every fixed index k G {1,..., m},
we can equivalently write
Ф(и) = (expufcGfc)(y),
with
у = (expumGm) ••• (expitfc+iGfc+i)(expufe-iGfe_i) (expUiGiXx).
Differentiating the above expression, we obtain
д . d
-—Ф(и) = —
дик	дик
(expufcGfc)(y) = Gfc ((expufcG*:)(y))
= Gfc(*(u)).
thus proving (A.77).
Notice that, with a slight abuse of notation, if the commutativity assump-
tions (A.76) hold, one can write
Ф(и) = | exp^'UiGj j (ж).	(A.79)
\ i=i /
We conclude with a useful lemma. In the following, Dz(exp VjGj) denotes
the differential of the map x i—> (exp VjGj)(x).
Lemma A. 10.4. With the same assumptions as in Theorem A. 10.3, let
(m	\
expy%jGj (*)
j=i /
Then
(A.80)
300
A Appendices
Proof. Consider the map
m	m
t ( exp VjGj^ (exp tGi)(x) — (exp tGf) ( exp У^ vjGj^ (x)
j=i	j=i
= (expiGi)U).
Computing the derivative of the first and last term w.r.t. t at the time t = 0,
one obtains precisely (A.80).
Problems
A.l. Let X be a Banach space. We say that the norm || • || is strictly convex if
Д? ~h 7/
—-— < 1 whenever 11 x 1| = ||?/|| = 1, x/y.
Prove that the following three statements are equivalent:
•	The norm || • || is strictly convex.
•	Given 0 < 0 < 1 and any two distinct vectors ж, у E X. x =4 y, one has
||fe + (1 - 0)i/|| < max{||x||, IIj/II} .
•	Every point x E dB on the boundary of the unit ball В — {ж E
X ; ||ж|| < 1} is an extreme point of B.
A.2. Show that the set of equations
2ж = cos у, 2y = sin ж
admits a solution (ж*,?/*) inside the unit disc.
Hint: apply a fixed point theorem to the map /(ж, у) = cosy, | sin ж) .
A.3. Let t н-> F(Z) C IRn be a multifunction with closed graph but possibly
unbounded values. Prove that F admits a measurable selection.
Hint: define r(t) = min3z€F(t) |t/| and consider the multifunction F*(t) =
{x€F(t); |z| = r(i)}.
A.4. The contingent cone to a set S at the point £ € S' is defined as
Cs(£) =	: lim inf s-1d(£ + ev,S) = ()j> .
Compute the tangent and contingent cones at 0 E IR2 to the spirals (writ-
ten in polar coordinates ж = pcosO, у = psin0)
Si = {(r, 0); r = 6, 0 < 0 < oo},
$2 = {0} U {(r, 0); r = ee, - oo < 0 < oo},
S3 = {0} U {(r,0); r = 0-1, 0 < 0 < oo}.
A. 10 Lie brackets and Frobenius’ theorem 301
A.5. Consider the function f : [0,1] i—> IR defined by setting f(t) = t if t is
irrational, f(t) = 0 if t is rational.
Show that f is continuous only at the point t = 0. Find the Lebesgue
points of f. Construct a compact set К C [0,1] such that meas(/C) >1/2
and the restriction of f to the set К is continuous.
Hint: let , <72,	... be the list of all rational points in [0,1]. Let Ik by
an open interval containing qk with length 2“fe“1. Consider the union of
all intervals Ik and its complement.
A.6. Let f : [a, b] x IRm h-> ]Rn be continuous and let и : [a, b] > IRm be a
bounded measurable map. If r is a Lebesgue point for u, show that r is
also a Lebesgue point for the composed mapping t f(t,
Hint: for every 8 > 0 prove that
lim_ ^-meas{£ 6 [r — e,r + e] : |/(t,u(t)) - /(t,u(t))| > 5} = 0.
A.7. Let A c IR3 be the set
A = {(x,j/,0); x2 + y2 = 1} U {(1,0,1)} U {(1,0, -1)} .
Prove that its convex hull co A is compact. However, show that the set of
extreme points of co A is not closed.
A.8. In Theorem A.2.1, assume that ||Ф(А,ж) — Ф(Л',x)|| < L || A — A'||, for all
A, A' € A, x E X. Show that the distance between the two fixed points is
bounded by
^(A) -xfA'XI < _А^||Д_ A'||
1 n.
for all A, A' € A.
A.9. Prove the representation formula (A.32) for the convex hull of the set A.
A. 10. Given an open set 12 Q IRn, consider the Banach space C1(J2) of all
continuously differentiable mappings f : 12 IR, with norm
||/||C1 = sup |/(x)| + sup|V/(x)|.
Here V/ = (/X1, ..., fXn) denotes the gradient of f. If 12 is convex, prove
that every f € C^J?) is Lipschitz continuous and ||/||ыр < ll/llc1- How-
ever, show that the above can fail if 12 is not convex.
Hint: let 12 be the union of two disjoint open circles, tangent at one point.
A. 11. On the plane IR2, consider the closed disc
D = {(x,y); x2 + y2 < 4},
302 A Appendices
and the circumference
s = {(x,y);	- I)2 + y2 = 1} 
Observe that S C D. Find a point £ such that the two tangent cones
7p(£) and Ts(£) are weakly separated.
A. 12. Describe the tangent cones to the sets S,S+ C IR3 defined as
S = | (x, y, z); x2 + y2 = 1, x + z — o|,
| (ar, ?/, z); x2 + y2 = 1, x + z = 0 , z > o|,
at the point £ = (0,1,0).
A. 13. Show that the two sets
K = {(x,y); x2+y2<l},	K' = {(x,y)- |x + 2| + |2/-2| < 1} .
can be strictly separated, i.e. find explicitly a unit vector p and e > 0 so
that (A.40) holds.
A.14. Let К, К' C IRn be two compact convex sets such that
max p • x < max p • x
хек ~ хек'
for every unit row-vector p 6 Rn. Prove that К С K'. In particular, show
that K' contains the closed ball 22(0, r) centered at the origin with radius
r > 0 if and only if
max p • x > r
x£K'
for every unit row-vector p G IRn.
A. 15. In the plane IR2. consider the two sets
2<={(ar, 2/); x > 0, у > 0, д?2/>1|, К' = {(я, у); у = 0} .
Notice that К, К' are closed and disjoint. However, prove that they cannot
be strictly separated, i.e. one cannot find any unit vector p and s > 0 such
that (A.40) holds.
A. 16. Let x i—> F(x) C IRn be a Hausdorff continuous multifunctions with con-
vex, compact values. Fix a point у 6 lRn and define f(x) 6 F(x) as the
perpendicular projection of у on the set F(t), i.e. (see figure A. 15)
= 7Гг(х)(у).
Prove that the map x f (x) provides a continuous selection of F.
A. 10 Lie brackets and Frobenius’ theorem 303
A. 17. On the space IR2, consider the multifunction
x !-> F(x) =	€ IR2; |y| < 1,
|y-a:| > l/з} .
Show that F is Hausdorff continuous with compact (but not convex) values
(see figure A. 15). Prove that there exists no continuous selection x i—>
/(x) e F(x).
Fig. A. 15. A continuous selection of a convex-valued multifunction and a continuous
multifunction without any continuous selection.
A.18. Let the assumptions of Theorem A.10.3 hold. For any u,u' G IRm and
x G IR a , prove the identity
m	\	/	m	\	/	m	\
exp^2(ui +u'i)Gi j (я) = j exp^u'G; j I exp^PujGj I (ж).
i=l	/	\	i=l	/ \	i=l	/
A. 19. Consider two linear vector fields on IRn, say f(x) = Ax, g(x) = Bx for
some n x n matrices A, B. Compute the Lie bracket [/, g] and show it is
also a linear vector field.

References
[1]	A. A. Agrachev and Yu. Sachkov, Control Theory from the Geometric View-
point. Springer Ver lag, Berlin, 2004.
[2]	F. Ancona and A. Bressan, Patchy vector fields and asymptotic stabilization.
ESAIM - Control, Optimiz. Calc. Var. 4 (1999), 445 471
[3]	F. Ancona and A. Bressan, Flow stability of patchy vector fields and robust
feedback stabilization SIAM J. Control Optim. 41 (2003), 1455-1476.
[4]	F. Ancona and A. Bressan, Nearly time optimal stabilizing patchy feedbacks.
Ann. Inst. Henri Poincare, Analyse Non Lineaire, 24 (2007), 279-310.
[5]	J.P. Aubin and A. Cellina, Differential Inclusions. Springer-Ver lag, 1984.
[6]	J.P. Aubin and H. Frankowska, Set Valued Analysis. Birkhauser, Boston, 1990.
[7]	A. Bacciotti, Local Stabilizability of Nonlinear Control Systems. World Scien-
tific, River Edge, NJ, 1992.
[8]	M. Bardi and I. Capuzzo Dolcetta, Optimal Control and Viscosity Solutions
of Hamilton-Jacobi-Bellman Equations. Birkhauser, 1997.
[9]	G. Barles, Solutions de Viscosite des Equations de Hamilton-Jacobi. Springer,
1994.
[10]	L.D. Berkovitz, Optimal Control Theory. Springer-Verlag, New York, 1974.
[11]	V.G. Boltyanskii, Sufficient conditions for optimality and justification of the
dynamic programming method, SIAM J. Control, 4 (1966), pp. 326-361.
[12]	U. Boscain and B. Piccoli, Optimal Syntheses for Control Systems on 2-D
Manifolds. Springer, New York, 2004.
[13]	A. Bressan, Singularities of stabilizing feedbacks, Rend. Sem. Mat. Univ. Pol.
Torino 56 (1998), 87-104.
[14]	A. Bressan, Impulsive control of Lagrangian systems and locomotion in fluids,
Discr. Cont. Dynam. Syst., to appear.
[15]	A. Bressan and B. Piccoli, A Baire category approach to the bang-bang prop-
erty, J. Diff. Equations 116 (1995), no. 2, 318-337.
[16]	A. Bressan and F. Rampazzo, On differential systems with vector-valued im-
pulsive controls, Boll. Un. Matematica Italiana 2-B, (1988), 641-656.
[17]	A. Bressan and F. Rampazzo, Impulsive control systems with commutative
vector fields, J. Optim. Theory & Appl. 71 (1991), 67-84.
[18]	A. Bressan and F. Rampazzo, On systems with quadratic impulses and their
application to Lagrangean mechanics, SIAM J. Control 31 (1993), 1205-1220.
306 References
[19]	A. Bressan and F. Rampazzo, Moving constraints as stabilizing controls in
classical mechanics, preprint (2007).
[20]	Aldo Bressan, Hyper-impulsive motions and controllizable coordinates for La-
grangean systems Atti Accad. Naz. Lincei, Memorie, Serie VIII, Vol. XIX
(1990), 197-246.
[21]	Aldo Bressan, On some control problems concerning the ski or swing, Atti
Accad. Naz. Lincei, Memorie, Serie IX, Vol. I, (1991), 147-196.
[22]	R. W. Brockett, Asymptotic stability and feedback stabilization, in Differen-
tial Geometric Control Theory, R. W. Brockett, R. S. Millman and H. J. Suss-
mann Eds., Birkhauser, Boston, (1983), 181-191.
[23]	P. Brunovsky, Existence of regular synthesis for general problems,
J. Diff. Equations, 38 (1980), pp. 317-343.
[24]	F. Cardin and M. Favretti, Hyper-impulsive motion on manifolds. Dynam.
Contin. Discrete Impuls. Systems 4 (1998), 1-21.
[25]	D. A. Carlson, A. B. Haurie, and A. Leizarowitz, Infinite Horizon Optimal
Control: Deterministic and Stochastic Systems. Springer-Verlag, 1991.
[26]	L. Cesari, Optimization Theory and Applications. Springer-Verlag, 1983.
[27]	F.H. Clarke, Optimization and Nonsmooth Analysis. Wiley Interscience, New
York, 1983.
[28]	F. H. Clarke, Necessary conditions in dynamic optimization, Memoirs Amer.
Math. Soc. 816, vol. 173 (2005).
[29]	F. H. Clarke, Yu. S. Ledyaev, E. D. Sontag, and A. I. Subbotin, Asymptotic
controllability implies feedback stabilization. IEEE Trans. Automat. Control
42 (1997), 1394-1407.
[30]	F. H. Clarke, Yu. S. Ledyaev, R. J. Stern, and P. R. Wolenski, Nonsmooth
Analysis and Control Theory. Graduate Texts in Mathematics 178, Springer-
Verlag, New York, 1998.
[31]	R. Conti, Problemi di Controllo e di Controllo Ottimale, U.T.E.T., Torino,
1974 (Italian).
[32]	R. Conti, Processi di controllo lineari in IRn, Pitagora Editrice, Bologna, 1985
(Italian).
[33]	J. M. Coron, A necessary condition for feedback stabilization. Systems Control
Lett. 14 (1990), 227 232.
[34]	M. Crandall and P. L. Lions, Viscosity solutions of Hamilton-Jacobi equations,
Trans. Amer. Math. Soc. 277 (1983), 1-42.
[35]	M.G. Crandall, L.C. Evans and P.L. Lions, Some properties of viscosity so-
lutions of Hamilton-Jacobi equations, Trans. Amer. Math. Soc.. 282 (1984),
pp. 487-502.
[36]	M. Crandall. II. Ishii and P. L. Lions, User’s guide to viscosity solutions of
second order partial differential equations, Bull. Amer. Math. Soc. 27 (1992),
1-67.
[37]	G. Dal Maso, P. LeFloch, and F. Murat, Definition and weak stability of
nonconservative products. J. Math. Pures Appl. 74 (1995), 483-548.
[38]	K. Dcimling, Nonlinear Functional Analysis, Springer-Verlag, Berlin, 1985.
[39]	K. Deimling, Multivalued Differential Equations, W. de Gruyter, Berlin, 1992.
[40]	J. Dieudonne, Foundations of Modern Analysis, Academic Press, New York,
1960.
[41]	L.C. Evans and M.R. Janies, The Hamilton-Jacobi-Bellmann equation for
time-optimal control. SIAM J. Control, 27 (1989), pp. 1477-1489.
References
307
[42]	A.F. Filippov, On certain questions in the theory of optimal control, SIAM
J. Control, 1 (1962), pp. 76-84.
[43]	A.F. Filippov, Differential equations with multivalued discontinuous right
hand side, Dokl. Akad. Nauk. USSR, 151 (1963), pp. 65-69.
[44]	A.F. Filippov, Classical solutions of differential equations with multivalued
right hand side, SIAM J. Control, 5 (1967), pp. 609-621.
[45]	W.H. Fleming and R.W. Rishel, Deterministic and Stochastic Optimal Con-
trol. Springer-Verlag, New York, 1975.
[46]	G.B. Folland, Real Analysis, Wiley-Interscience, New York, 1984.
[47]	A. T. Fuller, Study of an optimum non linear control system, J. Electronics
Control, 15 (1963), pp. 63-71.
[48]	R. Gabasov and F.M. Kirillova, High order necessary conditions for optimal-
ity, SIAM J. Control, 10 (1972), pp. 127-168.
[49]	R. V. Gamkrelidze, On sliding optimal regimes, Soviet Math. Dokl., 3 (1962),
pp. 390-395.
[50]	O. Hajek, Discontinuous differential equations I, J. Differential Equations 32
(1979), 149-170.
[51]	P. Hartman, Ordinary Differential Equations, S.H. Hartman, Baltimore, 1973.
[52]	H. Hermes, On local controllability. SIAM J. Control Optim. 20 (1982), 211-
220.
[53]	H. Hermes, Nilpotent and high-order approximations of vector field systems,
SIAM Review, 33 (1991), pp. 238-264.
[54]	H. Hermes and J.P. LaSalle, Functional Analysis and Time Optimal Control,
Math, in Sci. & Engr., vol 56, Academic Press, 1969.
[55]	A. Isidori, Nonlinear Control Systems. Third edition. Communications and
Control Engineering Series. Springer-Verlag, Berlin, 1995.
[56]	V. Jurdjevich, Geometric Control Theory. Cambridge University Press, 1996.
[57]	M. Kawski, High-order small-time local controllability, in Nonlinear control-
lability and optimal control, H.J. Sussmann ed., M. Dekker, New York, 1990.
[58]	H.W. Knobloch, High order necessary conditions in optimal control theory,
Lecture Notes in Control and Information Sciences, Springer-Verlag, Berlin,
1981.
[59]	A.J. Krener, The high order maximal principle and its applications to singular
extremals, SIAM J. Control, 15 (1977), pp. 256-293.
[60]	S. Kruzhkov, First order quasilinear equations with several space variables,
Math. USSR Sbomik 10 (1970), 217-243.
[61]	J. Kulkarni and B. Piccoli, Pumping a swing by standing and squatting: do
children pump time optimally?, Control Systems Magazine, IEEE 25 (2005),
48-56.
[62]	E.B. Lee and L. Markus, Foundations of Optimal Control Theory, John Wiley,
New York, 1967.
[63]	J.P. Lions, Optimal Control of Systems Governed by Partial Differential Equa-
tions, Springer-Verlag, Berlin, 1976.
[64]	I. Lasiecka and R. Triggiani, Control theory for partial differential equations:
continuous and approximation theories. I. Abstract parabolic systems. Ency-
clopedia of Mathematics and its Applications 74. Cambridge University Press,
2000.
[65]	I. Lasiecka and R. Triggiani, Control theory for partial differential equations:
continuous and approximation theories. II. Abstract hyperbolic-like systems
308 References
over a finite time horizon. Encyclopedia of Mathematics and its Applications
75. Cambridge University Press, 2000. pp. 645-1067.
[66]	C. Marie, Geometric des systeines mecaniqucs a liaisons actives, in Symplectic
Geometry and Mathematical Physics, 260-287, P. Donato, C. Duval, J. El-
hadad, and G. M. Tuynman Eds., Birkhauser, Boston, 1991.
[67]	L. W. Neustadt, The existence of optimal control in absence of convexity
conditions, J. Math. Anal. Appl., 7 (1963), pp. 110-117.
[68]	H. Nijmejer and A.J. van der Schaft, Nonlinear Dynamical Control Systems,
Springer-Verlag, New York, 1990.
[69]	C. Olech and A. Lasota, On the closedness of the set of trajectories of a
control system, Bull. Acad. Polon. Sci., 16 (1968), pp. 711-716.
[70]	B. Piccoli, Classification of generic singularities for the planar time optimal
synthesis, S.I.A.M. J. Control Optim. 95 (1996), 59-79.
[71]	B. Piccoli and H.J. Sussmann, Regular synthesis and sufficiency conditions
for optimality, SIAM J. Control Optim. 39 (2000), no. 2, pp. 359-410.
[72]	L.S. Pontryagin and V.G. Boltyanskii, R.V. Gamkrelidze and E.F. Mishenko,
The Mathematical Theory of Optimal Processes, John Wiley, New York, 1962.
[73]	F. Rampazzo, On the Riemannian structure of a Lagrangean system and
the problem of adding tiine-dependent coordinates as controls European J.
Mechanics A/Solids 10 (1991), 405-431.
[74]	L. Rifford, Semiconcave control-Lyapunov functions and stabilizing feedbacks.
SIAM J. Control Optim. 41 (2000), 659-681.
[75]	R.T. Rockafellar, Convex Analysis, Princeton University Press, 1970.
[76]	W. Rudin, Functional Analysis, McGraw-Hill, New York, 1973.
[77]	W. Rudin, Real and Complex Analysis, McGraw-Hill, New York, 1974.
[78]	G. V. Smirnov, Introduction to the Theory of Differential Inclusions, A.M.S.
Graduate Studies in Mathematics, vol. 41 (2002).
[79]	E. D. Sontag, A Lyapunov-like characterization of asymptotic controllability,
SIAM. J. Control Optim. 21 (1983), pp. 462-471.
[80]	E. D. Sontag, Mathematical Control Theory. Deterministic Finite-dimensional
Systems, Second edition. Springer-Verlag, New York, 1998.
[81]	G. Stefani, Polynomial approximations to control systems and local control-
lability, Proc. 24th IEEE Conf. Decision & Control, Ft. Lauderdale (1985),
pp. 33-38.
[82]	A. I. Subbotin, A theory of generalized solutions to first-order PDEs with
an emphasis on differential games. In Advances in nonlinear dynamics and
control: a report from Russia, 189 229, Birkhuser, Boston, 1993.
[83]	H. J. Sussmann, On the gap between deterministic and stochastic ordinary
differential equations, Ann. Prob. 6 (1978), 17-41.
[84]	H. J. Sussmann, Lie brackets, real analyticity and geometric control theory, in
Mathematical Control Theory, R.W. Brockett, R.S. Milimam and H.J. Suss-
mann eds., Birkhauser, Boston Inc. (1983), pp. 1-115.
[85]	H. J. Sussmann, The structure of time-optimal trajectories for single-input
system in the plane: The C°° nonsingular case, SIAM J. Control, 25 (1987),
pp. 433-465.
[86]	H.J. Sussmann, A general theorem on local controllability, SIAM J. Control,
25 (1987), pp. 158-194.
[87]	H.J. Sussmann and V. Jurdjevic, Controllability of nonlinear system, J. Diff.
Equations, 12 (1972), pp. 95-116.
References 309
[88]	D. V. Tran, M. Tsuji and D. T. S. Nguyen, The Characteristic Method and
its Generalizations for First Order Nonlinear Partial Differential Equation,
Chapman&Hall/CRC, 1999.
[89]	T. Wazewski, Systemes de commande et equations au contigent,
Bull. Acad. Polon. Sci., 9 (1961), pp. 151-155.
[90]	L.C. Young, Calculus of Variations and Optimal Control Theory, Chelsea
Publ. Co., New York, 1980.
[91]	J. Zabczyk, Mathematical Control Theory: an Introduction, Birkhuser,
Boston, 1992.
[92]	V. Zeidan, Sufficient conditions for the generalized problem of Bolza,
Trans. Amer. Math. Soc., 275 (1983), pp. 561-586.

Index
absolutely continuous functions, 276
algebraic Riccati equation, 154
asymptotic stability, 199
Banach space, 264
Banach’s contraction mapping theorem,
265
Bang-Bang theorem, 69
Bolza problem, 93
Brouwer theorem, 267
Caratheodory solution, 13
Caratheodory theorem, 284
Cauchy problem, 14, 37
Cauchy sequence, 263
chattering controls, 66
closed loop control, 3
constrained optimization, 291
control system, 1, 35
controllability matrix, 56
convex cone, 288
convex function, 135
convex hull, 284
convex set, 263, 283
differentiability w.r.t. initial data, 26
differential inclusion, 2, 36
direct method, 90
dynamic programming, 137, 186
dynamic programming principle, 186
Euler-Lagrange and Weierstrass
necessary conditions, 121
feedback, 3
fit for jumps, 240
Frobenius theorem, 298
fundamental matrix solution, 24
graph completion, 249
Gronwall’s lemma, 17
Hamilton-Jacobi-Bellman equation, 189
Hausdorff distance, 280
impulsive control system, 233
input-output map, 38
Lebesgue dominated convergence
theorem, 275
Lie algebra, 62
Lie bracket, 62, 294
linear O.D.E., 21
linear system, 11, 56
linear-quadratic problem, 125, 150
Lipschitz continuous, 265
lower semicontinuous, 274
Lyapunov function, 76
Lyapunov stability, 76
Lyapunov’s theorem, 277
Mayer problem, 88
minimum time problem, 121
multifunction, 36, 280
necessary condition, 10, 99
needle variation, 101
normed space, 263
312 Index
O.D.E., 13
open loop control, 3
optimal control, 9
optimal synthesis, 155
ordinary differential equation, 1, 13
partial differential equation, 165
patch, 201
patchy feedback, 203
patchy vector field, 201
pole shifting, 80
Pontryagin Maximum Principle, 10,
100, 111, 116
Rademacher’s theorem, 278
reachable set, 8, 51
Riccati differential equation, 151
small time local controllability, 9, 60
stability, 76, 207
stabilizing feedback, 79
strongly fit for jumps, 240
sub-differential, 170
sufficient condition, 10, 133
super-differential, 170
trangent cone, 288
transversality, 30
value function, 186
viscosity solution, 175