Optimal Control of ODEs:Introductory Lecture

Suzanne Lenhart

University of Tennessee, Knoxville

Department of Mathematics

Lecture1 – p.1/55

Outline

1. Motivating Example2. Set-up of optimal control problem3. Necessary Conditions to characterize OC4. Simple Examples5. Back to motivating example

Lecture1 – p.2/55

Fishery Model

� � � � � � � � � � � �

� �� �

population level of fish� �� �

harvesting control

Maximizing net profit:

� � �� ����� � � � ��� � � � �� � �� � ��� �

where � �� is discount factor, � � � ��� � �� terms represent profit

from sale of fish, diminishing returns when there is a large

amount of fish to sell and cost of fishing. � � � � ��� � �� are

positive constants.Lecture1 – p.3/55

Overview

To formulate an appropriate OC problem, thesystem of differential equations must be areasonable representation of the scenario to beconsidered.Designing an appropriate objective functional isequally important.Balancing competing goals may be crucial.The form of the OC results depend strongly onthe system of equations, how the controls enterthat system, and the objective functional.

Lecture1 – p.4/55

Optimal Control

Adjust controls in a system to achieve a goalSystem:

Ordinary differential equations

Partial differential equations

Discrete equations

Stochastic differential equations

Integro-difference equations

Lecture1 – p.5/55

Big Idea

In optimal control theory, after formulating aproblem appropriate to the scenario, there areseveral basic problems :

(a) to prove the existence of an optimal control,

(b) to characterize the optimal control,

(c) to prove the uniqueness of the control,

(d) to compute the optimal control numerically,

(e) to investigate how the optimal controldepends on various parameters in the model.

Lecture1 – p.6/55

Notation

� � � ���

�

�

! � " �#

� ! $ � #

� � ! % � exponential growth

� & � ' ! ��( ) *+time is underlying variable.

Lecture1 – p.7/55

Deterministic Optimal Control

Control of Ordinary Differential Equations (DE), -�. /

control0 -�. /

stateState function satisfies DEControl affects DE

0 1 -�. / 2 3 -.4 0 -�. /4 , -. / /

, -�. / 0 -�. /

Goal (objective functional)

Lecture1 – p.8/55

Deterministic Optimal Control- ODEs

Find piecewise continuous control 5 6�7 8and

associated state variable 9 67 8

to maximize

: ;<=

>6�7? 9 6�7 8? 5 6�7 8 8@ 7

subject to

9 A 6�7 8 B C 6�7? 9 67 8? 5 6�7 8 8

9 6ED 8 B 9 >and 9 6 8 FG G

Lecture1 – p.9/55

Contd.

Optimal Control H I J�K L

achieves the maximum

Put H I J�K L

into state DE and obtain M I J�K L

M I J�K L

corresponding optimal state

H I J�K L

, M I J�K L

optimal pair

Lecture1 – p.10/55

Necessary and Sufficient Conds.

Necessary ConditionsIf N O P�Q R

, S O PQ R

are optimal, then the followingconditions hold...

Sufficient ConditionsIf N O P�Q R

, S O P�Q R

and

T

(adjoint) satisfy the conditions...then N O P�Q R

, S O P�Q Rare optimal.

Lecture1 – p.11/55

Adjoint

like Lagrange multipliers to attach DE to objective func-

tional.Lecture1 – p.12/55

Deterministic Optimal Control- ODEs

Find piecewise continuous control U V�W Xand

associated state variable Y VW X

to maximize

Z [\]

^V�W_ Y V�W X_ U V�W X X` W

subject to

Y a V�W X b c V�W_ Y VW X_ U V�W X X

Y VEd X b Y ^and Y V X ef f

Lecture1 – p.13/55

Quick Derivation of Necessary Condition

Suppose g h is an optimal control and i hcorresponding state.

j k�l m

variation function,n o .

t

u*(t)+ ah(t)

u*(t)

g h k�l m n j k�l m

another control.p klq n m state corresponding to g h n j

,

r p klq n mrl s t k lq p k�lq n mq k g h n j m k�l m m m

Lecture1 – p.14/55

Contd.

At

u v w

, x y wz { | v }�~

t

x * (t)

y(t,a)

x 0

all trajectories start at same position

x y uz w | v } � y u | when { v wz control � �

� y { | v�

~y uz x y uz { | z � � y u | { � y u | |� u

Maximum of�

w.r.t. { occurs at { v w

.Lecture1 – p.15/55

Contd.

�� ��� ���

���������� � � �

��

��� �� � � ��� � ��� � � � �� � � �� ��� �� � � ��� � � � ��� � �� � �

� ��

��� �� � � �� � ��� � � � �� � � � � �� � �� � � � � �� ��� �� � � � � �¡ 

Adding

�

to our

� �� �gives

Lecture1 – p.16/55

Contd.

¢ £�¤ ¥§¦ ¨© ª £«�¬ ­ £« ¬ ¤ ¥¬ ® ¯±° ¤ ² ¥ ° ³

³« £´ £« ¥ ­ £«�¬ ¤ ¥ ¥ ³«

° ´ £µ ¥ ­ £µ ¬ ¤ ¥�¶ ´ £· ¥ ­ £· ¬ ¤ ¥

¦ ¨© ¸ ª £« ¬ ­ £«�¬ ¤ ¥¬ ® ¯�° ¤ ² ¥ ° ´ ¹ £« ¥ ­ £« ¬ ¤ ¥

° ´ £« ¥�º £«�¬ ­¬ ® ¯° ¤ ² ¥» ³« ° ´ £µ ¥�¼ ©¶ ´ £· ¥ ­ £· ¬ ¤ ¥

here we used product rule and º ¦ ³ ­ ½ ³«

.

Lecture1 – p.17/55

Contd.Take the derivative of J with respect to aand evaluate it at a =0.¾¿ ¾ÁÀ

SIMPLIFY using the adjoint

Lecture1 – p.18/55

Contd.Choose

 Ã�Ä Å

to simplify this derivative.

Æ ÇÈÉ Ê§Ë Ì ÍÎÐÏ ÈÉ�Ñ Ò ÓÑ Ô Ó ÊÖÕ ÆÈÉ Ê�× Ï ÈÉ Ñ Ò ÓÑ Ô Ó ÊØ adjoint equationÆÈÙ ÊË Ú

transversality condition

ÚË ÛÜ È ÎÐÝ Õ Æ× Ý ÊÞ ÈÉ Êß É

Þ ÈÉ Ê

arbitrary variation

à ÎÐÝ ÈÉ Ñ Ò ÓÑ Ô Ó Ê Õ ÆÈÉ Ê× Ý ÈÉ Ñ Ò ÓÑ Ô Ó ÊË Ú

for all

Úá É á Ù â

Optimality condition.

Lecture1 – p.19/55

Using Hamiltonian

Generate these Necessary conditions fromHamiltonian

ã�äå æå çå è é ê ã äå æå ç é èìë ã�äå æå ç éintegrand (adjoint) (RHS of DE)

maximize w.r.t. ç at ç í

îî ç ê ï ð èë ð ê ï

optimality eq.

è ñ ê òî

î æ è ñ ê ò ãó èìëó é

adjoint eq.

è ã é ê ïtransversality condition

Lecture1 – p.20/55

Given ô õ÷ö ø ùú�û ôû ü ý DEô ùúÿþ ý ö ôþ IC �Use Hamiltonian to get other conditions� �� ü ö �

� õö � � �� ô� ù� ý ö � �

Converted problem of finding control to maximize objective

functional subject to DE, IC to using Hamiltonian pointwise.Lecture1 – p.21/55

For maximization��� � �

at � � as a function of

For minimization��� � �

at � � as a function of

Lecture1 – p.22/55

Two unknowns � � and � �

introduce adjoint

�

(like a Lagrange multiplier)

Three unknowns � � , � � and

�nonlinear w.r.t. �

Eliminate � � by setting � � �and solve for � � in terms of � � and

�

Two unknowns � � and�

with 2 ODEs (2 point BVP)+ 2 boundary conditions.

Lecture1 – p.23/55

Pontryagin Maximum Principle

If � � ��� � and � � ��� � are optimal for above problem, then thereexists adjoint variable

� �� �

s.t.

� ��� � � � ��� �� � ��� �� � ��� � � ! � �� � � � ��� �� � � ��� �� � ��� � ��

at each time, where Hamiltonian�

is defined by

� ���� � ��� �� � ��� �� � ��� � � " # ���� � ��� �� � ��� � � $ �&% �� � � ��� �� � ��� � �'

and

� ( ��� � " ) * � ���� � ��� �� � ��� �� � ��� � �* �� �+ � " , transversality condition

Lecture1 – p.24/55

Hamiltonian

-/. 0 1�2 3 43 5 6 7 8 1 2 6 0 1 23 43 5 6

5 9 maximizes

-

w.r.t. 5, -

is linear w.r.t. 5

-/. : 1�2 3 43 8 6 5 1 2 6 7 ; 1 23 43 8 6

bounded controls, < = 5 1�2 6 = >.

Bang-bang control or singular control

Example:

-/. ? 5 7 8 5 7 4A@ 8 4 BC -C 5 . ? 7 8 D. E

cannot solve for 5

-

is nonlinear w.r.t. 5, set

-GF . E

and solve for 5 9

optimality equation.Lecture1 – p.25/55

Example 1

H IKJ LM N OQP�R ST R

U V W U X NZY UP[ S W \

] W integrand X ^

RHS of DE W N O X ^P U X N S_ ]_ N W ` N X ^ W[ a N b Wdc ^` at N b

_ O ]_ N O W `^ V Wc _ ]_ U Wdc ^ ^P \ S W[

^ W ^ M e f g h[ W ^ M e f L a ^ M W[^ji [ Y N b i [ Y U b W e g

Lecture1 – p.26/55

Example 2

k lnmop

q r sut v wx y s t v p{z t

subject to r | sut v } r s t v y sut v~ r s�� v } wx � p � w�

What optimal control is expected?

Lecture1 – p.27/55

Hamiltonian

� � �� � � � � � � �

The adjoint equation and transversality conditiongive

� � �u� � � � �� � � � � � �� � � � � � � � �u� � � � �� � � � �

Lecture1 – p.28/55

Example 2 continued

and the optimality condition leads to

� � ��� � � � � � �u� � � � � �   � ¡ ¢£ ¤¥

The associated state is

¦ � �u� � �  § ¡ ¢£ ¤ �   ¥

Lecture1 – p.29/55

Graphs, Example 2

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2−1

0

1

2

3

Time

Sta

te

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

2

4

6

8

Time

Adj

oint

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2−8

−6

−4

−2

0

Time

Con

trol

Example 2.2

Lecture1 – p.30/55

Exercise

¨©ª«¬

­®°¯ ± ²³ ´

¯ µ ¶ · ¸ ± ¹º ¯ ®�» ² ¶ · º ± control ¼ ¶½½ ± ¶ »

¾ µ ¶ ¾ ¶

¾ ® · ² ¶± ¿ ¶ ¯ ¿ ¶

Lecture1 – p.31/55

Example

À ÁÂÃÄ

Å ÆÈÇ ÉËÊ ÌÍÎ

Ç ÏÑÐ ÒÔÓ Ê ÕÖ Ç Æ× Ì Ð Ò Ê controlØ ÆÎ Ö Ç Ö Ê Ö Ù Ì Ð Ç É Ê É Ù Æ ÒÔÓ Ê Õ Ì

Ú ØÚÊ Ð Ò Ó Û ÙÊ Ð × Ü Ê Ð ÒÛ Ù Ý Ø Ã ÃÐ Ó Û ÙÞ × Ö

Ù ÏÐ Ó Ú ØÚÇ Ð Ó ÒÖ Ù Æ Ò Ì Ð × Ö Ü ÙÐ Ò Ó Î

Ç ÏÑÐ ÒÔÓ Ê Õ Ð Ò Ó Òß Æ ÒÔÓ Î Ì Õ

Ç à ÆÎ Ì Ð Î Ó Ò á ß Æ ÒÔÓ Î Ì É â á ßÖ Ê à ÆÎ Ì Ð Ò á Û Æ Ò Ó Î Ìã

Lecture1 – p.32/55

Contd.

äå

æ°ç è éuê ë ì è éuê ëí îê ïThere is not an “Optimal Control" in this case.

Want finite maximum.

Here unbounded optimal stateunbounded OC

Lecture1 – p.33/55

Salvage Term

ðñò ó ôÈõ ôö ÷ ÷ùø úû ü ô�ý þ õ ô�ý ÷þ ÿ ô�ý ÷ ÷� ý

õ � � � ô�ýþ õ þ ÿ ÷ õ ô� ÷ � õ ûwhere

ó ôõ ôö ÷ ÷

is final payoff. What change results?

� ô�� ÷ � úû ü ô�ýþ � ô�ý þ � ÷þ ÿ ø � ÷� ý ø ó ô � ôö þ � ÷ ÷

...� �� � ô� ÷ � � � úû � � � ý

same as before� � ôö ÷ � �� � ôö þ � ÷ ø ó � ôÈõ ôö ÷ ÷ � �� � ôö þ � ÷

Lecture1 – p.34/55

Only change

� � � � � ��� � � � �Example

� �� ��� � � � �

�"!# � # $ �% !

��� � � �� � � � &' �

� � � � &' � � � �

Lecture1 – p.35/55

Back to Fishery Model

( )+* , ( - . ( / . 0 (

( -21 /

population level of fish0 -21 /

harvesting control

Maximizing net profit:

34 5 6 78 9;:=< 0 ( . :=> - 0 ( /> . ?< 0 @"A 1

where 5 6 78 is discount factor, : <CB :=>CB ?< terms represent profit

from sale of fish, diminishing returns when there is a large

amount of fish to sell and cost of fishing. B : < B :=>CB ?< are

positive constants.Lecture1 – p.36/55

Contd.

D E F GH IKJML NO P JMQ R NO SQ P TL N U

V R O R P O S P NO S

V W D PX

XO D P Y E F GH R J L N P Z J"Q NQ O S

V R P Z O P N S[

XX N D E F GH R J L O P Z JMQ NO Q P TL S V R P O S D \

N ] D P V O ] E F GH R J"L O ] P TL S

Z E F GH J"Q R O ] SQ

Lecture1 – p.37/55

Contd.

Solve for ^ _` a _` b

numerically.

Need control bounds

c ^ d"e f gMhRef:B D Craven bookControl and Optimization

Lecture1 – p.38/55

Interpretation of Adjoint

i jklmon

mopq"rs ts u vw r x q tMy s r y v

( Definition of value function )

t z { | q"rs ts u v

t qr y v { t y

}} t q t y s r y v { ~ q"r y v

�� i� � yq t y �s r y v � q t�y s r y v

�

Units: money/unit item in profit problems.Lecture1 – p.39/55

� �"��� �

= marginal variation in the optimal objectivefunctional value of the state value at

� � .“Shadow price”� additional money associated with addi-tional increment of the state variable

���� � � � �� �� � � � � �"� �

for all

� � � ���

“If one fish is added to the stock, how much is thevalue of the fishery affected ?"

Lecture1 – p.40/55

���� � �M� � �� � � � � �� �

Approximate

� � � � � �� � � � �"� � �� �

� � � � �� �

� � � � � �� � � � � � � �� � � � �� �

New value Original value + adjoint

Lecture1 – p.41/55

Principle of Optimality

If   ¡¢ £ ¡

is an optimal pair on¤�¥ ¤ ¤�¦ and

¤�¥ §¤ ¤�¦ ,then   ¡¢ £ ¡

is also optimal for the problem on§¤ ¤ ¤¦ :

¨©ª«¬�­

®¬¯ ¤¢ £¢   °± ¤ £ ² ³ ´ ¯ ¤¢ £¢   °

£ ¯ §¤ ° ³ £ ¡ ¯ §¤ °t

x 0

t 0 t t 1

(t, x*(t ))

x

Lecture1 – p.42/55

Optimality System

State system coupled with adjoint system- optimal control’s expressions substituted in

Numerical Solutions by Iterative Method- with Runge Kutta 4, Matlab or favorite ODE solver- guess for controls, solve forward for states- solve backward for adjoints- update controls, using characterization- repeat forward and backwards sweeps and controlupdates until convergence of iterates

Lecture1 – p.43/55

Bounded Controls

As long as the final position of the state variableis not fixed in advance:Control µ ¶ ·"¸ ¹ º

Solve for the optimal control using the optimalitycondition and then impose the bounds on theformula.In that exercise, suppose for all controls» ¶ · ¸ ¹ ¼

Then ¶ ½ ·"¸ ¹ ¾ ¿ ÀoÁ · ¼Â ¿ µÃ · »Â ÄÅ Æ ¹ ¹

Lecture1 – p.44/55

Multiple States and Controls

Consider a problem with Ç state variables, È

control variables, and a payoff function ,

É ÊËÌÎÍÐÏÑ Ñ Ñ Ï ÌÓÒÔ�Í

ÔoÕÖ"×Ø Ù"Ú Ö"× ÛØ Ü Ü ÜØ ÙÞÝ Ö"× ÛØ ßÚ Ö"× ÛØ Ü Ü ÜØ ßÞà Ö× Û Ûá ×

Ö Ù�Ú Ö"× Ú ÛØ Ü Ü ÜØ Ù Ý Ö× Ú Û Û

subj. to Ù âã Ö"× Û ä å ã Ö×Ø ÙÚ Ö"× ÛØ Ü Ü ÜØ ÙæÝ Ö× ÛØ ßÚ Ö"× ÛØ Ü Ü ÜØ ßà Ö"× Û ÛØ

Ù ã Ö"×�ç Û ä Ù ãç fixed, Ù ã Ö"× Ú Û

free

Lecture1 – p.45/55

Vector Notation

è éêëíìîoï

îoðñ"òó ô�õ ñ"ò öó ô�÷ ñ"ò ö öø ò

ñ ôõ ñ"ò�ù ö ösubject to

ô�õ ú ñ"ò ö û ô�ü ñ"ò ó ôõ ñ"ò öó ô�÷ ñ"ò ö öóô�õ ñò�ý ö û ô�õý ó ô�õ ñò ù ö

free þ

Lecture1 – p.46/55

Hamiltonian

ÿ�� � ��� � ��� � ��� ÿ� � ��� � ��� �� ÿ�� �� �� ÿ�� � ��� � ��� �

ÿ�� � ��� � ��� � ��� ÿ� � ��� � ��� �

��� �� �ÿ�� �ÿ�� � �� � ���

Lecture1 – p.47/55

Necessary Conditions

� �� ��� � � ��� � � � � ���� ��� �� �� � � ���"! � � � �! � # � $� % % %� &�

� �' ��� � � (�

� � '� � ' ���*) � � +-, � � ��*) � �� . � $� % % %� &�

/ � �� 10 � 2 � $� % % %� 3�

Lecture1 – p.48/55

Optimality Conditions

44�576 8 9

to

:;;=<

5 6 8 >6 if

?@?BAC D 9

>6 516 E6 if

?@?BAC 8 9

5 6 8 E6 if

?@?BAC F 9G

Lecture1 – p.49/55

States and Adjoints

To each state, there corresponds an adjoint.The first adjoint corresponds to the first state...

Lecture1 – p.50/55

Simple Example

Consider a well-stirred bioreactor in whichcontaminant and bacteria present in spatiallyuniform, time varying concentrations.

H I�J K L concentration of contaminant

M I�J K L concentration of bacteria

The bioreactor is rich in all nutrients except oneto be controlled,

N IJ K L concentration of input nutrient O

Lecture1 – p.51/55

The bacteria degrades contaminant viaco-metabolism, meaning degradation of thecontaminant is a byproduct of the bacteriametabolism. The growth of the bacteria as resultof the nutrient P is a limited growth function,called Monod or Michealis-Menton kinetics.

Q RS�T U V S P U QXW S Q U YZ where

S P U V PP

[ RS�T U V W Q [with

\ PS�T Ucontrol, QS \ U Z [S \ U

known.Lecture1 – p.52/55

Objective

We wish to minimize the final contaminantconcentration and total injection of the nutrient.Thus, the objective functional

] ^ _ `a b ^�c _d c

should be minimized.Hamiltonian

e b fhg ^ ^ b _"i j i k _ f k ^ j i ] _

Lecture1 – p.53/55

Two adjoints

l mn o�p q r s tu tBvl mw o�p q r s tu tBxl n o q r yz l w o q r {

Lecture1 – p.54/55

Optimality Condition

| } ~�� � � } �"� � � � � ~ � � � � � �with� } � | ����� �� ��� | � ~h� � � �� �� � ��

When } � ��� � � �

, then } � | � � ~ � � � � � � �

One must check that this expression is positive.

Lecture1 – p.55/55