Optimal control, part 2

In the first part (here), we introduced Bellman’s idea of backward induction. But what if we consider now infinite time horizon ? Actually, the maths will be even more simple… and we will be able to use fixed pointed theorem to derive solutions.

  • The mathematical framework

Here, consider the following value function

https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-01.png

and define

https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-02.png

A sequence https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-04.png is said to be an admissible solution for starting point x,

https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-05.png

if https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-06.png and https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-07.png. If we reformulate the dynamic programming idea, we obtain that if https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-04.png is a solution to problem https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-08.png, then for all https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-09.png, sequence https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-10.png is a solution to problem https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-11.png. It comes that function v is a solution of Bellman’s equation

https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-12.png

Note that is can be ssen as some fixed point resul, since

https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-13.png

i.e.

https://perso.univ-rennes1.fr/arthur.charpentier/latex/opt-contr-14.png

So far, it shouldn’t be so hard….

  • Frank Ramsey’s model (discrete version)

In 1928, Frank Ramsey wanted to understand the amount of savings in a dynamic perspective (in how much of its income should a nation save). Consider the following infinite horizon problem, where some planifier wants to maximize

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-01.png

subject to constraints https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-02.png https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-03.png and https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-04.png.
Before looking at dynamic programming answers, we might start with standard Lagrangian optimization techniques.
Assuming concavity of utility function and production function, we should look only for interior solutions. Define the Lagrangian as

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-05.png

Thus, the first order conditions are then given by

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-06.png

and

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-088.png

Assume further some terminal condition, e.g.

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-08.png

(also called transversality condition). If we combine those two conditions, and assume that the first constraint is saturated, we obtain the so-called Euler equation,

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-09.png

It is also possible to use Bellman’s equation: given the dynamic of the capital

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-10.png
https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-11.png

The first order condition states

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-12.png

But since v is unknown, so is its derivative. But from the enveloppe theroem, we obtain something like

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-13.png

where

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-14.png

We can then write

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-15.png

i.e.

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-16.png

and finally

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-17.png

which is, Euler’s equation.

  • A specified model, with calculations

As in the previous post (here), consider a log utility function, and a power production function,https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-18.png and https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-19.png. The dynamic is then

 https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-20.pngand https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-21.png

Note that fixed points are here

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-22.png

and

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-23.png

Recall that the value function is defined as

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-24.png

A natural idea to derive the value function can be to iterate, i.e.

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-25.png

starting with a simple function, e.g the null function, at step 0. At step n=1

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-26.png

thus

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-27.png

At step n=2,

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-28.png

i.e.

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-29.png

The first order condition is then

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-30.png

and thus, we obtain

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-31.png

that can be plugged in the previous equation, i.e.

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-32.png

At step 3, we start from that new expression, derive the first order condition, and we get

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-33.png

and

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-34.png

and so on…
And finally, we can prove that

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-35.png

i.e. https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-36.png. Assuming that

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-37.png

actually, we can prove that

https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-38.png

(and https://perso.univ-rennes1.fr/arthur.charpentier/latex/co-opt-39.png has a form that can be explicited).


Leave a Reply

Your email address will not be published. Required fields are marked *