I've had questions to this. This document is "kinda optional"; the confusion I think I might have caused, is not going to be an exam topic - but then, you might want to understand what is going on, and besides the document will cover more of that Seminar 12 problem in case you did not understand it.

Setup: $LaTeX: x_{ t+1}=g(x_t,u_t)$ (no explicit "t") and payoff is exp-discounting times a function of (x_t, u_t)

This is "the most relevant" simplication: time enters as exponential discounting, and otherwise it is just indexing; if the planning horizon is "2021 to 2025" or "2022 to 2026" you can accommodate that by shifting time "zero" to the first period.

Deviating from the problem as formulated: "K" for current-value "J" for ... the setup more suited to the book.

The problem posted used "J" when formulated with a discounting $LaTeX: \beta^{t-t_0}$ , the solution used it when the discounting was $LaTeX: \beta^t$ and LaTeX: t_0 were zero. In this note, I will use the "K" (mnemonic for "kurrent") for the $LaTeX: \beta^{t-t_0}$ . Note, in both cases it is understood that $LaTeX: T\geq t_0\geq 0$ , but I don't think that confused anyone.

For both the the "J" function and the "K" function, the value $LaTeX: \displaystyle\max_{u_t\geq 0}\Big\{\beta^{T}(x_T+\sqrt{x_T})+\sum_{t=0}^{T-1}\beta^{t}\big(u_t+\sqrt{u_t}\big)\Big\}$ is the special case of both the following two formulations:

$LaTeX: \displaystyle J_{t_0}(x)=\max_{u_t\geq 0}\Big\{\beta^{T}(x_T+\sqrt{x_T})+\sum_{t=t_0}^{T-1}\beta^{t}\big(u_t+\sqrt{u_t}\big)\Big\}$

$LaTeX: \displaystyle K_{t_0}(x)=\max_{u_t\geq 0}\Big\{\beta^{T-t_0}(x_T+\sqrt{x_T})+\sum_{t=t_0}^{T-1}\beta^{t-t_0}\big(u_t+\sqrt{u_t}\big)\Big\}$

both starting at $LaTeX: x_{t_0}=x.$ Now look at how they differ when LaTeX: t_0=T and the interpretations:

$LaTeX: \displaystyle J_T(x)=\beta^{T}(x+\sqrt{x})$ is the present value viewed from time zero, of
"you will enter the program at time T with state x"

$LaTeX: \displaystyle K_T(x)=x+\sqrt{x}$ is the current at T value of
"now time is T and you enter the program with state x"

How to get to the dynamic programming equation?

We want to get to the equation where we are to maximize the sum of period t-payoff and t+1 value. Look first at the un-optimized performance, just inserting arbitrary states and controls, starting from time LaTeX: t_0 . Because I split the sum at "t", I need to use another variable s as summation index.

The present value formulation has performance: $LaTeX: \displaystyle\sum_{s=t_0}^{t-1}\beta^{s}\big(u_s+\sqrt{u_s}\big)+\beta^{t}\big(u_t+\sqrt{u_t}\big)+\sum_{s=t+1}^{T-1}\beta^{s}\big(u_s+\sqrt{u_s}\big)+\beta^{T}\big(x_T+\sqrt{x_T}\big)$ = $LaTeX: \beta^{t_0}$ times the formula in the right column.

At time "t" the first sum is "sunk cost" and we are left with [first sum] + $LaTeX: \displaystyle\beta^{t}\big(u_t+\sqrt{u_t}\big)+\sum_{s=t+1}^{T-1}\beta^{s}\big(u_s+\sqrt{u_s}\big)+\beta^{T}\big(x_T+\sqrt{x_T}\big)$
which equals $LaTeX: \displaystyle\beta^{t}\big(u_t+\sqrt{u_t}\big)+\text{[what when optimized becomes $J_{t+1}(g(x_t,u_t))$]}$

because the two latter terms optimized, is what you get - present value evaluated at zero! - of
inheriting the program at time t+1 with state $LaTeX: x_{t+1}=g(x_t,u_t)$

The current value formulation has performance: $LaTeX: \displaystyle\sum_{s=t_0}^{t-1}\beta^{s-t_0}\big(u_s+\sqrt{u_s}\big)+\beta^{t-t_0}\big(u_t+\sqrt{u_t}\big)+\sum_{s=t+1}^{T-1}\beta^{s-t_0}\big(u_s+\sqrt{u_s}\big)+\beta^{T-t_0}\big(x_T+\sqrt{x_T}\big)$ = $LaTeX: \beta^{-t_0}$ times the formula in the left column.

At time "t" the first sum is "sunk cost" and we are left with [first sum] + $LaTeX: \displaystyle\beta^{t-t_0}\Big[u_t+\sqrt{u_t}+{\color{magenta}{\beta}\cdot\;}\sum_{s={\color{magenta}{t+1}}}^{T-1}\beta^{s-({\color{magenta}{t+1}})}\big(u_s+\sqrt{u_s}\big)+{\color{magenta}{\beta}\cdot\;}\beta^{T-({\color{magenta}{t+1}})}\big(x_T+\sqrt{x_{T}}\big)\Big]$ which equals $LaTeX: \displaystyle\beta^{t-t_0}\Big[\big(u_t+\sqrt{u_t}\big)+{\color{magenta}{\beta}\cdot\;}\text{[what when optimized becomes $K_{t+1}(g(x_t,u_t))$]}\Big]$

- note the magenta parts. The "t+1" takes the role of the t_0, but current-at-time-t there is an extra beta discounting to the current-at-t+1 value of inheriting the program with state $LaTeX: x_{t+1}=g(x_t,u_t)$

Notice that what is to be maximized, is the same thing up to a positive number, so the optimal choice is unaffected by the formulation.

That was how to formulate. Next up: how to solve this particular problem. Note, "solvability by hand" depends crucially upon the constant in $LaTeX: x_{t+1}=(x_t-u_t)/\beta$ being 1/discount factor, cancelling a term. Let me first jump at the induction part, and if you want to see the T-1 and T-2 calculations, they follow afterwards.

This particular problem $LaTeX: x_{t+1}=(x_t-u_t)/\beta$ solved by verifying a form by induction.

We guess and verify the following form:

Claim: discounted to zero present value has the form

$LaTeX: \displaystyle\beta^t\cdot\big(x+\sqrt{A_{T-t}x\:}\,\big)$

True at t=T with LaTeX: A_0=1 . Suppose for induction: true at t+1. Then at t, we get:

$LaTeX: \displaystyle\max_{u\geq 0}\Big\{\beta^{t}\big(u+\sqrt u\big)+\beta^{t+1}\big(g(x,u)+\sqrt{A_{T-(t+1)}g(x,u)\:}\,\big)\Big\}$

which we can write, with short form "A" for $LaTeX: A_{T-(t+1)}$ , as

$LaTeX: \beta^t\displaystyle\max_{u\geq 0}\Big\{u+\sqrt u+\beta \frac{x-u}\beta+\beta \sqrt{A\frac{x-u}\beta\:}\Big\}$
which is just a discounting times the formula in the right column.

Tidying up a $LaTeX: \beta$ , cancelling a u and taking the x out of the maximization:

$LaTeX: \beta^tx+\beta^t\displaystyle\max_{u\geq 0}\Big\{\sqrt u+\sqrt{\beta A} \sqrt{x-u\:}\Big\}$

Claim: current at t value has the form

$LaTeX: x+\sqrt{A_{T-t}x\:}$

True at t=T with LaTeX: A_0=1 . Suppose for induction: true at t+1. Then at t, we get:

$LaTeX: \displaystyle\max_{u\geq 0}\Big\{u+\sqrt u+\beta\big(g(x,u)+\sqrt{A_{T-(t+1)}g(x,u)\:}\,\big)\Big\}$

which we can write, with short form "A" for $LaTeX: A_{T-(t+1)}$ , as

$LaTeX: \displaystyle\max_{u\geq 0}\Big\{u+\sqrt u+\beta \frac{x-u}\beta+\beta \sqrt{A\frac{x-u}\beta\:}\Big\}$
which is just the formula in the right column except no discounting.

Tidying up a $LaTeX: \beta$ , cancelling a u and taking the x out of the maximization:

$LaTeX: x+\displaystyle\max_{u\geq 0}\Big\{\sqrt u+\sqrt{\beta A} \sqrt{x-u\:}\Big\}$

The maximization is the same for both, of course. As a function of u, the maximand is concave and with vertical tangent at 0 and x; therefore it must have maximum at a stationary point. The FOC reads $LaTeX: 0=\frac1{2\sqrt u}-\frac1{2\sqrt{x-u}}\sqrt{\beta A\:}$ , cross-multiply to get $LaTeX: \sqrt{x-u}=\sqrt{\beta A\:}\cdot\sqrt{u}$ . This is going to become convenient when we shall later insert for $LaTeX: \sqrt{\beta A\:}\sqrt{x-u}=\beta A\cdot\sqrt{u}$ to get $LaTeX: \sqrt u+\sqrt{\beta A} \sqrt{x-u\:}=(1+\beta A)\sqrt u$ . But now we square and solve: $LaTeX: x-u=\beta A u$ yields $LaTeX: u=\frac1{1+\beta A}x$ , and so $LaTeX: \max_{u\geq 0}\Big\{\sqrt u+\sqrt{\beta A} \sqrt{x-u\:}\Big\}=(1+\beta A)\sqrt u\Big\vert_{u=x/(1+\beta A)}=\sqrt{(1+\beta A)x\:}$ , verifying the induction claim with $LaTeX: A_{k+1}=1+\beta A_k.$

Here is a trick if you want one

... if you did spot it after I used it in the lecture, good for you. When x=0 we must choose u=0, so there is no restriction in assuming u=hx, and then because u is anyway restricted to be no greater than x, h must be between 0 and 1.

$LaTeX: \ \ x+\displaystyle\max_{u\geq 0}\Big\{\sqrt u+\sqrt{\beta A} \sqrt{x-u\:}\Big\} \\=\displaystyle x+\max_{ h\in[0,1]}\Big\{\sqrt h\sqrt x+\sqrt{\beta A} \sqrt x\cdot\sqrt{1-h\:}\Big\}\\ \displaystyle=x+\underbrace{\max_{ h\in[0,1]}\Big\{\sqrt h+\sqrt{\beta A} \cdot\sqrt{1-h\:}\Big\}}_{=\ \text{previous period's }A }\cdot \sqrt x$

This immediately proves the form, but not the difference equation for A. For that, go on do a special case of the same maximization.

Doing "T-1" manually

That is maybe more hands-on. Discounted to zero value of entering the program at T-1 at state x:

$LaTeX: \displaystyle\max_{u\geq 0}\Big\{\beta^{T-1}\big(u+\sqrt u\big)+\beta^{T}\big(g(x,u)+\sqrt{g(x,u)\:}\,\big)\Big\}$

Inserting for g, to get:

$LaTeX: \displaystyle\max_{u\geq 0}\Big\{\beta^{T-1}\big(u+\sqrt u\big)+\beta^{T}\frac{x-u}\beta +\beta^{T}\sqrt{\frac{x-u}\beta} \big)\Big\} \\\displaystyle =\max_{u\geq 0}\Big\{\beta^{T-1}\big(u+\sqrt u\big)+\beta^{T-1}x-\beta^{T-1}u+\beta^{T-1/2}\sqrt{x-u}\Big\} \\\displaystyle=\beta^{T-1}x+\max_{u\geq 0}\Big\{\beta^{T-1}\sqrt u+\beta^{T-1/2}\sqrt{x-u}\Big\}$

Now you can do the same routine using the FOC as the above. You will get $LaTeX: \beta^{T-1}\cdot\frac1{2\sqrt u}=\beta^{T-1/2}\frac1{2\sqrt{x-u}}$ , cancel $LaTeX: \beta^{T-1}/2$ to get $LaTeX: \sqrt{\beta u}=\sqrt{x-u}$ , square and solve to get $LaTeX: \beta^{T-1}x+\beta^{T-1}\cdot\sqrt{1+\beta\:}\sqrt x$

For the current value setup, you get a similar thing except a multiplicative factor $LaTeX: \beta^{T-1}$ is already gone.