2.12 Machine learning lecture 12 course notes  (Page 8/8)

In detail, the algorithm is as follows:

1. Randomly sample $m$ states $s^{\left(1\right)},s^{\left(2\right)},...s^{\left(m\right)}\in S$ .
2. Initialize $\theta :=0$ .
3. Repeat $\{$
4. 1. For $i=1,...,m$ $\{$
      1. For each action $a\in A$ $\{$
      2. 1. Sample $s_1^{\text{'}},...,s_k^{\text{'}}\sim P_{s^{\left(i\right)}a}$ (using a model of the MDP).
         2. Set $q\left(a\right)=\frac{1}{k}{\sum }_{j=1}^{k}R\left(s^{\left(i\right)}\right)+\gamma V\left(s_j^{\text{'}}\right)$
         3. $//$ Hence, $q\left(a\right)$ is an estimate of $R\left(s^{\left(i\right)}\right)+\gamma {\mathrm{E}}_{s^{\text{'}}\sim P_{s^{\left(i\right)}a}}\left[V\left(s^{\text{'}}\right)\right]$ .
      3. $\}$
      4. Set $y^{\left(i\right)}={max}_{a}q\left(a\right)$ .
      5. $//$ Hence, $y^{\left(i\right)}$ is an estimate of $R\left(s^{\left(i\right)}\right)+\gamma {max}_a{\mathrm{E}}_{s^{\text{'}}\sim P_{s^{\left(i\right)}a}}\left[V\left(s^{\text{'}}\right)\right]$ .
   2. $\}$
   3. $//$ In the original value iteration algorithm (over discrete states)
   4. $//$ we updated the value function according to $V\left(s^{\left(i\right)}\right):=y^{\left(i\right)}$ .
   5. $//$ In this algorithm, we want $V\left(s^{\left(i\right)}\right)\approx y^{\left(i\right)}$ , which we'll achieve
   6. $//$ using supervised learning (linear regression).
   7. Set $\theta :=arg{min}_{\theta }\frac{1}{2}{\sum }_{i=1}^m{\left({\theta }^T,\phi ,\left(s^{\left(i\right)}\right),-,y^{\left(i\right)}\right)}^2$
5. $\}$

Above, we had written out fitted value iteration using linear regression as the algorithm to try to make $V\left(s^{\left(i\right)}\right)$ close to $y^{\left(i\right)}$ . That step of the algorithm is completely analogous to a standard supervised learning (regression) problem in which we have a training set $(x^{\left(1\right)},y^{\left(1\right)}),(x^{\left(2\right)},y^{\left(2\right)}),...,(x^{\left(m\right)},y^{\left(m\right)})$ , and want to learn a function mapping from $x$ to $y$ ; the only difference is that here $s$ plays the role of $x$ . Eventhough our description above used linear regression, clearly other regression algorithms (such as locally weighted linear regression) can also be used.

Unlike value iteration over a discrete set of states, fitted value iteration cannot be proved to always to converge. However, in practice, it often does converge (or approximately converge), and works well for many problems.Note also that if we are using a deterministic simulator/model of the MDP, then fitted value iteration can be simplified by setting $k=1$ in the algorithm. This is because the expectation in Equation  [link] becomes an expectation over a deterministic distribution, and so a single example is sufficient to exactly compute that expectation.Otherwise, in the algorithm above, we had to draw $k$ samples, and average to try to approximate that expectation (see the definition of $q\left(a\right)$ , in the algorithm pseudo-code).

Finally, fitted value iteration outputs $V$ , which is an approximation to $V^{*}$ . This implicitly defines our policy. Specifically,when our system is in some state $s$ , and we need to choose an action, we would like to choose the action

The process for computing/approximating this is similar to the inner-loop of fitted value iteration, where for each action, we sample $s_1^{\text{'}},...,s_k^{\text{'}}\sim P_{sa}$ to approximate the expectation. (And again, if the simulator is deterministic, we can set $k=1$ .)

In practice, there're often other ways to approximate this step as well. For example, one very common case is if thesimulator is of the form $s_{t+1}=f(s_t,a_t)+{ϵ}_t$ , where $f$ is some determinstic function of the states (such as $f(s_t,a_t)=A{s}_t+B{a}_t$ ), and $ϵ$ is zero-mean Gaussian noise. In this case, we can pick the action given by

In other words, here we are just setting ${ϵ}_t=0$ (i.e., ignoring the noise in the simulator), and setting $k=1$ . Equivalently, this can be derived from Equation  [link] using the approximation

where here the expection is over the random $s^{\text{'}}\sim P_{sa}$ . So long as the noise terms ${ϵ}_t$ are small, this will usually be a reasonable approximation.

However, for problems that don't lend themselves to such approximations, having to sample $k\left|A\right|$ states using the model, in order to approximate the expectation above, can be computationally expensive.