Machine learning lecture 1 course notes  (Page 3/13)

1. Repeat until convergence $\{$
   1. ${\theta }_j:={\theta }_j+\alpha {\sum }_{i=1}^m\left(y^{\left(i\right)},-,h_{\theta },\left(x^{\left(i\right)}\right)\right)x_j^{\left(i\right)}$ (for every $j$ ).
2. $\}$

The reader can easily verify that the quantity in the summation in the update rule above is just $\partial J\left(\theta \right)/\partial {\theta }_j$ (for the original definition of $J$ ). So, this is simply gradient descent on the original cost function $J$ . This method looks at every example in the entire training set on every step, and iscalled batch gradient descent . Note that, while gradient descent can be susceptible to local minima in general, the optimization problem we haveposed here for linear regression has only one global, and no other local, optima; thus gradient descent always converges (assuming the learning rate $\alpha$ is not too large) to the global minimum. Indeed, $J$ is a convex quadratic function. Here is an example of gradient descent as it is run to minimize a quadratic function.

The ellipses shown above are the contours of a quadratic function. Also shown is the trajectory taken by gradient descent, which was initializedat (48,30). The $x$ 's in the figure (joined by straight lines) mark the successive values of $\theta$ that gradient descent went through.

When we run batch gradient descent to fit $\theta$ on our previous dataset, to learn to predict housing price as a function of living area, weobtain ${\theta }_0=71.27$ , ${\theta }_1=0.1345$ . If we plot $h_{\theta }\left(x\right)$ as a function of $x$ (area), along with the training data, we obtain thefollowing figure:

If the number of bedrooms were included as one of the input features as well, we get ${\theta }_0=89.60,{\theta }_1=0.1392$ , ${\theta }_2=-8.738$ .

The above results were obtained with batch gradient descent. There is an alternative to batch gradient descent that also works very well.Consider the following algorithm:

1. Loop $\{$
   1. for i=1 to m, $\{$
      1. ${\theta }_j:={\theta }_j+\alpha \left(y^{\left(i\right)},-,h_{\theta },\left(x^{\left(i\right)}\right)\right)x_j^{\left(i\right)}$ (for every $j$ ).
   2. $\}$
2. $\}$

In this algorithm, we repeatedly run through the training set, and each time we encounter a training example, we update the parametersaccording to the gradient of the error with respect to that single training example only. This algorithm is called stochastic gradient descent (also incremental gradient descent ). Whereas batch gradient descent has to scan through the entire training setbefore taking a single step—a costly operation if $m$ is large—stochastic gradient descent can start making progress right away, and continues to makeprogress with each example it looks at. Often, stochastic gradient descent gets $\theta$ “close” to the minimum much faster than batch gradient descent. (Note however that it may never “converge” to the minimum,and the parameters $\theta$ will keep oscillating around the minimum of $J\left(\theta \right)$ ; but in practice most of the values near the minimum will be reasonably good approximations to the true minimum. While it is more common to run stochastic gradient descent as we have described it and with afixed learning rate $\alpha$ , by slowly letting the learning rate $\alpha$ decrease to zero as the algorithm runs, it is also possible to ensure that the parameters will converge to the global minimum rather then merely oscillatearound the minimum. ) For these reasons, particularly when the training set is large, stochastic gradient descent is often preferred over batch gradient descent.