Investigating Gradient Descent behavior in linear regression
I used 2-dimentional datasets with different shapes like blobs, circles, moons, s_curves and swiss_roll to investigate Gradient Descent behavior in linear regression. First, with the closed-form solution, I obtained the line corresponding to the final solution of the linear regression (line-of-best-fit) and drew it. Then, with the Gradient Descent algorithm, I obtained the optimal response of the linear regression parameters for different learning rate values of 0.5, 0.2, 0.01 and 0.9. I also drew the starting (orange), middle (blue) and final (green) lines.
For each data set, I drew the 3-dimensional view of the loss function and for each learning rate, I showed the movement of the Gradient Descent method to the global optimal minimum on the 3-dimensional shape and its 2-dimensional contour.
In solving with GD method, all figures were drawn for the following three error functions.
