Single-Variable Gradient Descent

Hopefully, a very straightforward approach to understanding gradient descent

I wanted to create a very accessible and straightforward walk-through of a foundational machine learning topic: gradient descent. There are plenty of resources out there. What might distinguish this one is that I work through a single-variable scenario, while the actual implementation is of course composed of many variables that will be iteratively adjusted. However, the principles behind it are the same, so if you’ve consulted resources explaining gradient descent in its full-form before and still walked away with some confusion, this might be helpful in that it’s a going back to a first-principles approach. The only downside, of course, is that this is a highly synthetic scenario. But nevertheless, one that still holds up as a hopefully friendly introduction to gradient descent.

Brief Review of Machine Learning Foundations

Optimization is at the heart of machine learning, the ability to fine-tune models to perform better on unseen data is what differentiates good and bad models. The process to optimize neural networks is usually by an algorithm known as backpropagation, which as its name indicates works from “back-to-front” (last layer to first), it’s a process used to update the weights and biases (these are parameters inside the network) through gradient descent in order to minimize the cost or loss function, which essentially measures the error in the model’s prediction. Gradient descent is an iterative optimization algorithm for finding the minimum of a function. By repeatedly adjusting the parameter(s) in the opposite direction of the gradient, it converges towards a local minimum, with the size of each step being determined by the learning rate of the model. However, this is multivariable scheme, and we’re trying to illustrate this with a very easy and straightforward example. Therefore, we’ll isolate a specific scenario in the process where we can do a single-variable computation

The Setup

One of the foundational algorithms in machine learning is logistic regression, which is used in binary calculations. We’ll work to optimize the parameters in a logistic regression model to correctly classify data points.

Logistic regression: 

Given an input x, a logistic regression model predicts the probability p that x belongs to the positive class (1), using the sigmoid function:

Loss Function

Gradient Descent Update Rule

The rule states that the updating of a parameter consists of subtracting the learning rate times the derivative of the loss function with respect to the probability from the old parameter.

Optimization

Data

Putting It Together

Gradient of The Loss Function

Iterative Optimization

Conclusion

From these iterations, we can see that the value of our parameter θ is increasing slightly with each iteration, which is indicative of moving in the direction of minimizing the loss function (we’re moving against the gradient). Our predicted probability remains relatively constant through iterations, but this is likely due to the small learning rate value. However, by maintaining the learning rate small, we ensure that we are able to see the convergence of the values arise more clearly.

Gradient Descent is a universal optimization technique that can be applied to a wide variety of machine learning algorithms, not just logistic regression. Its effective and simple approach to finding local minima makes it very suitable to many machine learning tasks. From linear regression to minimize mean squared error between predicted and actual values to deep neural networks, in which it plays a crucial role in backpropagation (see my previous post for that) where it’s used to minimize the loss function by adjusting the neural network’s weights and biases.

Thank you so much for reading! I hope this short post was helpful either as an introduction or as review of some foundations. And sorry for the weird formatting, unfortunately with the current limitations of Substack’s editor I’m left to just paste screenshots from my LaTeX document in many occasions (Please, Substack team, add at least some basic math notation to the editor!) More machine learning stuff to come of course! I want to do a thorough review of linear regression for nonlinear relationships as it’s used in deep learning and also something on the current interpretability landscape. So be on the lookout for those future posts!