Introduction to Optimizing Training from Scratch

This course is the third in the “From Scratch” series. In the previous courses, we built a neural network and derived backpropagation from first principles. Now we take that working network and focus on making it learn faster, generalize better, and train more reliably.

Prerequisites

Multivariable calculus and linear algebra is required. Understanding the material in Neural Networks from Scratch is strongly recommended. However, you can use this course to do a deep dive into individual topics. Backpropagation from Scratch is conceptually useful but optional.

Looking Forward

We’ll start by looking at activation and cost functions, seeing how these choices shape what a network can learn and how quickly it learns. From there, we’ll turn to regularization and normalization to keep training stable and prevent overfitting, before diving into techniques that accelerate gradient descent itself. We’ll close by tying it all together with a practical guide to tuning hyperparameters.