Introduction to Backpropagation from Scratch

This short textbook is the second course in the “From Scratch” series. In the previous course, we went over the mathematical derivation of backpropagation for feed-forward neural networks. As we explore new architectures in the upcoming courses, it’s good to get an understanding of how libraries, such as PyTorch, implement a more general form of backpropagation.

To do this, we’re going to build a tiny backpropagation engine using Python. This is because doing so forces you to understand exactly what’s happening and fills in any gaps in your understanding that you have missed.

Course Structure

This course is focused and practical. We start by developing an intuition for why backpropagation is necessary and how it works conceptually. We then implement a working backpropagation engine that can automatically compute gradients for any function built from the operations it supports. Finally, we validate the engine by using it to train a small neural network.

Prerequisites

The only hard prerequisites are Python and comfort with basic calculus (derivatives and the chain rule). Having completed the previous course in this series—Neural Networks from Scratch—is recommended since it gets you familiar with backpropagation, and you will understand the neural network we train in chapter 3.

Looking Forward

We’ll start in Chapter 1 by building intuition for what backpropagation is doing and why simpler approaches fall short. From there, Chapter 2 puts that understanding into code, and Chapter 3 applies it to a simple neural network.