Conclusion
If you made it to the end of the course, congratulations. You’ve built a neural network that reads handwritten digits using nothing but Python and NumPy, so you got to see the full details of a neural network and understand what’s going on.
What You’ve Learned
In chapter 1, you learned about the architecture of feed forward neural networks. Then in chapter 2, you got to see how to train neural networks using backpropagation and gradient descent. Chapter 3 went over the intuition behind training, and why it works. Lastly, chapter 4 explained what the neural networks were learning, and why they’re so hard to understand.
Where to Go Next
This course was an introduction, and there’s a lot of ground left to explore. Here are some directions you could take.
Continue the “From Scratch” series:
The natural next step is Backpropagation from Scratch, where you’ll build a tiny automatic differentiation engine and see how backpropagation generalizes beyond the layered networks we worked with here.
After that, Optimizing Training from Scratch will go over all the arbitrary decisions made in this course (i.e. using the sigmoid instead of another activation function), and cover techniques that make neural networks train faster and generalize to new data better.
We’ll be releasing more courses and articles over the coming months for many more topics spanning Machine learning such as Convolutional Neural Networks, Recurrent Neural Networks, and Attention/Transformers.
Experiment with your own network. You have a complete implementation that you understand top to bottom, and that makes it a perfect testing ground. You can try to slowly improve on your network by changing the number of hidden layers or neurons and watching what happens to accuracy. Try different learning rates and mini-batch sizes. While doing this, you’ll most likely break the neural network, and then have to figure out why. This is one of the best ways of developing your intuition behind building and debugging neural networks.
Explore the broader field. Feedforward networks are just one paradigm. Convolutional networks dominate computer vision, and transformers power large language models you’ve used such as ChatGPT or Claude. Those architectures introduce new ideas, but the fundamentals are the same ones you now know: weighted sums, activation functions, cost functions, gradient descent, and backpropagation. If you’re interested in LLMs, Andrej Kaparthy’s Zero To Hero course on youtube is a good place to start.
Go deeper on the theory. If you want a more thorough treatment of everything we’ve covered, Michael Nielsen’s “Neural Networks and Deep Learning” remains the best free textbook on the subject, and 3Blue1Brown’s neural network series is unmatched for visual intuition. Both heavily influenced this course and the “from scratch” series.
Thank You
Thank you for taking this course. It was written to be the kind of introduction I wish I’d had when I started: mathematical enough to be honest, but focused on understanding over formality. If you’d like to see everything that inspired and informed the material, the next chapter lists the full credits, and each entry doubles as excellent further reading.
Now go build something.