A primer on designing neural networks. Volume I: A tour of the land
Neural networks surround us, in the form of large language models, speech transcription systems, molecular discovery algorithms, robotics, and much more. Stripped of anything else, neural networks are compositions of differentiable primitives, and studying them means learning how to program and how to interact with these models, a particular example of what is called differentiable programming.
This primer is an introduction to this field, imagined for someone who, like Alice, has just ventured into this strange differentiable wonderland. I overview the basics of optimizing a function via automatic differentiation, and a selection of the most common designs for handling sequences, graphs, texts, and audio. The focus is an intuitive, self-contained introduction to the most important design techniques, including convolutional, attentional, and recurrent blocks, bridging the gap between theory and code (PyTorch and JAX) and leaving the reader able to understand some of the most advanced models out there, such as large language models and multimodal architectures.
Get the book
Free PDF
The latest version of the book, free to download.
Download the PDF · updated 29 August 2025
For the differences between the printed and the digital versions, see the errata list.
Additional chapters 2
Here I publish additional chapters on advanced material that I could not fit into the first volume. Eventually, I hope they will become part of a second volume. They are drafts and may still contain errors, so corrections and suggestions are very welcome.
Draft Updated 15 October 2026Extends Chapter 6
Advanced automatic differentiation
Differentiation in generic vector spaces, computational graphs, implicit differentiation (briefly), and an overview of multilinear algebra.
Draft Updated 15 October 2026
Post-training techniques
A historical overview of post-training for large language models, from 2012 to the present, and a technical focus on reinforcement learning.
Contents of Volume I
- Foreword and introduction
- Mathematical preliminaries
- Datasets and losses
- Linear models
- Fully-connected layers
- Automatic differentiation
- Convolutional layers
- Convolutions beyond images
- Scaling up the models
- Transformer models
- Transformers in practice
- Graph layers
- Recurrent layers
- Probability theory
- 1D universal approximation
Learn with the book
- Guided lab sessions covering most of the topics in the book, in PyTorch and JAX, updated continuously.
- I use the book as the reference text for my courses on neural networks. Their slides and notebooks are on the teaching materials site.
- An unofficial (incomplete) Italian translation by Giovanni Borrelli.
License and citation
The book is released under the CC BY-SA 4.0 license. You can share, remix and adapt it in any medium or format, including for commercial use, as long as you give attribution and release the modified material under the same license.
To cite the book:
@article{scardapane2024alice,
title = {Alice's Adventures in a Differentiable Wonderland --
Volume I, A Tour of the Land},
author = {Scardapane, Simone},
journal = {arXiv preprint arXiv:2404.17625},
year = {2024}
}