Theoretical principles for deep learning

Lectures

Tuesday and Thursday, 9h30-11h30

Dates

January 12 to April 15, 2027; no class during the winter break (March 1-5)

First class

Tuesday, January 12, 2027

Location

Mila, room TBA

Instructor

Ioannis Mitliagkas

Registration

IFT 6169 on the Université de Montréal course page, section A (class number 12767)

Language

English

Description

Deep learning works far better than classical theory predicts. Networks with more parameters than data points generalize. Non-convex training reliably finds good solutions. Small changes in scale or parameterization change what a network learns. Over the past few years, theory has started to give sharp answers to these puzzles.

This is a graduate research class on those answers. Its through-line: the same network can behave as a kernel machine or as a feature learner, depending on its scale, its parameterization, the structure of the data and the optimizer. We start from the classical tools of optimization and statistical learning theory, see where they break, and study what replaces them:

• Lazy versus rich training: the neural tangent kernel, mean-field limits, and parameterizations such as muP.
• Provable feature learning: how SGD learns low-dimensional structure in single- and multi-index models, how many samples it needs, and the computational limits.
• Training dynamics: implicit bias, incremental and saddle-to-saddle learning, the edge of stability.
• Generalization: double descent and benign overfitting, non-vacuous PAC-Bayes bounds, information-theoretic bounds, the theory of scaling laws.
• Frontier topics: transformers and in-context learning, optimization at LLM scale, diffusion models.

Lectures are proof-driven. Each session aims for one result that students can reproduce from the notes, and one small experiment that checks its prediction. Students scribe lecture notes, present papers and carry out a research project.

Prerequisites

This is a graduate class for students who want to do theory-driven deep learning research.

• Probability and linear algebra, at the level of a strong undergraduate.
• Machine learning, for example IFT 3395/6390.
• Comfort with reading and writing proofs.

Basic convex optimization and statistical learning theory are pre-reading, not lecture material. A diagnostic Homework 0 at the start of the term tells you what to review; the references below include the texts for it.

Evaluation

These weights are a first draft and may change before the start of the semester.

Research project. Three options: reproduce and extend a theorem numerically, write a literature synthesis around one of the themes of the class, or do original research. Every project has a check-in midway through the term and ends with a presentation.

Paper presentation. One paper per student, drawn from the reading lists of the feature-learning, dynamics and frontier blocks.

Scribing. Students take turns writing up lectures in LaTeX, using the class template. Scribed notes are due one week after the lecture.

Schedule

First draft: topics and order may change before the start of the semester, and during the term. Notes, slides and readings will be posted here as the term goes.

Generative AI use

In this class, the use of generative AI tools is permitted, but restricted. These tools may be used only for the specific learning-support uses listed below. Any use beyond these is considered academic fraud. The permitted uses are:

• finding definitions of concepts;
• planning a presentation;
• language revision of texts;
• helping you understand complex concepts;
• helping you learn to program and improve your own code.

The use of these tools is forbidden for the following:

• writing solutions to problem sets, including proofs;
• writing scribed lecture notes;
• writing the project report.

Transparency is required: the generative AI use declaration form, available on the libraries’ website, is recommended. Logging your conversations with generative AI is mandatory. The libraries’ website gives guidance on this. When you submit work for which you used AI tools, you must include a short “AI use” section in your report explaining in detail how you used AI.

Students are responsible for the accuracy of any material produced with these tools.

Any use of generative AI tools beyond the permitted uses above is considered academic fraud.

References

Main texts and surveys for the class. Readings for each session will be added to the schedule.

1. Learning Theory from First Principles, Francis Bach. MIT Press, 2024 (free PDF).
2. Deep Learning Theory, lecture notes, Matus Telgarsky.
3. Six Lectures on Linearized Neural Networks, Theodor Misiakiewicz and Andrea Montanari.
4. A Survey on Algorithms for Multi-Index Models, Joan Bruna and Daniel Hsu. Statistical Science, 2025.
5. The Principles of Deep Learning Theory, Daniel A. Roberts, Sho Yaida and Boris Hanin. Cambridge University Press, 2022.
6. Deep learning: a statistical viewpoint, Peter L. Bartlett, Andrea Montanari and Alexander Rakhlin. Acta Numerica, 2021.
7. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation, Mikhail Belkin. Acta Numerica, 2021.
8. User-friendly introduction to PAC-Bayes bounds, Pierre Alquier. Foundations and Trends in Machine Learning, 2024.
9. Convex Optimization: Algorithms and Complexity, Sébastien Bubeck. Pre-reading for Homework 0.

Past editions

The Winter 2023 edition is the last edition of the original class. It covers the optimization and learning-theory foundations in more depth, and its lecture notes remain online.