Ioannis Mitliagkas

Adaptive batch-size rules rely on the gradient noise scale, which assumes the Euclidean geometry of SGD. We derive the noise scale for sign and spectral methods (Signum, Muon) from their dual norms, and estimate it cheaply from gradients already on each data-parallel worker. On a 160M-parameter Llama model, this matches the loss of fixed batch sizes with up to 66% fewer training steps.

Theory often explains Adam with assumptions that do not change when the parameter space is rotated. We show that Adam on transformers gets worse under random rotations, while some structured rotations help, and that existing rotation- dependent assumptions do not explain this. The orthogonality of Adam's update emerges as a promising quantity for a better theory.

Why do stochastic first-order methods train non-convex networks so reliably? We measure two quantities along the optimization path, tied to the restricted secant inequality and the error bound, and find that they behave predictably across vision and language tasks, architectures and optimizers. These properties are enough to guarantee linear convergence and to motivate learning-rate schedules like those used in practice.

Next-token prediction trains a model to look one step ahead, which limits long- horizon reasoning and planning. We add an auxiliary head that predicts a summary of the long-term future, either handcrafted (a bag of words) or learned with a reverse language model. In 3B and 8B pretraining runs, it beats next-token and multi-token prediction on math, reasoning and coding benchmarks.

Classifiers fail when test data contain combinations of attributes never seen together in training. We model each attribute as an additive energy term, train a classifier with that structure, then adjust it for the new combinations. The method, compositional risk minimization, provably extrapolates beyond the training combinations and is more robust than methods built for subpopulation shift.

When an image is a sum of object-specific parts, a decoder with the same additive structure recovers each object's latent variables from reconstruction alone, with no independence assumptions on the latents. The same decoder can then generate combinations of objects never seen together in training. This gives a theoretical account of object-centric decoders and of how disentanglement enables extrapolation.

Certified unlearning adds noise, and the noise costs utility, especially for large deletion requests. We show that mixing in public data cuts the cost of unlearning by a factor of O(1/n_pub²), and we account for shifts between the public and private data. This makes it practical to unlearn a constant fraction of the training set while keeping utility.

We ran the first NeurIPS machine unlearning competition, with nearly 1,200 teams, scoring forgetting through a formal definition of unlearning together with model utility. The top entries beat existing algorithms, and their ranking stays stable across variations of the evaluation, so it can be made cheaper. We also analyze where the methods succeed and fail.

Many problems in deep learning, such as min-max games and projected Bellman error minimization, are variational inequalities, and standard gradient methods cycle or diverge on them. We solve them through a sequence of surrogate losses that work with standard optimizers such as Adam, with convergence guarantees under hidden monotonicity. The approach also yields a more compute- and sample-efficient variant of TD(0).

Min-max training, as in GANs, stalls because of rotational dynamics. Treating the two players as particles under physical forces, we derive LEAD, a second-order method that damps the rotation. It converges linearly on quadratic games and improves GAN training on CIFAR-10.

We introduce magnetic mirror descent, one algorithm that serves both as an equilibrium solver and as a reinforcement learning method for two-player zero-sum games. It is the first solver of quantal response equilibria with linear convergence in extensive-form games from first-order feedback, and the first standard RL algorithm competitive with counterfactual regret minimization in tabular settings. As a deep self-play method, it does well on Dark Hex and Phantom Tic-Tac-Toe.