Publications
Peer-reviewed papers, workshop papers and preprints, newest first. Also on Google Scholar and DBLP.
Peer-reviewed
2026
- A Surrogate Perspective on Convergence of Fixed-Target DQN
-
Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data
Certified unlearning adds noise, and the noise costs utility, especially for large deletion requests. We show that mixing in public data cuts the cost of unlearning by a factor of O(1/n_pub²), and we account for shifts between the public and private data. This makes it practical to unlearn a constant fraction of the training set while keeping utility.
-
Adaptive Batch Sizes Using Non-Euclidean Gradient Noise Scales for Stochastic Sign and Spectral Descent
Adaptive batch-size rules rely on the gradient noise scale, which assumes the Euclidean geometry of SGD. We derive the noise scale for sign and spectral methods (Signum, Muon) from their dual norms, and estimate it cheaply from gradients already on each data-parallel worker. On a 160M-parameter Llama model, this matches the loss of fixed batch sizes with up to 66% fewer training steps.
- Generalization Measures under Controlled Covariate Shift: A Regime-Aware Benchmark
-
Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
Next-token prediction trains a model to look one step ahead, which limits long- horizon reasoning and planning. We add an auxiliary head that predicts a summary of the long-term future, either handcrafted (a bag of words) or learned with a reverse language model. In 3B and 8B pretraining runs, it beats next-token and multi-token prediction on math, reasoning and coding benchmarks.
- A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyond
- DisTaC: Conditioning Task Vectors via Distillation for Robust Model Merging
2025
- An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration
-
Understanding Adam Requires Better Rotation Dependent Assumptions
Theory often explains Adam with assumptions that do not change when the parameter space is rotated. We show that Adam on transformers gets worse under random rotations, while some structured rotations help, and that existing rotation- dependent assumptions do not explain this. The orthogonality of Adam's update emerges as a promising quantity for a better theory.
- Pseudo-Asynchronous Local SGD: Robust and Efficient Data-Parallel Training
-
Compositional risk minimization
Classifiers fail when test data contain combinations of attributes never seen together in training. We model each attribute as an additive energy term, train a classifier with that structure, then adjust it for the new combinations. The method, compositional risk minimization, provably extrapolates beyond the training combinations and is more robust than methods built for subpopulation shift.
-
Solving hidden monotone variational inequalities with surrogate losses
Many problems in deep learning, such as min-max games and projected Bellman error minimization, are variational inequalities, and standard gradient methods cycle or diverge on them. We solve them through a sequence of surrogate losses that work with standard optimizers such as Adam, with convergence guarantees under hidden monotonicity. The approach also yields a more compute- and sample-efficient variant of TD(0).
2024
- Feature learning as alignment: a structural property of gradient descent in non-linear neural networks
-
Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition
We ran the first NeurIPS machine unlearning competition, with nearly 1,200 teams, scoring forgetting through a formal definition of unlearning together with model utility. The top entries beat existing algorithms, and their ranking stays stable across variations of the evaluation, so it can be made cheaper. We also analyze where the methods succeed and fail.
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection
-
No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths
Why do stochastic first-order methods train non-convex networks so reliably? We measure two quantities along the optimization path, tied to the restricted secant inequality and the error bound, and find that they behave predictably across vision and language tasks, architectures and optimizers. These properties are enough to guarantee linear convergence and to motivate learning-rate schedules like those used in practice.
- Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation
2023
-
Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation
When an image is a sum of object-specific parts, a decoder with the same additive structure recovers each object's latent variables from reconstruction alone, with no independence assumptions on the latents. The same decoder can then generate combinations of objects never seen together in training. This gives a theoretical account of object-centric decoders and of how disentanglement enables extrapolation.
- CADet: Fully Self-Supervised Out-Of-Distribution Detection With Contrastive Learning
- A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods
- Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize
- Synergies Between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning
- Empirical Study on Optimizer Selection for Out-of-Distribution Generalization
-
LEAD: Min-Max Optimization from a Physical Perspective
Min-max training, as in GANs, stalls because of rotational dynamics. Treating the two players as particles under physical forces, we derive LEAD, a second-order method that damps the rotation. It converges linearly on quadratic games and improves GAN training on CIFAR-10.
-
A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Games
We introduce magnetic mirror descent, one algorithm that serves both as an equilibrium solver and as a reinforcement learning method for two-player zero-sum games. It is the first solver of quantal response equilibria with linear convergence in extensive-form games from first-order feedback, and the first standard RL algorithm competitive with counterfactual regret minimization in tabular settings. As a deep self-play method, it does well on Dark Hex and Phantom Tic-Tac-Toe.
- Neural Networks Efficiently Learn Low-Dimensional Representations with SGD
- Performative Prediction with Neural Networks
2022
- Gradient Descent Is Optimal Under Lower Restricted Secant Inequality And Upper Error Bound
- Optimal transport meets noisy label robust loss and MixUp regularization for domain adaptation
- Towards efficient representation identification in supervised learning
2021
- Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivity
- Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization
- Adversarial score matching and improved sampling for image generation
- A Study of Condition Numbers for First-Order Optimization
2020
- In search of robust measures of generalization
- Stochastic Hamiltonian Gradient Methods for Smooth Games
- Linear Lower Bounds and Conditioning of Differentiable Games
- Accelerating Smooth Games by Manipulating Spectral Shapes
- A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Differentiable Games
2019
- Reducing the variance in online optimization by transporting past gradients
- Multi-objective training of Generative Adversarial Networks with multiple discriminators
- Manifold Mixup: Better Representations by Interpolating Hidden States
- State-Reification Networks: Improving Generalization by Modeling the Distribution of Hidden Representations
- h-detach: Modifying the LSTM Gradient Towards Better Optimization
- Negative Momentum for Improved Game Dynamics
- YellowFin and the Art of Momentum Tuning
2018
- Learning Representations and Generative Models for 3D Point Clouds
- Accelerated stochastic power iteration
- YellowFin: Adaptive optimization for (A)synchronous systems
2017
- Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
- Improving Gibbs Sampler Scan Quality with DoGS
2016
- Scan Order in Gibbs Sampling: Models in Which it Matters and Bounds on How Much
- Asynchrony begets Momentum, with an Application to Deep Learning
2015
2014
2013
2011
- User Rankings from Comparisons: Learning Permutations in High Dimensions
- Joint Power and Admission Control for Ad-hoc and Cognitive Underlay Networks: Convex Approximation and Distributed Implementation
2010
- Strong Information-Theoretic Limits for Source/Model Recovery
- Distributed Joint Power and Admission Control for Ad-hoc and Cognitive Underlay Networks
2008
Workshop papers
2026
2024
- Smoothness-Adaptive Sharpness-Aware Minimization for Finding Flatter Minima
- Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones
2019
- In Support of Over-Parametrization in Deep Reinforcement Learning: an Empirical Study
- A Modern Take on the Bias-Variance Tradeoff in Neural Networks
2016
Preprints
2026
2025
2024
2022
2021
- Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks
- Gotta Go Fast When Generating Data with Score-Based Models
2020
- Generalizing to unseen domains via distribution matching
- Gradient penalty from a maximum margin perspective