Nathan Godey
Postdoctoral researcher at Cornell Tech
I'm a postdoc in Yoav Artzi's lab at Cornell Tech. Before that, I did my PhD at Inria ALMAnaCH, advised by Benoît Sagot and Éric de la Clergerie. I work at the intersection of NLP and representation learning, with a focus on understanding the representations language models build, and improving them.
News
- New preprint: Copy-Paste Language Models, with an interactive project page.
- New preprint with IBM Research on shared memory in looped Transformers.
- New preprint with Microsoft Research: Almost Free State Prediction Separation.
- New preprint: Matryoshka Language Model Suites.
- New preprint: Co-LMLM, a language model that retrieves knowledge instead of memorizing it.
- New preprint: The State-Prediction Separation Hypothesis.
Selected papers
Lost in Backpropagation: The LM Head is a Gradient Bottleneck
The softmax bottleneck isn't only an expressivity issue. It suppresses 95-99% of the gradient norm during backprop, distorting training feedback for the entire model.
COLM 2026
Copy-Paste Language Models
Preprint, 2026
Matryoshka Language Model Suites
Preprint, 2026
The State-Prediction Separation Hypothesis
NeurIPS 2026
Gaperon: A Peppered English-French Generative Language Model Suite
ACL 2026
Improving Representations for Language Modeling
PhD thesis, 2024 · ATALA best PhD thesis prize 2025
Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck
COLM 2024