Copy-Paste Language Models

Nathan Godey & Yoav Artzi, Cornell University

Read the paper Get the models Speedrun record BibTeX

Transformers are thought to be good at copying from their context, yet every prediction still goes through the low-rank LM head. CPLMs add a lightweight Copy-Paste head that predicts the next token by pointing at a past position. It costs ~3% compute. In return, models reach the same loss with up to 1.5× fewer training tokens and improve perplexity, downstream, SFT and long-context results at every scale and on every backbone. Sliding-window CPLMs also match full attention with 12× less memory.

Key results

Pretraining1.2–1.5×fewer training tokens to reach the vanilla lossall three 1.7B backbones Generalization0.5–1.5lower perplexity on out-of-domain FinePDFsevery 1.7B backbone Memory12×less memory: SW CPLM matches the Full LMcopy cache included Reasoning+1.2–5.1average points after SFTmath, code & instruction following Long context+4.6–21.3points on RULER long-context retrieval1.7B base models Extrapolation6.4–7.0perplexity at 16k tokens, 4× the training lengthvanilla: 9.6–11.3, or diverges Retrieval+5–85points of needle-in-a-haystack accuracyaveraged over 1k–64k tokens RAG49×smaller cache: HotpotQA documents read through copy keys onlyFull keeps 89–95% of its accuracy Interpretability4 vs 13–15induction heads in the backboneCPLM vs. vanilla, T=128 Speedrun11.3%faster NanoGPT speedrun recordmodded-nanogpt PR #379

What the Copy-Paste head does

Below, the released 1.7B CPLMs read each text, one token at a time. An arrow is drawn whenever the model routes probability for the next token through the Copy-Paste head: it starts at the context token being copied and ends at the predicted token. Width is the copy mass c·α sent through that position. Color is the sink probability: green means the head committed to copying, and red means it mostly deferred back to the LM head. Hover over (or tap) any token for details.

Chatting with the SFT models

The same view on the Tulu-3 SFT checkpoints ({backbone}_sft). Each model wrote its own answer, using greedy decoding on the mixed LM + CP distribution. The answers are shown unedited, then re-read by the same model to get the copy statistics. Answers differ between backbones, so switching the backbone changes the text.

A second head that points instead of decoding

A CPLM keeps the standard LM head and adds a Copy-Paste (CP) head next to it. The CP head scores every earlier position with a query and key projection of the final hidden states, as in self-attention. It then turns those scores into a distribution over tokens by summing positions that hold the same token.

The LM head decides when to copy, through a dummy <copy> token in its vocabulary (gate ct). A learnable <sink> key lets the CP head hand control back to the LM head (st) when nothing in context is worth copying.

Training uses plain next-token cross-entropy, with no auxiliary loss and no copy supervision. The CP head adds two D×D matrices, which is 0.5% of the parameters and 3% of the compute at 1.7B.

Why it helps. The LM head squeezes a V-dimensional logit gradient through a D-dimensional bottleneck and loses over 99% of it. The CP head's output space is the t context positions instead of the vocabulary, which opens a second, much wider gradient path into the backbone.
… ?
LM head
Copy-Paste head
Prediction
from the LM headcopiedsink
One real prediction of the 1.7B Full CPLM, from the Lovelace text above. The LM head alone prefers engine, but it also emits <copy> with probability . The CP head points back at the two earlier mentions of Analytical, and the mixture predicts the right word.

Better at every scale, on every backbone

We pretrain models from 135M to 1.7B parameters on up to 200B FineWeb-Edu tokens, with three backbones: full attention, sliding window (w=256), and a 3:1 hybrid of the two. Both arms share the SmolLM2 recipe and every hyperparameter.

CPLMVanilla LMHover any chart for exact values.

Faster training

At 1.7B, CPLMs reach the final loss of their vanilla counterparts with 1.2× fewer tokens on Full and Hybrid backbones, and 1.5× fewer with sliding windows. At 360M it is 1.25–1.5×, and at 135M 1.5–1.8×.

Validation loss curves at 135MValidation loss curves at 1.7B

FineWeb-Edu validation loss over training, before the learning-rate cooldown.

More out of every byte of memory

A sliding-window CPLM matches the full-attention LM's perplexity while caching 12× less per token, including the copy keys. At every window size, CPLMs push the memory/perplexity frontier down.

FinePDFs perplexity vs memory for 1.7B models

1.7B models. Arrows go from each vanilla model to its CPLM.

Perplexity vs memory across window sizes at 135M

135M models across sliding-window sizes from 32 to 1,024.

The standout downstream results

Gains show up on every benchmark family and every 1.7B backbone. They are largest where copying matters: long-context retrieval, chain-of-thought math after fine-tuning, and in-context examples. Each card shows the three backbones.

All benchmark numbers
Zero-shot, 1.7B base models
After SFT on Tulu-3
Long-context benchmarks

Long context, beyond the training length

On 16k-token documents, 4× the pretraining context, CPLM perplexity keeps falling while every vanilla model degrades. The CP head is used more as the context grows. The SW CPLM copies from more than 2,000 tokens back, far beyond its 256-token window.

Rolling perplexityCopy gate by positionSink by position

Averages over 5,000 FinePDFs documents. Full and Hybrid models use YaRN (×4).

Needle in a haystack, 1k to 64k tokens
Needle-in-a-haystack heatmaps

Retrieval accuracy (%) of a 7-digit number by needle depth and context length, up to 16× the training context. With a window of only 256 tokens, the SW CPLM finds the needle almost everywhere.

Retrieval through the copy cache alone

On HotpotQA with up to 500 distractor documents, each document can be encoded once and read only through its copy keys. CPLMs keep 90–97% of their accuracy that way, and match or beat models that keep every document in their KV cache, with a 50× smaller cache.

Each panel is the model drawn bottom-up: tokens, then layers 1 to 24 with their KV cache, then the copy keys. Grey is kept, hatched is thrown away, green is the copy cache. Pick a backbone and a number of distractors; exact match and memory are measured HotpotQA numbers.

HotpotQA docs visibleHotpotQA docs droppedMemory per document

Exact match after fine-tuning on HotpotQA (distractor setting). In the right panel, the vanilla control reads documents through its last-layer keys and values, which take twice the memory of the CP head cache.

Why it works: a wider gradient channel

The LM head discards over 99% of the logit gradient. The CP query path keeps up to 87% of it at early positions. On a trivial repetition task, vanilla models fail once the vocabulary is large, while CPLMs solve it at every size. Blocking gradients through the CP head removes most of the gains. Byte-level models, which have no bottleneck, gain nothing.

The backward pass through each head. Ribbon width is gradient norm: what does not make it through is lost to the head's nullspace. Slide along the sequence to see the CP head's share shrink at later positions, where it has more keys.

Gradient lost per pathway
SpamLang accuracy

Left: 135M Full CPLM on 2,048-token FineWeb-Edu sequences; random is a random matrix shaped like the CP head output. Right: 135M models with a fixed hidden size of 576.

Better induction with fewer heads

CPLMs copy repeated random sequences almost perfectly at every length, with far fewer induction heads in their backbone: 4 at T=128, against 13–15 for vanilla models. The CP head takes over copying work that vanilla models spread across many attention heads.

Induction NLLInduction head countInduction score map

A NanoGPT speedrun record

Adding the CP head to modded-nanogpt improved the previous speedrun record by 11.3% (PR #379). That is the largest architecture-centric speed-up, and the second largest overall, since December 2024.

Use the models

All 1.7B checkpoints are on the Hugging Face Hub (collection). Each backbone (full, hybrid, sw) lives on its own branch, alongside intermediate and Tulu-3 SFT checkpoints ({backbone}_sft).

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nthngdy/cplm-1.7B"
tok = AutoTokenizer.from_pretrained(repo, revision="full")
model = AutoModelForCausalLM.from_pretrained(repo, revision="full", trust_remote_code=True)

# "natural" (default), "no_copy" (LM head only) or "force_copy" (CP head only)
model.config.inference_mode = "natural"

Citation

@article{godey2026copypaste,
  title   = {Copy-Paste Language Models},
  author  = {Godey, Nathan and Artzi, Yoav},
  journal = {arXiv preprint arXiv:2610.19247},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.19247}
}