Identity on Demand: Multi-head Gated Value Residuals

Abstract

A Transformer layer that needs a token's identity, to copy it or to look something up with it, has to recover it from a residual stream that every earlier layer has written into. Value residual learning hands every layer a fixed share of the first layer's values instead. We make that share a decision.

Multi-head Gated Value Residuals (MGVR) let each layer compute from its own input, for every token and every key-value head, a gate on its values and a mix of the first layer's value heads, so that each head draws as much token identity as it needs, when it needs it. MGVR adds 0.2% of the active parameters, leaves the attention kernel and the size of the key-value cache unchanged, and starts exactly as a value-residual Transformer.

In a controlled pretraining comparison over 1B tokens, MGVR reaches the Transformer's held-out loss with 16 to 22% fewer steps from step 200 onwards, worth 1.15 to 1.23 times the effective compute after its 4.3% step-time overhead, and ends 0.050 nats lower. In a synthetic study of six variants with shared data and initialisation over three seeds, MGVR forms induction heads faster at every learning rate, cutting the time from the attention pattern to accurate copying by 40%, and learns two-hop associative recall 25 to 49% faster than a Transformer and 7 to 18% faster than a static value residual. Ablations trace the composition gains to the token-dependent mixing of value heads.

Key findings

The same loss in 16 to 22% fewer steps.In a controlled 1B-token pretraining comparison, MGVR reaches the Transformer's held-out loss with 16 to 22% fewer steps at every evaluation from step 200 onwards. Charged its 4.3% longer step time, that is 1.15 to 1.23 times the effective compute. It ends 0.050 nats lower (2.812 against 2.862).
Copying works sooner.Induction heads turn into accurate copiers earlier at every learning rate. The time from the induction attention pattern to 90% copy accuracy falls by 40%, from 133 to 80 steps.
Composition is learnt faster.On two-hop associative recall, MGVR learns 25% and 49% faster than the Transformer and 7% and 18% faster than a static value residual, with a four to ten times smaller spread across seeds.
Token-dependent mixing does the work.Mixing the first layer's value heads per token, on its own, matches full MGVR on composition. The method adds 0.2% of the active parameters and drops into any multi-head or grouped-query attention layer.

**Figure 1. One MGVR layer.** MGVR computes a gate on a layer's own values and a per-token mix of the first layer's value heads. The attention kernel and the key-value cache are unchanged.

**Figure 2. Pretraining comparison.** (a) Held-out loss over 1B tokens for a Transformer and MGVR with identical initialisation, data order and hyperparameters. (b) How many fewer steps MGVR needs to reach the Transformer's loss at each evaluation; the dashed line is the break-even point for MGVR's 4.3% step-time cost.

Citation

@techreport{lehmann2026mgvr,
  title       = {Identity on Demand: Multi-head Gated Value Residuals},
  author      = {Lehmann, Leon},
  institution = {Empero AI Research},
  type        = {Empero AI Technical Report},
  year        = {2026},
  month       = {October},
  url         = {https://github.com/empero-org/mgvr}
}

Correspondence

Questions and corrections: [email protected]. The repository holds the code, the results of every run and the paper.