empero
Research

Research

Technical reports, methods and evaluation code. Each report ships with the code and the results of every run, under Apache-2.0.

Technical reports

Figure 1. Table layouts.
Technical reportOctober 2026Leon Lehmann

PageGram

Multi-Head N-gram Memory at the Read Cost of a Flat Table

Hashed n-gram tables add billions of parameters for little compute, but their heads scatter reads across flash. PageGram places every head of an n-gram in one page: two reads per token instead of sixteen, at equal or better loss.

Summary Paper (PDF) Code
Figure 1. One MGVR layer.
Technical reportOctober 2026Leon Lehmann

Identity on Demand

Multi-head Gated Value Residuals

Each attention head decides, per token, how much of the first layer's values to draw on. In a controlled 1B-token comparison, MGVR reaches the same loss in 16 to 22% fewer steps for 0.2% more parameters.

Summary Paper (PDF) Code
Methods

Final-Token Preference Optimization

Method, July 2026

Finds the position where a failure starts and trains the model to prefer a coherent alternative there. About 2,000 automatically mined preference pairs took repetition loops in Qwythos-9B from 6.7% to 0.0%.

Code and evaluations

rlt-evaluation

Evaluation code, September 2026

Training and benchmark code for a controlled evaluation of the Recurrent Looped Transformer against matched Transformer baselines: held-out loss, throughput and memory.

nanoAttnRes

Code, March 2026

A nanochat fork with an implementation of Attention Residuals.

Correspondence

Questions about a report, or a result you could not reproduce: [email protected]. Corrections are welcome.