empero

PageGram

Multi-Head N-gram Memory at the Read Cost of a Flat Table

Empero AI Technical ReportOctober 2026Leon Lehmann
Paper (PDF) Code and results
Abstract

Hashed n-gram tables give a language model billions of parameters that cost memory but almost no compute, and the strongest designs hash every n-gram with several independent heads. On a device, such a table lives in flash, and independent heads turn every token into sixteen scattered reads. PageGram keeps the heads but places all heads of an n-gram order inside one page, so a token reads two pages, as with a flat table.

At equal table size the locality costs no quality. In a controlled study PageGram is better than a flat table and than Engram-style multi-head addressing on each of five seeds after 60M training tokens, by 0.067 and 0.142 nats. With twice the training tokens it is level with Engram-style addressing, 0.010 apart, and still below the flat table on average; in a larger model it is level with the flat table within seed noise.

The multi-head layouts differ only in how collisions correlate across heads, and a single parameter, the number of slots per head in a page, controls that correlation; four slots is the most reliable setting. Stored at four bits per value, which costs 0.0004 nats, a 20B-parameter PageGram table needs 0.99 flash reads per token with a 1 GiB page cache, against 9.18 for Engram-style addressing, and serves five to eight times as many lookups per second from an NVMe drive.

Key findings
  • Eight times fewer flash reads.A token reads one page per n-gram order, as with a flat table, where Engram-style multi-head addressing needs sixteen scattered reads. On a 20B-parameter table with a 1 GiB page cache, that is 0.99 reads per token against 9.18.
  • No quality cost, an early lead.At equal size (200M parameters), PageGram is better than a flat table and than Engram-style addressing on all five seeds after 60M training tokens, by 0.067 and 0.142. With twice the tokens it is level with Engram-style addressing and still below the flat table on average.
  • Correlated collisions are the lever.Per head, PageGram's collisions are as frequent as Engram-style addressing's; only their correlation across heads differs. Four slots per head per page gives the best mean, and no seed fails at that setting.
  • Deployable as trained.Snapping the table to 4-bit NF4 costs 0.0004 nats, and training runs within 5% of the speed of a model without a table.
Three ways to lay out an n-gram table. A flat table reads one row per order; Engram-style addressing keeps one table per head and reads eight places; PageGram lets the heads pick slots inside one page and reads one.
Figure 1. Table layouts. Three ways to lay out an n-gram table. A flat table reads one row per order; Engram-style addressing keeps one table per head and reads eight places; PageGram lets the heads pick slots inside one page and reads one.
(a) Held-out loss over 60M tokens, mean of five seeds. (b) Loss against the flat table, paired by seed. (c) Flash reads per token for a 20B-parameter table as the RAM page cache grows.
Figure 2. Quality and read cost. (a) Held-out loss over 60M tokens, mean of five seeds. (b) Loss against the flat table, paired by seed. (c) Flash reads per token for a 20B-parameter table as the RAM page cache grows.
Citation
@techreport{lehmann2026pagegram,
  title       = {PageGram: Multi-Head N-gram Memory at the Read Cost of a Flat Table},
  author      = {Lehmann, Leon},
  institution = {Empero AI Research},
  type        = {Empero AI Technical Report},
  year        = {2026},
  month       = {October},
  url         = {https://github.com/empero-org/pagegram}
}
Correspondence

Questions and corrections: [email protected]. The repository holds the code, the results of every run and the paper.