Quark: 128K Context and RL-Only Agentic Post-Training at 165M Parameters for $4,200 of Compute

Abstract

Quark is a language model from Empero with about 165 million parameters, built as a low-cost testbed to validate a full pretraining, long-context and reinforcement-learning stack before scaling up. Under each global attention layer it stacks three local layers, which pair sliding-window attention with short gated convolutions. The main run trains on 500 billion tokens at 1K, 4K and 32K context. A separate 10-billion-token extension, branched shortly before the end of that run, trains at 128K. The report covers one run of each stage, without ablations or seed replication.

The extension beats the final main checkpoint on all nine zero-shot benchmarks we report, reaching 38.28% normalized accuracy on HellaSwag and 55.43% on ARC-Easy, and it has the best average among open models of similar size trained on similar token budgets. On teacher-forced token-check accuracy at 128K, it passes every check in five of eight synthetic long-context task families, and fails quote attribution entirely.

We then post-trained the extension with reinforcement learning alone, with no supervised fine-tuning stage or distillation. In training rollouts on simple terminal tasks across 15 emulated coding-agent harnesses with real sandboxes, the share solved rises from 3% to 65% and transfers to harnesses added late in the run; on an in-house task suite, correct answers rise from 13% to 36% before the run was stopped early for budget with scores still rising. The whole project, from the first data-generation job to the last reinforcement learning step and including a discarded first design, cost about $4,200 of rented compute, and training came back from seven node interruptions without a loss spike.

Compute for this work was partly sponsored by lium.io.

Key findings

About $4,200 for the whole project.About 1,300 H100-hours of rented compute, from the first data-generation job to the last reinforcement learning step, including failed attempts and a discarded first design.
The best average in its group.52.8 across nine zero-shot benchmarks, against open models of 120M to 170M parameters trained on similar token budgets. SmolLM2-135M, trained on 2T tokens, scores higher.
Five of eight long-context families pass at 128K.Retrieval, recall, counting, arithmetic and variable tracking pass every token check at 128K. Quote attribution fails at every length.
Agentic skills from reinforcement learning alone.No supervised fine-tuning stage and no distillation. On simple terminal tasks across 15 emulated harnesses, the share solved in training rollouts rose from 2.6% to 64.5%.

**Figure 1. Three local layers precede each global layer.** Left, the 16-layer stack; right, the wiring of one local and one global layer. Dashed arrows mark the gated reuse of the first layer's values by every later attention.

**Figure 2. Each stage settles at a loss level set by its own mixture.** Training loss as logged, its moving average and the validation-mixture loss for each stage. Each panel has its own axes. One main run and one extension.

**Figure 3. The 128K model scores highest of the token-matched models on six of nine benchmarks.** Zero-shot scores, one panel per benchmark. Pythia-160M is shown in float32. SmolLM2-135M, hollow and set apart, was trained on 2T tokens.

**Figure 4. Retrieval and tracking hold up to 128K; quote attribution fails at every length.** Teacher-forced token-check accuracy (%) at the final checkpoints. Cells at 32K have 24 items; cells at 64K and 128K have eight.

**Figure 5. Twelve of the 15 harnesses finish between 58% and 90% solved.** Share of replies solved per anonymized harness in on-policy training rollouts, first 100 trained steps against the last 100 steps of the run. Harnesses added late in the run are marked.

Citation

@techreport{lehmann2026quark,
  title       = {quark: 128K context and RL-only agentic post-training at 165M parameters for \$4,200 of compute},
  author      = {Lehmann, Leon},
  institution = {Empero},
  type        = {Empero AI Technical Report},
  year        = {2026},
  month       = {November},
  url         = {https://empero.org/research/quark/}
}

Correspondence

Questions and corrections: [email protected]. The model cards on Hugging Face have the weights, the code to run them and sample outputs.