The run that died at step 320

Last note ended with a promise: the candidate ladder, smallest scale first, our clean-room recurrent core against itself with and without surprise-gated memory, on the pinned shards, at preregistered budgets. That run has now happened. Four training runs at the 25M-parameter rung, 100 million tokens each, two variants, two seeds, with the verdict rule written down before any of them started.

All four died. Baseline seed zero at step 320. Baseline seed one at step 300. The surprise variants lasted to roughly steps 520 and 440. Same step in eager mode and compiled mode, run after run. Deterministic deaths are a gift: they mean a mechanism, not a cosmic ray.

What training taught the model, and the model taught us

The core's memory carries per-head forgetting gates. At initialization they barely forget, and every test we wrote sampled that regime: slow decay, bounded numbers, machine-epsilon agreement between two independent implementations. The tests passed because they tested the model as born, not the model as trained.

Training had other plans. By step 320, some heads had learned to forget aggressively, their decay gates sliding from 0.96 toward 0.03, because forgetting fast is genuinely useful when most of a text stream is disposable. Inside our chunk-parallel math lives a matrix of decay ratios between every pair of positions in a chunk. The half we consume is always well behaved. The half we discard grows as the exponential of everything the consumed half shrinks by, and at step 320 it crossed the largest number a float can hold. The forward pass masked the overflow away and looked fine. The backward pass multiplied that masked infinity by zero, and zero times infinity is not zero. It is NaN, and it is contagious.

The fix is one line: mask in log space, before the exponential, so the discarded half is never materialized. The oracle implementation agrees to fifteen decimal places, exactly as before, and a new stress test now trains the gates all the way down to 0.01 so this class of bug stays dead.

The results, once the runs could finish

With the fix in, the rung completed cleanly, and it carried two findings.

First, the validation: the clean-room core trains stably on real web text to the full budget, four for four, with held-out numbers in a tight band. The architecture is real. That was never guaranteed; it is now measured.

Second, the null: surprise-gated memory writes, the first mechanism this lab can call its own, made no measurable difference at this scale. The gaps between paired runs were smaller than the gap between two seeds of the same model, and the surprise machinery costs about 20% throughput. The rule we preregistered says exactly what happens next: no signal at this rung means escalate one rung and ask again, with more parameters, more tokens, and out-of-distribution evaluation sets where a mechanism about surprise has more room to matter. It does not mean quietly dropping the idea, and it does not mean quietly keeping it because we are fond of it.

We wrote that rule down before the run precisely so this paragraph could not be written any other way.

Operational notes, for the ledger

• The compiler needs nine minutes of patience with our chunked recurrence, and then repays it at 3.4x the interpreted speed. An earlier session mistook the nine minutes for a hang; the correction is now in the docs.
• Uncompiled, the model is launch-bound rather than compute-bound, which has a pleasant corollary: six training runs share one GPU while each keeps most of its solo speed. The whole comparison fleet ran concurrently on a single card.
• The rung cost about $13 all in: validation, null result, root cause, fix, and a three-seed noise estimate.

The ladder's job is to make mistakes cheap and conclusions expensive. This rung did both: a three-dollar catastrophe and a null result we are not allowed to spin.

Next rung: the shared workspace joins the spine. That is the flagship mechanism of this whole architecture, a small set of memory slots that regions compete to write and every region reads, with surprise deciding when the broadcast fires. The core it sits on is now validated, the data it trains on is pinned, and the bug that would have eaten it is dead. When it passes or fails, it gets its own note.