One hour on a B300, and the plan got a fifth cheaper

The plan said 35%. That was a literature-informed guess, and the whole cost table stood on it: the 300B-token training campaign, the ablation budget, the price of being wrong. So before writing another line of model code, we rented a single NVIDIA B300 for an hour and measured.

The first thing real hardware does is disagree with your toolchain. The chip is new enough that the compiler bundled inside our kernel stack had never heard of it: every fused kernel died on launch with an error naming a GPU architecture the compiler did not know existed. The fix is one environment variable pointing at the system's newer compiler, and it took a measurable slice of the hour to find. That is what first hours are for. It is now one line in a setup script, and no future session will pay it again.

Then the measurements. A plain reference transformer, four sizes from 25M to a 1B-shape, trained on synthetic tokens while we watched tokens per second and memory. Small models waste a big chip: 16% of peak at 25M parameters. The curve climbs with scale, and at the 1B shape, compiled, with the batch pushed to the memory ceiling, it lands here:

Feed the measured number back into the cost model and the campaign reprices: the recommended 300B-token run drops from 692 assumed GPU-hours to 538 measured GPU-hours, about $3,200 at 2026 on-demand rental rates, under $2,000 on spot. A full architecture-gate run at the 100M scale, 2B tokens, prices at about five dollars. The unit of science in this lab is now a five-dollar bill.

The dataset is pinned

A second short session built the other asset every experiment needs: data that never moves again. 2,600,191,051 training tokens from a fixed revision of a public educational web corpus, tokenized with our own 32k vocabulary, cut into eleven shards, every shard hashed, every hash recorded in a manifest next to the dataset's exact revision id. Forty-one minutes of work, and the entire experiment ladder now trains from bytes that are content-addressed.

One honest operations lesson from that session: tokenization needs zero GPU, and we ran it on a machine that had one. The idle accelerator watched twenty CPU cores work. Next corpus, if there is one, gets a CPU-only box at a fiftieth of the price.

As a chain proof, a reference 100M transformer then trained for fifteen minutes on the fresh shards: loss fell from 10.40, exactly random for a 32k vocabulary, to 3.58, and its samples read as fluent, confidently wrong English. That is par for 100M parameters and a fraction of the tokens a real run gets. It is not a gate result; it is evidence the shards, tokenizer, and training loop form one working pipeline.

The recurrent core, written clean

The candidate ladder needs a recurrent spine, and the strongest published one comes with a catch: the official implementation carries a non-commercial license. The equations, though, belong to everyone. So the core was written twice from the paper alone, once as a slow, obviously-correct sequential loop, once as the fast chunk-parallel form, and the two implementations agree to machine epsilon on outputs and on gradients. Agreement between two independent derivations is the correctness proof. No licensed source was consulted, which keeps every future for this code open, including the open-source one.

It ships with the first ablation this lab can call its own. The delta rule already computes, for every token, how wrong the memory's prediction of that token was. We wired that error into the write gate: a token that surprises the memory writes itself in strongly, a token the memory saw coming barely leaves a trace. Surprise deciding what deserves to be remembered is the thesis of this whole project, expressed in one small mechanism. Whether it earns its place is now an empirical question with a five-dollar price tag.

What this does not say

• 45.1% belongs to a plain reference transformer. The candidate architectures will differ, likely downward at first. The repriced table is a floor-cost scenario, and it is labeled as one.
• Everything so far is single-GPU. The campaign plans for two; the scaling efficiency between them is unmeasured until a two-GPU session runs the probe, and the code for that probe is already written.
• The 100M run above had an ad-hoc token budget and no held-out evaluation. Indicative, not a baseline. Gate baselines get preregistered budgets and a fixed validation set.

Gate before scale also applies to spending. An assumed efficiency number is a rumor about your own future costs. An hour of measurement turned the rumor into a receipt.

The next gate is the real one: the candidate ladder, smallest scale first, our recurrent core against itself with and without surprise-gated memory, on the pinned shards, at preregistered budgets. When it passes or fails, it gets its own note.