Why we are not building a transformer

We like transformers. It is hard not to admire a machine that turned out to be one idea, repeated, that you can grow almost arbitrarily by adding blocks and tokens. Simplicity that scales is rare and worth respecting.

But the same property that makes a transformer easy to scale makes it wasteful to run. It applies the same dense arithmetic to every token, holds every past token in a cache that grows without bound, and multiplies floating-point numbers through channels that, for most inputs, never change. A brain does none of these things. It does not run every region at once, it does not grow its memory without limit, and it does not spend energy on signals it can predict.

Mind-1B starts from that gap. Rather than make the dense block a little cheaper, we name three specific wastes the transformer pays for by design, and we pay each one down with a separate mechanism borrowed from how brains actually work.

01Uniform compute per token

Every token through a transformer costs the same, whether it was obvious or genuinely hard. The model has no notion of this one was easy. That is comfortable for a scheduler and terrible for an energy budget: most of a real text stream is predictable filler, and the network burns full price on all of it.

The fix is to make prediction error, not token position, decide how much compute a step gets. Mind-1B keeps a running guess of its own next input. The residual, surprise = actual - predicted, costs one subtraction, and it becomes the throttle. Predictable channels coast at almost no cost; a surprising token opens the taps. Compute follows surprise, which is exactly how the cortex allocates its own metabolic budget.

02Memory that grows without bound

The transformer remembers by keeping everything. Its KV cache holds a key and value for every past token, so memory grows linearly with the conversation and the cost of each new token grows with it. It is a filing cabinet that never throws anything out, and eventually the cabinet is the building.

A brain stores memory in its weights, not in an ever-lengthening tape of the past. Mind-1B follows suit: a fast-weight associative store writes experience into the network itself during waking, backed by a bounded ring for recent episodes. Memory is O(1), there is no cache to grow, and what gets kept is decided later, offline, rather than hoarded by default.

03Dense floating-point matmul everywhere

Most of the arithmetic in a transformer is floating-point multiplication, run across channels that are inert for the input at hand. It is precision spent where nothing is happening.

Mind-1B moves almost all of its inference arithmetic off the multiply path. The core is ternary: every weight is one of three values, {-1, 0, +1}, which turns a matrix multiply into integer addition and subtraction over only the channels that are active. No multiply, and the entire 1.02B-parameter core packs to 221 MB at two bits per weight. Small enough to think about running at the edge.

One signal, wired to four things

Here is the part that makes this a research program rather than three separate optimizations. The same surprise signal that throttles compute is the currency for everything else the model does. Ternary regions compete to write into a small shared workspace, and the write only fires, all or none, when surprise crosses a threshold. Regions bind into the winning coalition by locking oscillator phase rather than being chosen by a learned router. Memory is written in proportion to surprise, so the model encodes the moment the stream broke its prediction rather than the predictable stream itself.

Nobody has built that combination. Each primitive is individually proven; the novelty lives in the wiring.

And because a stream of surprise is exhausting to carry, the model sleeps. Wake and sleep are anticorrelated states that do not run at once. Sleep replays the most surprising episodes of the day, downscales the noise, and burns what mattered into the ternary base, where a pruned synapse is simply a weight set back to zero. That last note has its own entry, because the numbers surprised us too.

The rule that keeps this honest

A design this ambitious is easy to oversell, so we tied our own hands. No novel mechanism reaches the 1B run until it beats a proven baseline at 100M first. A model that is 100% novel and does not converge teaches nothing; radical spine, proven legs. Every headline number on this site is reproduced by a small script in the repo, at toy scale on purpose, and the bets that remain open are named on the roadmap rather than hidden.

That is the shape of the thing. The next two notes are the first measured results: what happens when compute is allowed to follow surprise, and what a night of sleep does to a memory that would otherwise be forgotten.