0-bro lab notes https://0-bro.com/blog/ Measured progress on Mind-1B, a ~1B-parameter brain-like non-transformer language model. One note when a milestone gate passes or fails, and what the failure taught. en urbano@urbanodx.com (0-bro) Mon, 06 Jul 2026 08:00:00 GMT 1440 https://0-bro.com/assets/plates/og.webp 0-bro lab notes https://0-bro.com/blog/ Sleep, and the 81% that survived https://0-bro.com/blog/sleep-and-what-survives/ https://0-bro.com/blog/sleep-and-what-survives/ Mon, 06 Jul 2026 08:00:00 GMT Sleep and memory A model that never sleeps forgets its first task by the time it learns its third. A wake and sleep consolidation loop (replay, homeostatic downscale, prune) kept 81% recall over eight cycles, against 34% without it. Gate M2, passed. Teach a network task A, then B, then C, and by the time it knows C it has usually forgotten A. The brain solves this every night. We tried the brain's method.

Catastrophic forgetting is the oldest embarrassment in neural networks: a model learns something, learns a second thing on top of it, and the second lesson quietly overwrites the first. Brains learn in sequence, all day, and keep the important parts. The trick is that they stop and sleep.

Mind-1B sleeps on purpose. A wake and sleep consolidation loop, replay the day's most surprising episodes, downscale the noise, prune the weakest connections back to the native ternary zero, kept 81% recall over eight wake and sleep cycles, against 34% without it. In the prototype the wake and sleep activity patterns settle into anticorrelated attractors (measured correlation -0.74), which is what lets consolidation protect a memory instead of overwriting it.

This is gate M2 on the roadmap, proven in numpy at toy scale. The next rung asks it to hold inside a real model on the two-B300 box.

Read the full note.

]]> Seven percent of the FLOPs, and no error to show for the rest https://0-bro.com/blog/seven-percent-of-the-flops/ https://0-bro.com/blog/seven-percent-of-the-flops/ Sat, 04 Jul 2026 08:00:00 GMT Surprise economy A transformer spends the same compute on a comma as on a proof step. We wired one prediction-error signal to the throttle: on a redundant stream the prototype recomputed 7% of the dense FLOPs per token and produced the same output. Compute should follow surprise. If most of a text stream is predictable, a model that spends full compute on all of it is paying for work it did not need to do. So we stopped paying.

A transformer runs the same dense arithmetic on every token, whether the token was obvious or hard. Mind-1B instead carries a cheap running prediction of its own next input and subtracts it from what actually arrived. That residual, the surprise, costs one subtraction and becomes a throttle: a channel whose input matched the prediction is left alone, a surprised channel is recomputed in full.

On a deliberately redundant stream the prototype recomputed 7% of the dense FLOPs per token, at zero output error. The gated model produced the same answer as the fully dense one; the other 93% was work it correctly declined to do. A companion result: with no learned router at all, regions self-assemble into a coalition by phase-lock, and about 24% are active on an average token, growing with difficulty.

It is a numpy prototype, not the 1B model. The next gate is to reproduce it inside a 100M PyTorch core that stays close to a dense baseline while most regions sit idle.

Read the full note.

]]> Why we are not building a transformer https://0-bro.com/blog/why-not-a-transformer/ https://0-bro.com/blog/why-not-a-transformer/ Thu, 02 Jul 2026 08:00:00 GMT Thesis The transformer is a magnificent, uniform machine, and the uniformity is the tax. It runs every region at once, grows its memory without bound, and multiplies floats it could skip. A brain does none of those things. The case for a different shape. A transformer spends the same compute on a comma as on a proof step. That uniformity is the whole reason it scales so cleanly, and it is also the tax.

The same property that makes a transformer easy to scale makes it wasteful to run: it applies the same dense arithmetic to every token, holds every past token in a cache that grows without bound, and multiplies floating-point numbers through channels that never change. A brain does none of these things.

Mind-1B names three transformer wastes and pays each one down with a separate brain-inspired mechanism: surprise-gated compute (FLOPs follow prediction error, not token count), weights as memory (a fast-weight store plus a bounded ring, so memory is O(1) with no cache to grow), and a ternary core (weights in {-1, 0, +1}, integer add and subtract instead of multiply, the whole 1.02B-parameter core in 221 MB). One surprise signal runs all of it, and the model sleeps to consolidate what it learned.

Read the full note.

]]>