# Mind-1B: A Bilateral, Ternary Cognitive Engine with Global Workspace and Sleep Consolidation Technical Architecture and Specification Whitepaper, Revision 6 UrbanoDX R&D. Design draft of 2026-07-11, evidence clarification 2026-09-05. > **Reading contract:** the configuration and later milestone tables are design > proposals, not a built-model inventory or an approved GPU schedule. Current > evidence is in [README.md](README.md); execution and budget are owned by > [ops/NEXT_STEPS.md](ops/NEXT_STEPS.md). No model execution or dataset downloads > are permitted on the owner's PC. The abstract below supersedes stronger status > or novelty language in the historical revision notes. > **Historical revision scope:** Revision 6 proposed precision-weighted workspace > slots and event-boundary episodic segmentation. Revision 5 expanded the cognitive > target. Revisions 3/4 introduced bilateral streams and a recurrent core with a > separately tested attention-hybrid control. Earlier revisions explored ternary > inference and wake/sleep schedules. These revisions describe planned components; > their presence in the specification does not imply implementation or validation. > [RESEARCH.md](RESEARCH.md) owns source assessments, [INNOVATION.md](INNOVATION.md) > owns the contribution thesis, and `calc/` and `proto/` contain historical > calculators and constructed-task demonstrations. --- ## Abstract Mind-1B is a proposed roughly 1B-parameter recurrent language-model architecture. Its measured starting point is a scalar-GDN C0 model with about 43.9M total parameters trained for 100M tokens per S2 run. Faithful GDN2 and a single-stream workspace exist as separate implementations without demonstrated CUDA quality benefits; the bilateral C2 model, useful episodic memory and sleep consolidation are not built. The current model path uses floating-point matrix multiplication. Ternary and efficient deployment paths remain later proposals. The central contribution hypothesis is that a bounded shared workspace can improve useful communication and task-state retention in a small recurrent model. The long-term design separates a fast precision stream from a slower context stream, then studies adaptive computation and offline consolidation in later ablations. These are proposed inductive biases, not established biological roles or measured capabilities. GDN2, shared workspaces, adaptive computation and continual-memory systems have prior research; their combination is not evidence that this project is first or that it will outperform similarly sized models. The intended first contribution is a reproducible mechanism-level improvement: a workspace variant outperforms a matched recurrent core and a simpler learned communication control, within frozen language-quality and resource limits, with causal interventions supporting useful information transfer. The protocol and unimplemented adapters are specified in the [workspace contribution gate](ops/EVALUATION.md#workspace-contribution-gate). Bilateral streams must subsequently earn their cost against the single-stream winner. Sleep must improve acquisition and retention over ordinary replay at matched replay tokens, update counts and memory allowance. The full target also includes episodic, semantic, procedural and metacognitive memory, predictive control and future action interfaces. Their proposed contracts live in [COGNITIVE_ARCHITECTURE.md](COGNITIVE_ARCHITECTURE.md). No same-size competitive advantage, general assistant quality or edge-device efficiency is demonstrated. Published component results and historical toy demonstrations are separate evidence classes, as recorded in [RESEARCH.md](RESEARCH.md) and [INNOVATION.md](INNOVATION.md). ## 1. Motivation and failure risks For conventional full self-attention, KV-cache storage grows linearly with context length, pairwise prefill attention work grows quadratically, and each decoded token attends over the cached context. Recurrent sequence mixing offers fixed-size state with respect to context length, but compressing history can lose exact associations. A recurrent design must demonstrate useful recall at its chosen state size; constant storage is not unlimited memory. The project targets three separately measured properties: 1. **Useful communication under a resource limit.** A small shared workspace may preserve information that ordinary recurrent state loses. It can also become a bottleneck or consume parameters better spent on the core. 2. **Effort allocated to the task.** Learned routing or adaptive depth may reduce work at a fixed quality level. Controller overhead and routing collapse may erase the benefit. Toy operation counts do not establish real-text sparsity or hardware speedup. 3. **Retention through controlled updates.** Bounded episodic storage and later replay may improve adaptation. Selective writes can retain noise, and updates can damage old knowledge. The historical surprise-beta null at S2 cautions against treating surprise as an automatic improvement. Ternary QAT and specialized kernels are possible later efficiency experiments. No compression ratio establishes inference speed, training savings or a supported phone/CPU deployment. Measure the complete path on the intended hardware with all state, cache, retrieval and iteration costs included. ## 1b. Proposed regions and communication The bilateral target has a token-synchronous precision stream and a slower context stream communicating through bounded workspace slots. Those names state intended computational roles. Specialization must be observed and tested; it cannot be inferred from labels or resemblance to a brain diagram. The current C1 prototype is single-stream and uses the legacy scalar-GDN streaming interface. All layers execute before their summaries compete to write, and workspace reads/writes use slot attention. Top-k writer selection therefore does not yet skip layer computation, and the full C1 model is not wholly attention-free. A GDN2 workspace port needs its own causal and gradient checks. The proposed cognitive target separates uncertainty, novelty and outcome/value rather than letting one surprise scalar stand in for every control signal. Inputs available before each decision must be explicit: a current token's target error cannot decide computation that preceded observing it. The historical `proto/` examples demonstrate constructed-task behavior only. They do not prove language-level specialization, consolidation, compute savings or superiority to a Transformer. See [BRAIN.md](BRAIN.md) for the earlier biological motivation and the current cognitive contracts for the proposed implementation boundaries. --- ## 2. Global configuration | Attribute | Value | Note | |---|---|---| | Total parameters | ~1.04B planning envelope (`calc/params.py`) | 18 bilateral GDN2 region blocks; exact count waits for M3a implementation | | M3a precision | bf16 | correctness baseline before quantization | | M3b core precision | Ternary {-1, 0, +1} | absmean QAT with fp latent weights, only after M3a passes | | Activation path | bf16 at M3a; int8 candidate at M3b | do not quantize state, routing, and memory at once | | Hidden dim (d_model) | 2048 | | | Core blocks (L) | 18 recurrent region blocks | total across both streams, not per hemisphere | | FFN width | 5120 | ternary SwiGLU | | Attention | None in M3 default | 3:1 GDN2 + gated-attention hybrid is a matched control only | | Holographic VSA dim (D) | 10,240, M5-gated | gist/episodic index, NOT a KV replacement | | Hippocampal cache | up to 65,536 tokens, M5-gated | bounded episodic buffer, not M3 context state | | Vocabulary | 32,000 | | | Bilateral allocation | One total ~1.04B planning budget, split at M3 | Not two 1B models; shared embedding, workspace, and head | | Workspace | 2-8 shared slots | Competitive writes; broadcast to both hemispheres | | Context-stream clock | Initial stride 4, ablated at M3 | Right stream consumes pooled/gist input; no fixed biological claim | `python3 calc/params.py` provides a **planning envelope**, not an implementation-exact count: currently 1.035B total, 969M non-embedding, and a 242 MB packed core. The clean-room GDN2 and workspace candidates must be measured and the provisional projection budget replaced before any hardware claim is made. Mind-1B alternates two regimes: **Waking Mode** (real-time tokens and fast-weight capture) and **Sleep Mode** (offline consolidation, cleanup, and pruning). Sleep never modifies persistent base weights on a request's latency path. --- ## 3. Architecture ### 3.1 Token ingress and holographic encoding **M3 scope:** token embedding is the only ingress. VSA is deliberately absent from the first trainable model; it is an M5-gated gist/index experiment after the recurrent workspace and its memory lifecycle are measured. It must not obscure a failure in the core with a second, low-capacity memory mechanism. Token indices map through an embedding table `[V, d_model]`. In parallel, a Vector Symbolic Architecture (VSA) block binds tokens into a fixed-size hypervector using element-wise product for binding and a cyclic permutation Pi for position: X_HDC = sum_i Pi^(N-i) ( v_i ), v_i in {-1, +1}^10240 This is a genuine hyperdimensional-computing construction (survey arXiv:2111.06077). Its honest limit is capacity. Superposition SNR is sqrt(D/M) for M bundled items, giving M_max ~= D / (4[ln K - ln 2*eps]). For D=10,240 that is roughly **195 cleanly recoverable items**, confirmed by our own `calc/vsa_capacity.py` (>=99% recall to ~256 items, collapse past ~512). **Design consequence (corrects the original draft).** The VSA trace is a compressed GIST / episodic index of hundreds of slots, not a replacement for a 64k-token KV cache. The 65,536-token hippocampal ring buffer remains the real working-context store; the VSA vector is a fast associative pointer into it. ### 3.2 Dimensionality gateway This gateway is specified only for the optional M5 VSA path. It is not present in M3a, M3b, or the attention-free versus hybrid comparison. The 10,240-dim hypervector is segmented into five 2,048-dim "cortical columns" and folded into a single 2,048-dim activation front by a fixed XOR mask plus a deterministic stride permutation. This is pure routing logic (zero FLOPs), used as a cheap fixed projection. Note it is lossy, not "lossless topological downsampling" as the original draft claimed; a permutation-XOR fold discards information, which is acceptable for a gist pathway feeding a learned core. ### 3.3 Bilateral streams and shared workspace The two streams receive the same token embedding but are intentionally not copies: - **Left precision stream.** Runs at every token. Its regions favour local, ordered computation: syntax, code-like spans, arithmetic, and rapid correction. It uses the high-frequency recurrent state and receives the adaptive CfC settling budget for difficult tokens. - **Right context stream.** Runs on a pooled/context update clock (initially every four tokens, with a learned carry state between updates). Its regions favour semantic context, discourse, and long-timescale memory retrieval. If VSA passes its M5 gate, its gist enters here first; it is not a second token-by-token Transformer. - **Shared workspace (the only cross-stream bandwidth).** At each workspace pulse, active regions from both streams propose writes to 2-8 slots. Surprise ignition decides whether a write occurs; lateral competition selects the winning proposal. The resulting slots are broadcast back to both streams before the next pulse. There is no final-only fusion layer. The streams share the embedding, workspace, a small always-on hub expert, and the language-model head. Their remaining capacity is partitioned within the single ~1.04B planning budget. M3 must compare this design with a parameter- and active-compute-matched single-stream workspace model; lateralization ships only if that ablation wins. ### 3.3b Precision Workspace (M4a-gated) M3 uses ordinary deterministic workspace slots. Only after that baseline trains and passes its causal audit do we test a **Precision Workspace**, inspired by Kalman Linear Attention's scan-parallel Bayesian filtering and explicit belief uncertainty (arXiv:2602.10743, ICML 2026). It is an ablation of the slot-update rule, not a second sequence mixer added to GDN2. Each slot j carries a mean belief `m_j` and diagonal variance `p_j`. A proposal `z_j` from either hemisphere includes learned observation variance `r_j`; a small predictor supplies process variance `q_j`. The bounded update is: e_j = z_j - m_j K_j = p_j / (p_j + r_j + eps) m_j' = m_j + K_j (.) e_j p_j' = (1 - K_j) (.) p_j + q_j surprise = mean(e_j^2 / (p_j + r_j + eps)) `K_j` is an element-wise Kalman-style gain. Reliable evidence can revise an uncertain slot; a noisy proposal is mostly ignored; a confident slot resists overwrite. The normalized innovation supplies the causal surprise signal for the next compute/write decision. Fast-stream proposals may learn larger process variance than slow contextual proposals, giving the bilateral clocks distinct adaptation rates without hard-coding which stream is always correct. This is only an approximate diagonal Bayesian filter. To prevent the model from gaming uncertainty, log-variances are bounded and trained with a proper predictive likelihood plus held-out calibration tests. The Precision Workspace ships only if, against the same deterministic workspace at equal parameters and active compute, it improves state tracking and uncertainty calibration while improving NLL or reducing writes/compute at matched NLL. Required metrics include Brier score/ECE, normalized-innovation calibration, overwrite rate, multi-key retrieval, workspace occupancy, and downstream language loss. ### 3.3c Full cognitive target: separate systems, separate gates Mind-1B deliberately has more than a cortical core, but not more than one trainable core at a time. The complete target adds the following systems around the bilateral workspace: | System | Functionally inspired role | Earliest milestone | Non-negotiable gate | |---|---|---|---| | Cerebellar forward model + Precision Workspace | Predicts the next recurrent/workspace state; tracks slot belief uncertainty and calibrates corrective compute | M4a | Improves state tracking and calibration plus quality/compute or calibrated early exit | | Prefrontal controller | Maintains task state, resolves conflict, requests planning/retrieval, halts | M4b | Beats fixed-depth control on planning at equal compute | | Hippocampal formation | Causal event segmentation, context index, sparse pattern separation, associative episode completion, recall mismatch | M5a | Improves boundary-aware retrieval, partial-cue recall, and retention without interference | | Default mode | NREM replay and later REM integration over candidate memory | M5b | Beats replay-only and no-sleep controls | | Neuromodulatory controller | Separates uncertainty, novelty/encoding, and outcome/value signals | M4/M5 | Each scalar improves its assigned metric alone | | Attention schema | Reports and controls resource allocation, uncertainty, and action intent | M6/M7 | Improves calibration/allocation; no awareness claim | | Basal-ganglia action gate | Go/no-go selection among answer, retrieve, act, defer, or halt | M7 | Improves verified agent behavior over direct action output | | Multimodal cortices | Typed perception and action interfaces | M7 | Adds grounded capability without text-core regression | These are functional analogies, not a claim that Mind-1B instantiates a human brain or has subjective experience. The detailed interfaces, memory hierarchy, and ablation criteria are in `COGNITIVE_ARCHITECTURE.md`. ### 3.4 Recurrent MatMul-free core (18 layers total) Each core block is attention-free in the primary M3 architecture: - **RMSNorm / SubLN-compatible normalization.** M3a uses the stable bf16 form. M3b tests the integer-safe SubLN path only after the recurrent baseline passes. - **Gated DeltaNet-2 mixer.** The mixer maintains a bounded recurrent state, using channel-wise decay and separate erase and write gates for targeted memory edits. It is a more specific successor to the MLGRU/Gated-DeltaNet sketch in Revision 3, not a collection of Mamba, RWKV, and DeltaNet components. Both hemispheres use this mixer; their update frequency and inputs differ, not the underlying recurrence. - **Fine-grained sparse regions plus shared hub.** A learned top-k router selects bilateral specialists, while an always-on shared expert carries common computation. This follows the shared-expert rationale, but the workspace, not a softmax-attention residual stream, is the cross-region broadcast path. - **SwiGLU region transform.** M3a uses bf16 SwiGLU. M3b replaces eligible projections with BitLinear/QAT while leaving recurrent state and routing at the precision required for stability. The published evidence for a 3:1 Gated DeltaNet plus attention hybrid is strong, so it is the required **control**: three GDN2 blocks followed by one local gated- attention block, with the same parameter and active-compute budget. It is not the default or a fallback silently enabled in the flagship. The attention-free model ships only if it matches the hybrid on quality and wins on bounded-memory efficiency. Softmax-free spiking attention is retired from the critical path. ### 3.5 Workspace observability and adaptive reasoning (M4-gated) M3 does not force the workspace to be verbalizable. Once a checkpoint is useful, we measure a **workspace Jacobian lens**: the corpus-averaged first-order effect of each workspace slot and hemisphere state on future output logits. Sparse decomposition over token and phrase directions defines a model-specific workspace/J-space analogue. This is an audit, not an inference-time operation and not a loss to optimize before we know it is valid. The audit must test verbal report, directed modulation, flexible reasoning, cross-hemisphere broadcast, selectivity, and causal ablation. Automatic language processing is expected to remain largely outside this space. The test protects us from calling a set of slots a global workspace merely because it exists. Only after that M3 baseline passes do we add the Closed-form Continuous-time (CfC) recurrence as an adaptive loop: h(t) = A (.) exp(-[f(X) + g(h(t-1))] dt) (.) h(t-1) + B (.) sigma(h(t-1)) The loop iterates until a learned halt head fires. Prior residual and uncertainty choose its budget; current-token error updates the next decision. Easy tokens exit in one step; hard tokens iterate. **Honest framing.** We do NOT train this by predictive coding: no PC method works at even 100M scale (best is muPC on CIFAR, arXiv:2505.13124). We train the loop by backprop-through-recursion, the method Ouro and EBT actually use. The "energy settling" language is a design metaphor; the gradient is standard. The ARC Prize dissection of HRM is the guide here: the refinement loop carries the gains, not the biological branding, so we sell the loop. ### 3.6 Output and multi-token prediction M3a uses a conventional tied next-token head so model failures are readable. M3b tests the ternary/PopCount output boundary. Pretraining also includes a gated four-head multi-token-prediction auxiliary loss; it is discarded at ordinary decoding unless speculative decoding proves correct and faster. This is a training/throughput improvement, not a claim that the model emits four trusted tokens without verification. --- ## 4. Waking-mode plasticity **M5-gated, absent from M3/M4.** Gated DeltaNet-2 already supplies a bounded, learned fast state. We will not add raw Hebbian tags while validating that state, the bilateral workspace, QAT, and routing. Plastic tags are introduced only after the no-tag continual-learning baseline is measured; a Titans-style gradient memory is the named alternative, not an extra system to combine by default. Every ternary base weight W_ij carries a low-precision plastic tag W_plastic that captures session experience online, without backprop: W_plastic <- W_plastic + eta_wake * (S_out (.) S_in) # Hebbian write W_plastic <- W_plastic - beta * (S_out (.) S_in) * W_plastic # delta correction The second term is the delta-rule correction from Fast Weight Programmers (arXiv:2102.11174): pure additive Hebbian writes saturate and have low capacity, so we correct existing key-value associations rather than only accumulating. **This is a research bet:** this project has no language-model evidence for Hebbian tags at the proposed scale. Compare the specific update mechanism with prior fast-weight systems before making a novelty claim. The event-driven form is a candidate whose accuracy, retention and full update cost require controls. If M5 plasticity is enabled, two controls are tested: a recurrent-state norm and gate-saturation controller, and a slow workspace-gain controller driven by write entropy and uncertainty. These replace the old spiking-specific membrane/GIF rules in the main path. Both are ablations; neither is a biological claim or a precondition for the core model. ### 4.1 Neuromodulatory control: split the overloaded surprise signal The original surprise economy remains the central compute signal, but it does not stand in for every brain chemical. M4/M5 separates three learned, bounded heads: | Head | Inputs | May control | Cannot do | |---|---|---|---| | `u_t`: uncertainty / unexpectedness | Prior residual, predictive entropy, retrieval mismatch | Region budget, workspace ignition, corrective compute | Read the current target before it exists | | `n_t`: novelty / encoding value | Episode distinctness, CA1-like recall mismatch, recency | Candidate episodic writes, pattern-separation sparsity, replay priority | Modify semantic weights online | | `v_t`: outcome/value | Verifier or tool result, task reward, policy advantage | Action gate and offline policy credit | Change language probabilities without an explicit objective | Every controller head is ablated independently. The model does not get an “emotion” subsystem by naming a scalar after one; a controller is retained only when it improves a measurable allocation, retention, or action decision. ### 4.2 Event-boundary episodic segmentation (M5a-gated) The hippocampal path does not write one memory every fixed number of tokens. Before any episodic write, a small causal boundary head estimates whether the current semantic event has ended. The design follows the useful engineering lesson from event-segmented hierarchical memory (ES-Mem, arXiv:2601.07582): memory units should preserve coherent events and use their boundaries as retrieval anchors rather than fragmenting experience into arbitrary chunks. At token t, the boundary head may read only information already available: - normalized Precision-Workspace innovation from earlier/current causal state; - displacement between the previous and current slow-context states; - CA1-style retrieval mismatch, novelty, and elapsed tokens since the last accepted boundary; and - explicit interaction boundaries such as a completed user/tool turn. It emits `b_t = sigmoid(f_boundary(...))`. A boundary above threshold closes the completed event and writes a bounded record containing a temporal-context key, compressed event summary, terminal workspace belief and uncertainty, source span, and boundary confidence. Minimum and maximum event lengths prevent one-token fragmentation and unbounded episodes. The detector may close the event only after observing its final token; its summary becomes available to subsequent tokens, so it cannot leak future context. The event hierarchy is **episode first, gist later**: raw/compact episode records remain the auditable source; sleep may later consolidate several related events into a slower semantic summary. Event segmentation never writes base weights on the serving path. Required controls are fixed 256/512-token chunks, turn-only boundaries, a flat semantic-retrieval memory with the same byte budget, and oracle boundaries only as a diagnostic ceiling. The learned detector ships only if it improves boundary-aware retrieval and partial-cue completion at equal memory and latency, without increasing false recall or held-out language loss. Metrics include boundary precision/recall, over/under-segmentation, bytes per retained token, retrieval recall@k, near-duplicate false recall, and continual-task retention. --- ## 5. Autonomous sleep consolidation and cleanup (centerpiece) Waking is net synaptic potentiation: fast tags grow for both real signal and random coincidences until memory saturates and signal-to-noise collapses. This is exactly the failure the **Synaptic Homeostasis Hypothesis** (Tononi and Cirelli) says sleep evolved to fix, and it is the failure our own simulation reproduces: with no cleanup, continual-learning recall falls from ~50% to ~34% over 8 tasks as tags saturate (`proto/sleep_cleanup.py`). Mind-1B runs an offline sleep phase that decides, per synapse, **what to keep and what to remove**, using three mechanisms the brain actually uses. ### 5.1 The keep-or-remove rule For each plastic tag i, with a nightly replay reactivation count r_i: utility_i = |tag_i| * (1 + lambda * r_i) # magnitude x salience tag_i <- gamma * tag_i # (1) SHY global downscale if utility_i >= theta: # (2) consolidate / KEEP W_base_i <- sign(tag_i) # burn ternary sign in else: # (3) prune / REMOVE tag_i <- 0 # synaptic elimination Three biological mechanisms, in order: 1. **Synaptic Homeostasis (SHY): global downscale.** Every tag is multiplied by gamma < 1 (default 0.85). Strong, replayed traces survive; weak spurious ones fall below threshold and disappear. This preserves the pattern while resetting the noise floor and recovering capacity, exactly the sleep function SHY describes. 2. **Replay-gated consolidation (hippocampal sharp-wave ripples to cortex).** Recent episodes from the hippocampal cache are replayed forward and backward through the core (Sleep Replay Consolidation, Tadros et al 2022). Synapses that reactivate often earn high salience r_i and are tagged for capture (synaptic tag-and-capture). Their ternary sign is burned into the permanent base W. This is the KEEP signal. 3. **Synaptic pruning (use-it-or-lose-it; microglial elimination).** Everything that survives neither magnitude nor replay is set to exactly 0 and removed. **Because weights are ternary, 0 is a native state, so removing a memory costs nothing and directly recovers capacity and inference sparsity.** This is the REMOVE signal and Mind-1B's cleanest original angle: pruning == ternarizing toward 0. ### 5.2 Two-stage schedule (NREM then REM) - **Stage A, NREM (downscale + consolidate + prune):** the rule above, plus stochastic noise injection on surviving tags, `W_plastic(t_sleep) = gamma * W_plastic(t-1) + eta * N(0, sigma^2)` with gamma = 0.85. Traces that resist the noise and align with stable attractors are burned in; conversational noise is pruned. - **Stage B, REM (dreaming + integration):** an offline self-play loop synthesizes predictive scenarios over internal states and applies a contrastive repulsion step that sharpens ternary decision boundaries, integrating new memories into existing schema. A holographic de-noising pass orthogonalizes the accumulated VSA trace for the next waking cycle. ### 5.3 Operational sleep schedules Sleep is a scheduled, observable state transition, not background mutation while a user waits for a token. The starting schedules below are configuration defaults to be ablated at M5. | Context | Wake interval | Sleep trigger | Sleep work | Safety gate | |---|---|---|---|---| | M5/M6 pretraining | 25M training tokens | End of each chunk | Replay a stratified episode buffer; NREM keep/prune; no REM synthesis initially | Compare loss and replay retention to a no-sleep run before accepting the checkpoint | | Continual-learning evaluation | One task/domain block | After each block | NREM plus balanced replay from prior tasks | Commit only if retained-task score does not regress beyond the preset bound | | Interactive session | Active request stream | At least 5 min idle **and** >=2,048 new observed tokens since the last cycle | Candidate-memory NREM; bounded to the session episode buffer | Copy-on-write candidate; never alter base weights in place | | Service maintenance | Accumulated approved session candidates | Low-traffic maintenance window, initially once per 24 h | NREM consolidation then REM synthesis/contrastive integration | Evaluate held-out regression suite; atomically publish or discard the candidate checkpoint | The right context stream may perform cheap state compression at its normal multi-token clock; that is **not** sleep. Sleep is the slower replay/consolidate/ prune operation and is mutually exclusive with interactive wake updates. The left stream supplies detailed episode traces; the right stream supplies gist and cross-episode context during replay. Both may propose tags, but workspace competition and the same keep-or-remove rule decide the final update. ### 5.4 Why this needs replay (the honest failure mode) Every prior attempt at offline self-modification hit catastrophic forgetting: SEAL's self-edits degrade earlier tasks monotonically (arXiv:2506.10943). Sleep cleanup mitigates this two ways: (a) generative replay during Stage A rehearses old tasks so consolidation does not overwrite them (Deep Generative Replay, Shin et al 2017), and (b) preferring to write into an additive product-key memory layer (Memory Layers at Scale, arXiv:2412.09764) rather than the dense backbone contains interference. Our `proto/sleep_cleanup.py` demonstrates the payoff: with cleanup, recall on all past tasks holds at ~81-89% across 8 wake/sleep cycles while the base stays sparse; without it, recall collapses to ~34%. **Status:** no production-scale LM has been shown to run a real wake/sleep cycle that burns experience into its own weights. The pieces exist separately (offline recurrence to 2B in arXiv:2605.26099; context-level sleep in production at Letta, arXiv:2504.13171). Mind-1B assembles them; that assembly is the research risk. --- ## 6. Training pipeline Mind-1B starts M3a on **one NVIDIA B300 (Blackwell Ultra)**. The four-stage pipeline below uses the later **2x-B300** case only as an M6 planning model; it is not current launch authorisation. The hardware bring-up, licence/data gates, and C0-C3 iteration matrix are in `ops/B300_HANDOFF.md`. Numbers below come from `calc/train_cost.py`. 1. **Stage 0 / M3a, bf16 architecture proof.** Train a 100M attention-free bilateral GDN2 workspace model, its matched 3:1 GDN2-plus-attention control, and a matched single-stream workspace model. No QAT, VSA, plastic tags, phase routing, or CfC loop. Train the workspace audit; do not train it as a loss. 2. **Stage 1 / M3b, QAT transition.** Apply continual-QAT to the winning M3a architecture only. Keep bf16 master weights, apply absmean ternarization in the forward pass, backprop with STE, and use BitNet's two-stage LR/WD schedule. The recurrence state, router, and workspace remain at the precision required for stability. 3. **Stage 2 / M6, 200-300B-token pretraining.** Scale only the architecture that passes M3-M5. This remains an architecture-proof budget, not a ship-competitive one (SmolLM2 used 11T; Qwen3-1.7B used 36T). Multi-token prediction is an auxiliary objective here, measured against single-token training. 4. **Stage 3, post-training and distillation.** Distill from an 8B teacher with logits and selected state targets, not attention maps, which the default model does not have. Then run counterfactual reflection SFT: append an ethical, uncertainty, or task-principle reflection and compute loss only on that counterfactual turn. It is removed at evaluation and must improve behavior through a measured workspace change, not hidden chain-of-thought at inference. Non-differentiable spike steps use arctan surrogate gradients on the backward pass, after SpikeGPT. During Stage 2, the 25M-token sleep cadence in Section 5.3 is enabled only after the no-tag M5 baseline trains stably. Start with NREM replay/consolidation alone; REM synthesis is an M5 ablation, not a pretraining assumption. Every sleep run logs active-region fraction by hemisphere, workspace-write rate, tag saturation, keep/prune counts, loss, retained-task score, and workspace-audit drift. ### 6.1 Throughput and cost (2x B300, corrected) The original draft claimed ~100,000 tokens/sec/GPU from "ternary addition kernels." That is false on two counts: ternary QAT trains at bf16 speed because latent weights and gradients stay floating point (no Blackwell ternary training datapath exists), and 100k tok/s/GPU at 1B implies ~66% H100 MFU, above the best published small-model runs. Ternary kernels accelerate INFERENCE only. Corrected planning numbers, 2x B300, bf16, 35% MFU: | Run | Tokens | Wall-clock | GPU-hours | On-demand | Spot | |---|---|---|---|---|---| | Architecture proof | 100B | ~4.8 d | 231 | $1,382 | $807 | | **Recommended run** | 300B | ~14.4 d | 692 | $4,145 | $2,422 | | Distill 300B from 8B teacher | 300B | ~49.7 d | 2,385 | $14,287 | $8,348 | | Competitive | 1T | ~48.1 d | 2,307 | $13,818 | $8,074 | Per-GPU throughput at 35% MFU is ~120k tok/s (B300 is ~2.3x an H100). Budget an extra ~50% for ablations and restarts (SmolLM3 spent +58%). fp8 training on Blackwell could roughly halve wall-clock if stable, keeping bf16 master weights. **Do not train on the dev laptop**; the prototypes in `proto/` are numpy toys that run in seconds and exist only to validate mechanisms. --- ## 7. Memory and deployment footprint From `calc/memory_budget.py`: - **Training per GPU (QAT, activation checkpointing):** ~14.8 GB in the current planning envelope (bf16 weights + grads + fp32 Adam + activations), excluding M5 plastic tags and implementation-specific recurrent-state buffers. Measure the real GDN2 allocation before treating this as a deployment budget. - **Inference (M3 minimum):** ~0.37 GB for the packed planning core plus tied embeddings, **excluding** measured GDN2 state and M5 memory. The previous 1.5 GB figure mixed optional tags, VSA, and a 64k hippocampal cache into M3. The calculator now makes every optional allocation explicit. --- ## 8. Milestone ladder (de-risk before scale) Each milestone has a runnable artifact. M0 and M2 already pass in `proto/`. | Milestone | Goal | Artifact | Status | |---|---|---|---| | M0 | Ternary QAT (absmean + STE) tracks fp32 at toy scale | `proto/toy_ternary_lm.py` | PASS: gap 0.07 loss, 35% weights pruned to 0 | | M1 | VSA capacity measured, GIST role confirmed | `calc/vsa_capacity.py` | PASS: ~256 clean items, not 64k | | M2 | Sleep cleanup beats the naive control on synthetic continual recall | `proto/sleep_cleanup.py` | CHECKED: 81.4% vs 34.4% at cycle 8 | | M3a | 100M bf16 bilateral GDN2 workspace beats/matches single-stream and 3:1 hybrid controls | (next) | TODO | | M3b | Winning M3a architecture survives ternary QAT without material regression | (next) | TODO | | M3c | Workspace Jacobian-lens audit passes causal report/broadcast/selectivity tests | (next) | TODO | | M4a | Cerebellar predictor plus Precision Workspace improves state tracking, calibrated uncertainty, and quality/compute | (next) | TODO | | M4b | PFC task state, adaptive loop, and phase routing each earn their own gain | (next) | TODO | | M5a | Causal event segmentation plus Entorhinal/DG/CA3/CA1 episodic path improves boundary-aware retrieval, partial-cue recall, and retention | (next) | TODO | | M5b | Scheduled replay-only sleep first; plastic-memory and REM ablations retain tasks without regression | (next) | TODO | | M6 | 1B run on 2x B300, 300B tokens, report the gap to Qwen3-0.6B | (next) | TODO | | M7 | Attention schema, basal-ganglia action gate, and typed multimodal/tool interfaces improve verified agent behavior | (next) | TODO | --- ## 9. Honest risk register 1. **Attention-free recurrent quality may lag the hybrid control.** Mitigation: make GDN2 attention-free the default experiment, but retain the 3:1 hybrid as the required control and ship it if the quality/recall gap is material. 2. **GDN2 state, bilateral workspace, and ternary QAT may interact badly.** Mitigation: bf16 M3a first; quantize only the winning architecture in M3b. 3. **Wake/sleep weight consolidation risks catastrophic forgetting.** Mitigation: establish no-tag replay first; use candidate checkpoints, then compare plastic tags with a Titans-style gradient memory rather than combining them. 4. **VSA cannot hold 64k tokens.** Already corrected: VSA is a gated gist index, the hippocampal cache is the context store. 5. **The workspace may be decorative rather than functionally global.** Mitigation: require the Jacobian-lens audit and causal ablations before claiming broadcast or using workspace-targeted post-training. 6. **200-300B tokens will not match Qwen3-0.6B.** Accepted: this budget proves the architecture; competitive quality needs multi-T tokens or heavy distillation. 7. **Predictive-coding training is unproven at scale.** Mitigation: train the loop with backprop-through-recursion (Ouro/EBT), keep PC as a metaphor only. 8. **Bilateral specialization may merely duplicate capacity.** Mitigation: share a fixed total parameter budget and require a matched single-stream ablation; remove the split if it does not improve quality at equal active compute. 9. **Sleep can corrupt a deployed model.** Mitigation: session sleep writes only copy-on-write candidates; persistent updates require replay, regression tests, and atomic checkpoint promotion in a maintenance window. 10. **More biological labels can conceal redundant mechanisms.** Mitigation: every added system has one software interface, one metric, and an ablation against the smaller architecture; remove it when it fails. 11. **An attention schema or self-model can be misread as evidence of experience.** Mitigation: frame them as resource-control and reporting mechanisms only; make no consciousness or human-equivalence claims. 12. **The Precision Workspace can cheat by inflating or collapsing variance.** Mitigation: bound log-variance, use proper predictive likelihood and held-out calibration, and compare against a deterministic workspace at equal compute. 13. **The event detector can collapse to every-token or never boundaries.** Mitigation: enforce only minimum/maximum duration safety bounds, report the full segmentation distribution, and require gains over fixed-chunk, turn-boundary, and flat-memory controls at the same byte budget. --- ## 10. Conclusion Mind-1B keeps the ambition of the original HoloMind draft: a bilateral, brain-inspired, event-driven, self-consolidating 1B cognitive engine. It replaces its impossible numbers and pure-spiking bets with the architecture the 2025-2026 literature actually supports: an attention-free Gated-DeltaNet-2 recurrent core, a fast precision stream and slower context stream coupled by a contested global workspace, staged QAT, an audited workspace representation, adaptive recursion, and scheduled sleep. Its fuller cognitive target adds a precision-weighted belief workspace, causally event-segmented hippocampal memory, cerebellar prediction, executive action control, and grounded interfaces, but only after the smaller core earns each extension. The runnable calculators and prototypes in `calc/` and `proto/` make the plan checkable rather than aspirational, and the milestone ladder turns the biggest bets into falsifiable experiments before any expensive training run on the 2x B300 cluster.