A quality signal still needs a cost review

A deliberately awkward comparison

The paired core-report test creates two seeds of control and candidate metric files. In its main fixture, the candidate has a lower final validation loss but takes twice the recorded wall time. These are fabricated test records, not newly trained models.

On September 20, 2026, all nine tests in m3a.test_core_report passed under the isolated standard-library runner. The receipt records the revision and file hashes. No model, training data or GPU was involved.

Read the three outputs separately

For the fixture with a consistent quality difference, the report returns quality_signal_review_cost. It also returns efficiency_proven: false and training_authorized: false. The elapsed-time ratio is 2. Those fields deliberately prevent a narrower observation from turning into a broader claim.

The fixture uses validation values of 4.70 for the control and 4.68 for the candidate. That arithmetic is only a test input for the report logic. It must not appear in a model comparison table as a measured September result.

Make the comparison contract fail visibly

The remaining tests challenge the conditions around the apparent signal. A single seed cannot stand in for the required pair. Mixed signs across seeds do not become a clean win. An out-of-distribution regression can block the favorable verdict.

The suite also rejects incompatible code, runtime, data or control settings, invalid final records and wrong parameter budgets. These checks express what this particular report is allowed to compare. They do not establish that all relevant confounders have been eliminated from a real campaign.

Keep bookkeeping and science distinct

The observed result is that the current report handles the supplied fixtures as expected. That is useful: an automated summary should not quietly promote a candidate because one attractive number improved. It should expose the conditions and the cost question that remain.

It is still possible for a real experiment to have problems outside these fixtures. Data preparation, numerical correctness, measurement conditions and the scientific usefulness of an evaluation require their own evidence. A report can enforce a contract without proving that the contract answers every important research question.

The reproduction path

From mind-1b at the recorded revision, run python3 -m unittest m3a.test_core_report -v. The suite imports the metrics-only report and standard-library helpers, not the training stack. Preserve the distinction between running these fixtures and launching an experiment.

Read the evidence ledger for the actual measured baseline. This note adds a verified reporting behavior: a quality signal can remain a reason to review cost, with neither an efficiency claim nor automatic permission to train again.