Skip to main content
Matter State Research

Needle-1.0: a model for typed decisions

Architecture, training, evaluation, and limitations of a 395M-parameter decision model, with results tied to individual experimental checkpoints.

Overview

Needle reads a supplied state and scores the possible answers to typed questions in a single forward pass. It does not generate text. Its model identifier is needle-1.0. The current architecture is a 395M-parameter bidirectional encoder, not a chatbot or a general-purpose reasoner.

This report brings together the architecture, the earlier experiments that exposed shortcuts, and the subsequent marker-encoder evaluations. Results belong to named experimental checkpoints, not to an unspecified production model. The inference API is in development and is not yet available.

On the tested decision families, later checkpoints outperform the commercial comparison API. On general knowledge and unfamiliar task families, that API remains substantially stronger. Latency measurements use different boundaries: Needle runs in-process on our GPU, while the reference includes HTTP to its provider. They do not establish a fair serving-speed advantage.

Throughout, the comparison product is called "the reference API". Detailed measurements remain available separately. Paired evidence and supporting records are linked at the end of this report.

Architecture and answers

  • A 395M-parameter bidirectional encoder with a ModernBERT-large backbone. Each candidate option gets a [MASK] marker. The marker's final hidden state scores that option. One forward pass scores every option in every question.
  • Choice selects among declared alternatives. Score uses an ordered rubric. Noul represents the probability that a statement is true.
  • Zero output tokens and no sampling. Stage G's 48-repeat audit observed no winner flips for Needle against 2.1% for the reference API. This does not establish invariance to rewording, reordered choices, or all future runs.
  • Training proceeds through experimental stages on synthetic data labelled by executable oracles. The reference API's answers are never training labels. A frozen panel of 1,000 cached reference requests is excluded from training.
  • Later stages use about 1,700 task families written by an LLM and labelled by running model-written code. Labels are program outputs, not model opinions. Executability does not prove that generated rules match their prose.

What the earlier experiments established

These are historical diagnostics from precursor checkpoints, not additional Needle-1.0 benchmark panels. Their scores should not be pooled with the later marker-encoder results.

An average can hide a failed model

The initial frozen comparison contained 1,000 requests and 2,600 fields. One candidate improved from 38.6% to 71.0% field accuracy against the reference's 92.5%. Yet it answered false on all 504 requirement-match fields. Balanced accuracy was 50% on those fields, including training data. The aggregate score had concealed class collapse rather than demonstrated policy understanding.

Balanced sampling reversed the bias instead of resolving it. The follow-up answered true on 440 of 504 requirement-match fields and fell to 63.9% overall accuracy. Neither candidate was promoted. Per-field balanced accuracy became part of the evaluation gate.

Correct prerequisites do not establish composition

A historical checkpoint scored 100% action accuracy in the original choice order and 0% when two choices were swapped on 48 replay cases. Randomizing choices on every presentation exposed a skill that the fixed ordering had allowed the model to avoid learning.

The composition curriculum taught small decisions separately before combining them. A Boolean formulation reached 100% on 832 training fields and a 256-field presentation audit. But composing those skills still produced 50% balanced accuracy on action and requirement, with only 7 of 32 complete requests correct. Computing equality correctly did not ensure that the model could express the answer through a variable set of labels.

Prompt wording also dominated. Replacing calibration instructions with training wording moved action accuracy from 28.5% to 97.2% without changing states or labels. The reverse substitution moved training accuracy from 99.3% to 30.2%. The final curriculum run reached 93.8% on the historical swapped-choice cases, still below its 95% gate. No checkpoint was promoted.

These failures motivate separate audits for class balance, label mapping, choice order, wording, and complete requests. They do not establish a causal comparison between the precursor architecture and the later encoder. The comparison record and composition record preserve the full procedures.

Evaluation protocol

  • Identical requests are sent to both models. Cross-field constraints are sent to neither. Answers are scored against oracle or official labels.
  • Exact McNemar tests use discordant fields. A request-clustered bootstrap provides 95% intervals. Paired evidence files are published for E, G, and K.
  • Reference API version 1.13.0 is called once per request, with no retries. None of those calls failed.
  • Needle latency is measured in-process on our GPU. Reference latency includes HTTP to its provider. The boundaries are not equivalent.
  • Most stages use one seed. A second K seed reproduces most panels closely, but the 675-field clean holdout changes from 80.4% to 73.8%. The working record describes that panel as accurate to about plus or minus 4 points per seed. It should not be presented as a precise, seed-independent result.

The original cached panel became development data through repeated use. It is distinct from the later blind test, wording holdouts, and unfamiliar-family panels. A held-out name does not make a repeatedly inspected panel untouched.

Marker-encoder results

Field accuracy compares Needle with the reference API. Every result names its checkpoint and panel.

  • Stage E, blind test, 1,180 fields: 89.7% vs 85.8%, with a 95% interval of +1.3 to +6.6 points. Negative log likelihood is 0.240 vs 0.331. Latency p50/p95 is 41/63 ms vs 109/192 ms, subject to the measurement-boundary caveat.
  • Stage G, blind test: 98.2% vs 85.8%. Policy development: 99.8% vs 78.3%. Paired counterfactual: 99.9% vs 76.7%. New propositions and layouts generated before G trained: 97.9% vs 88.2%.
  • Stage G, choice-order stability: reordering choices flips the winner 0% of the time vs 8.3% for the reference, or 15.6% on low-evidence cases. The first-listed option wins 33.3% of the time.
  • Stage K, the comparison checkpoint retained in the working record: blind 97.0%, new wording 98.1%, clean unfamiliar families 80.4% vs 96.1%, and synthesized held-out families 93.4%.
  • Stage G latency on policy-length prompts: 44/71 ms vs 118/190 ms. On short prompts: 20/35 ms vs 102/160 ms. These use the same unequal boundaries.

Task diversity helped, but not uniformly

Stage I scored 66.6% on the first unfamiliar-family panel, whose reference scored 94.1%. On the separate clean families_v2 panel, I scored 63.9% against 96.1%. Those are different panels, not interchangeable estimates.

Widening hand-written templates improved the panel that had informed the changes, but not the clean holdout. Stage K's synthesized families moved that clean holdout from 63.9% to 80.4%, a gain of about 16 points. Its first run rendered 1,705 admitted families to 68,194 training rows. The synthetic-family holdout and the independently authored unfamiliar-family panels remain distinct.

This gain came with a regression. K's order-panel accuracy fell from 97.8% to 82.9%. The loss concentrated in low-evidence judgment, not simply unstable choice ordering. Stage L's reweighting recovered order-panel accuracy to 90.0% but reduced clean unfamiliar-family accuracy from 80.4% to 71.6%.

The later stage M working record used 176,000 synthesized rows. It scored 79.7% on the clean holdout, 96.4% on the blind test, and 80.1% on the order panel. More synthesized data did not resolve the trade-off in that run. Remaining errors concentrated in counting rule-defined factors, numerical differences, and category-dependent thresholds. M's figures come from the working record, not a separately published paired JSON file. They do not replace the E, G, or K evidence files below.

Where the model loses

  • General knowledge at G: MMLU 41.0% vs 88.7%, HellaSwag 82.8% vs 92.6%, and WinoGrande 73.8% vs 92.6%. MMLU is bounded by what the 395M backbone learned in pretraining. Better-matched data moved it 1.6 points. Larger pretrained backbones without decision-task fine-tuning improved it by 17 to 29 points in the recorded comparison.
  • Unfamiliar task families remain a substantial gap, even after synthesis. A pretrained 7B model without decision-task fine-tuning scored about the same as I on the clean holdout. The experiments do not support size alone as an explanation for the gap.
  • At E, novel-wording accuracy was tied while calibration was worse than the reference's. Unconstrained policy actions scored 60.0% vs 63.3%.
  • A 1.5B causal backbone with LoRA gained about 12 MMLU points at twice the latency and was slightly behind on policy panels. Routing letter-choice questions to a pretrained 7B without decision-task fine-tuning gained 27 MMLU points but lost HellaSwag and WinoGrande performance relative to the fine-tuned encoder.

Limits of the conclusions

Needle is not a general reasoner or a demonstrated drop-in replacement for the reference API. Its strengths are narrow, and a good result on a trained family does not establish reliable reasoning on unfamiliar rules.

Probabilities on general-knowledge tasks are overconfident. Temperature calibration used decision-family splits, not the public general-knowledge panels. A threshold demonstration is not evidence of calibration on another domain.

Most stages have one seed. Comparisons across stages can involve changed data, training mixtures, and checkpoints. Do not combine their best numbers into a single model claim. Latency is not a matched serving comparison. The site demos are fixed illustrations, not a hosted inference service.

Paired evidence

Supporting records

These records preserve historical checkpoints, working procedures, and failed experiments. They support this report rather than represent separate releases.