Needle · experimental results
Checkpoint measurements
Results belong to individual experimental checkpoints, not to a released inference service. The Needle-1.0 report explains the architecture, training history, and limitations.
Stage E: the original paired comparison
The blind panel contains 500 requests and 1,180 fields. Another 99 public validation rows bring this comparison to 599 new reference API calls. Needle and the reference receive identical requests. Needle's training excludes the blind requests. The reference's training corpus is not independently known.
- 89.7%
- Needle E field accuracy
- blind panel, 1,180 fields
- 85.8%
- reference field accuracy
- same blind panel
- +1.3 to +6.6
- paired difference, points
- 95% bootstrap interval
- 0.240
- Needle E negative log likelihood
- reference 0.331, lower is better
One seed. Stage E paired evidence. Later checkpoints appear separately below.
Progress by training stage
Field accuracy by checkpoint on each named evaluation panel, against the reference API on identical requests. These are historical comparisons, not independent estimates of one released model. Solid points come from a published evidence file. Hollow points come from the lab note only. Stages are not evenly spaced in time or compute. A label such as needle-1.0-k identifies Stage K, not an immutable artifact hash.
Needle, stage L+11.8 pts vs reference
Needle, stage K+9.9 pts vs reference
Needle, stage L-24.5 pts vs reference
Needle, stage L-3.9 pts vs reference
› Table view
| Panel | Reference | E | G | I | J | K | L |
|---|---|---|---|---|---|---|---|
| Blind test | 85.8% | 89.7% | 98.2% | 98.6% | 97.5% | 97.0% | 97.6% |
| New wording and layouts | 88.2% | 97.9% | 96.3% | 98.1% | |||
| Unseen task families | 96.1% | 63.9% | 63.6% | 80.4% | 71.6% | ||
| Reordered choices | 93.9% | 97.0% | 97.8% | 82.9% | 90.0% |
Linked values are backed by a published evidence file. The others are taken from the lab note's tables.
Stage E blind test, by panel
Field accuracy on Needle's 1,180-field blind panel. These are the same Stage E results summarized above, not additional observations.
| Panel | Fields | Reference | Needle |
|---|---|---|---|
| Judgment, support, semantic, reasoning tests | 484 | 91.7% | 99.8% |
| Policy conditions: majority, override, veto | 240 | 92.5% | 100% |
| New propositions, layouts and policy text | 216 | 90.3% | 88.9% |
| Policy actions, unconstrained | 240 | 63.3% | 60.0% |
| All blind fields | 1,180 | 85.8% | 89.7% |
What this does not show
Needle is not a general reasoner. On 99 public validation questions from HellaSwag, MMLU and WinoGrande, Stage E scores 37.4% against the reference's 96.0%. This small subset is not a full evaluation of those benchmarks. It is trained for typed decisions over stated evidence and policies, and it should be judged there.
Novel wording at E is a tie. On propositions, layouts and policy text unlike anything in training, the two models are statistically indistinguishable, and the reference is better calibrated.
One seed, one run. The reference was called once per request with no retries. Needle's latency is measured in-process on our hardware. The reference's includes HTTP to its provider.
The report keeps each checkpoint and panel separate. The supporting working record preserves the full procedures.
Different latency boundaries
Stage E measured Needle at 41 ms median and 63 ms p95 in-process on our GPU. The reference measured 109 ms median and 192 ms p95 including HTTP to its provider. These numbers do not establish a serving-speed advantage or an API latency commitment.