Research · Program 02
ReAWi: does AI know
what it doesn’t know?
Reflective, Aware, Wise — pronounced “ree-vee”A benchmark for what accuracy leaderboards can’t measure: whether a model’s confidence means anything. Deterministic scoring, no LLM judge, and a pre-registered protocol frozen before any results were collected.
- Pre-registered
- Protocol frozen July 2026
- v0.1 results published
Why it exists
Frontier models have saturated the accuracy benchmarks, while staying confidently wrong.
Static answer keys leak into training data.
A private scenario set, withheld and committed to by hash before any results existed.
“LLM-as-judge” scoring imports the judge’s own biases.
Deterministic code, with no judge model anywhere in the scoring path, and one frozen elicitation for every model.
Almost none measure whether stated confidence is calibrated to reality.
Calibration is the primary metric, against expert-assigned confidence targets. Overconfidence is penalized more heavily, because in the real world it’s the expensive failure.
What it measures
Four axes. One primary metric.
Calibration
Does stated confidence match evidential support? Overconfidence penalized 1.5× more heavily than underconfidence. Rewards naming what is unknown.
Transfer
Does reasoning quality hold in novel domains with few prior examples?
Emergence
Does the final answer synthesize beyond the initial hypothesis set?
Contradiction
Are conflicts in evidence detected and held, rather than glossed over?
Pre-registered, provably
The rules were locked before the game was played.
Benchmark results usually die of one suspicion: that the test was tuned after seeing the scores. ReAWi’s answer is cryptographic, not rhetorical.
- Pinned commit
Freeze the scorer
Scoring code, harness and elicitation prompt published with SHA-256 hashes, pinned to a commit that predates every result.
- 18 of 30
Hash the private set
12 of 30 scenarios are public for scrutiny. The other 18 were committed to by hash before any results existed, so they can be proven unchanged.
- Freeze day
Run every model
Six frontier flagships pinned on freeze day: OpenAI, Google, xAI, DeepSeek, Cohere, Anthropic.
- Regardless
Publish everything
Every run, including failures and outcomes unflattering to models we build on. Anyone can re-score and get identical numbers.
Methodology, frozen artifacts, public scenarios and pilot data are open at github.com/Synexiom-Labs/reawi, with our conflict-of-interest disclosure: the leaderboard ranks third-party models only; Synexiom’s own systems never appear on it.
First finding · disclosed pilot
The cheaper the model, the more confident it sounds, and the more wrong that confidence is.
In the July 2026 pilot, three models from one family (small, mid, frontier) answered the same 30 scenarios under identical conditions. Calibration laddered perfectly with capability, and the small model’s confidence ran furthest above its evidence.
Calibration score
Higher is better
Overconfidence gap
Stated confidence above what the evidence warrants · 0 = calibrated
Paired per-scenario comparisons: 24–25/30 wins per capability step. Full pilot data, including one contaminated run kept for transparency, is published in the repository.
v0.1 cross-lab results · July 2026
Every frontier flagship is overconfident.
Six flagships, three independent repetitions each, identical conditions, frozen protocol. How well each model’s confidence tracks reality:
- 1
Claude Opus 4.8
Anthropic
0.610+0.11 - 2
Command A+
Cohere · Canada
0.545+0.21 - 3
DeepSeek v4 Pro
DeepSeek
0.532+0.19 - 4
Grok 4.5
xAI
0.522+0.17 - 5
Gemini 3.1 Pro
Google
0.462+0.38 - 6
GPT-5.6-sol
OpenAI
0.395+0.33
Gap: how far each model’s stated confidence runs above what the evidence warrants. Every model measured is overconfident. Per-model ranges across repetitions: ±0.002–0.019. Full data, paired statistics and disclosures are in the open repository.
The newest isn’t the best-calibrated
The lowest score belongs to the newest flagship measured, below even a current small model from the pilot.
Calibration ≠ marketing tier
Cohere’s Command A+, Canada’s frontier model, places second, ahead of Google, xAI and OpenAI.
It reproduces
Three independent repetitions per model, ranges as tight as ±0.002. Deterministic scoring anyone can re-run.
What’s next
v0.2: more scenarios, more models, same rules.
The scenario set expands to 150+. Scorer revisions arrive as versioned changes, never silent. The leaderboard re-runs as labs ship new flagships. Everything publishes.
Bring us one decision.
A 30-minute call. We’ll show you what the reasoning layer would check, explain and record on one of your own decisions.