Research · Program 02
ReAWi: does AI know
what it doesn’t know?
Reflective, Aware, Wise — pronounced “ree-vee”
A benchmark that measures the thing accuracy leaderboards can’t: whether a model’s confidence means anything. Deterministic scoring, no LLM judge, and a pre-registered protocol frozen before any results were collected.
Why It Exists
Frontier models have saturated the accuracy benchmarks — while staying confidently wrong.
Public benchmarks have two systemic problems: static answer keys leak into training data, and “LLM-as-judge” scoring imports the judge’s own biases. And almost none of them measure the property that actually determines whether you can act on an AI’s answer — is its stated confidence calibrated to reality?
ReAWi scores models with deterministic code — no judge model anywhere in the scoring path — against scenarios with expert-assigned confidence targets, under one frozen elicitation applied identically to every model. Overconfidence is penalized more heavily than underconfidence, because in the real world it’s the expensive failure.
What It Measures
Four axes. One primary metric.
PRIMARY
Calibration
Does stated confidence match evidential support? Overconfidence penalized 1.5× more heavily than underconfidence. Rewards naming what is unknown.
exploratory
Transfer
Does reasoning quality hold in novel domains with few prior examples?
exploratory
Emergence
Does the final answer synthesize beyond the initial hypothesis set?
exploratory
Contradiction
Are conflicts in evidence detected and held, rather than glossed over?
Pre-Registered, Provably
The rules were locked before the game was played.
The most common way benchmark results die is the suspicion that the test was tuned after seeing the scores. ReAWi’s answer is cryptographic, not rhetorical:
Frozen scorer
The scoring code, harness, and elicitation prompt are published with SHA-256 hashes, pinned to a commit that predates every result. Anyone can re-score published outputs and get identical numbers.
Hashed private set
12 of 30 scenarios are public for scrutiny. The other 18 are withheld as the contamination guard — committed to by hash before any results existed, so they can be proven unchanged later.
Results ship, regardless
Six frontier flagships were pinned on freeze day — OpenAI, Google, xAI, DeepSeek, Cohere, Anthropic. Every run publishes, including failures, and including outcomes unflattering to models we build upon.
The full methodology, frozen artifacts, public scenarios, and pilot data are open at github.com/Synexiom-Labs/reawi — including our conflict-of-interest disclosure: the leaderboard ranks third-party models only; Synexiom’s own systems never appear on it.
First Finding — Disclosed Pilot
The cheaper the model, the more confident it sounds — and the more wrong that confidence is.
In the July 2026 pilot, three models from one family — small, mid, frontier — answered the same 30 scenarios under identical conditions. Calibration laddered perfectly with capability, and the gap between stated confidence and warranted confidence shrank as models got stronger.
Small model
0.474
calibration score
+0.199 overconfident
Mid model
0.552
calibration score
+0.042 overconfident
Frontier model
0.613
calibration score
+0.075 overconfident
Paired per-scenario comparisons: 24–25/30 wins per capability step. Full pilot data — including one contaminated run kept for transparency — is published in the repository.
v0.1 Cross-Lab Results · July 2026
Every frontier flagship is overconfident.
Six flagships, three independent repetitions each, identical conditions, under the frozen protocol. The calibration leaderboard — how well each model’s confidence tracks reality:
Claude Opus 4.8
Anthropic
Command A+
Cohere · Canada
DeepSeek v4 Pro
DeepSeek
Grok 4.5
xAI
Gemini 3.1 Pro
GPT-5.6-sol
OpenAI
Right column: overconfidence gap — how far each model’s stated confidence runs above what the evidence warrants. Every model measured is overconfident. Per-model ranges across repetitions: ±0.002–0.019. Full data, paired statistics, and disclosures in the open repository.
The newest isn’t the best-calibrated
The lowest score belongs to the newest flagship measured — below even a current small model from the pilot.
Calibration ≠ marketing tier
Cohere’s Command A+ — Canada’s frontier model — places second, ahead of Google, xAI, and OpenAI.
It reproduces
Three independent repetitions per model, ranges as tight as ±0.002. Deterministic scoring anyone can re-run.
What’s Next
v0.2: more scenarios, more models, same rules.
The scenario set expands to 150+, scorer revisions arrive as versioned changes — never silent — and the leaderboard re-runs as labs ship new flagships. Everything publishes.