Interactive rule-discovery benchmark
WitnessBench
Unseen Witness-style line-puzzle games with novel rules. Agents are not told the rules: they must discover them by playing, using only experiments and verbal reasoning (reflecting, adding and deleting rule hypotheses, and planning). Every level the agent beats is scored against the shortest solution the game accepts (OAE).
This is the official WitnessBench site. It has the leaderboard, replays of models playing two demo games, and both games to play in your browser.
- Models
- 13
- Unseen games
- 26
- Levels / game
- 20
- Eval seeds
- 3
- Top OAE-L20
- 46.5Claude Opus 5.5
Replay Claude Opus 5.5 · Medallion Mirage · seed 2
Replay of a recorded episode: Claude Opus 5.5 plays Medallion Mirage (seed 2) and beats levels 1 to 4. The "open full replay" link shows every step.
A recorded demo episode, levels 1–4 of 20, with real frames, reasoning and rules. Pauses at each LLM call are added for reading, and the glowing line traces the agent's actual path.
Leaderboard · V2.0 · 26 games × 3 seeds
Model Evals
Every model plays 26 unseen games with novel rules, 20 levels each, at 3 seeds. Models run in harness v2, each at its own reasoning effort (high, xhigh or max). OAE-L20 is the headline score: action efficiency against the shortest solution over all 20 levels, with later levels weighted more.
| Claude Opus 5.5 | 46.5 ± 3.0 | 1098/1560(70%) | $85.39* | 4.21M | 69.9 | 30.9 | 13.0 |
| GPT-6 Astra Pro | 25.6 ± 3.7 | 882/1560(57%) | $58.88 | 0.94M | 73.4 | 38.7 | 12.8 |
| GPT-6.1 Sol Pro | 21.7 ± 3.8 | 792/1560(51%) | $11.81 | 0.94M | 70.3 | 37.2 | 12.7 |
| Kimi K3 | 18.8 ± 2.7 | 612/1560(39%) | $26.81 | 1.70M | 50.3 | 35.7 | 12.2 |
| Gemini 4 Argon | 17.9 ± 3.2 | 722/1560(46%) | $17.72 | 1.73M | 66.5 | 42.5 | 12.2 |
| GLM-5.3 | 16.6 ± 4.5 | 587/1560(38%) | $32.62 | 7.50M | 96.2 | 44.5 | 12.3 |
| Grok 4.7 | 14.8 ± 1.6 | 606/1560(39%) | $21.08 | 3.37M | 59.4 | 44.8 | 12.2 |
| Gemini 3.8 Flash | 6.7 ± 0.5 | 412/1560(26%) | $7.41 | 1.94M | 61.0 | 42.8 | 10.9 |
| DeepSeek V4 Pro 0813 | 5.5 ± 0.3 | 360/1560(23%) | $9.32 | 1.97M | 60.2 | 57.1 | 11.3 |
| Qwen3.8-2.4T-A95B | 5.0 ± 0.1 | 309/1560(20%) | $10.81 | 1.77M | 54.7 | 57.4 | 10.6 |
| Qwen3.8-27B | 3.1 ± 0.2 | 268/1560(17%) | $3.95 | 1.79M | 48.3 | 62.5 | 10.2 |
| Muse Glimmer 30B | 0.21 ± 0.04 | 65/1560(4%) | $2.14 | 1.40M | 62.9 | 50.1 | 6.9 |
| Qwen3.5-9B | 0.11 ± 0.04 | 54/1560(3%) | $0.06 | 0.23M | 55.6 | 97.5 | 6.4 |
OAE (Objective Action Efficiency) scores each level: scoreL = (shortestL / actionsL)² × 100 if the level is beaten, else 0, where shortestL is the fewest actions the game engine accepts for that level, counting the final submit. A level therefore scores at most 100. A game's score, OAE-L20, is the mean of scoreL over all 20 levels, with level i weighted i. The ± is the sample standard deviation, across the 3 seeds, of each seed's mean over the 26 games. It is not a confidence interval.
* Claude Opus 5.5 was not served through OpenRouter; its cost is estimated from token counts at $4 per million input and $20 per million output tokens, not billed.
GLM-5.3 is shown at its 128K output configuration.
Demo Games
Recorded episodes on two demo games. Click a model name or watch ▶ to open its replay, which shows the board and the model's reasoning, rule hypotheses, plan and actions.
Medallion Miragedemo game · 20 levels| model | seed | levels beaten | actions | LLM calls | Replay |
|---|---|---|---|---|---|
| Claude Opus 5.5, Medallion Mirage, seed 0 | 0 | 20/20 | 885 | 82 | watch ▶ |
| Claude Opus 5.5, Medallion Mirage, seed 1 | 1 | 6/20 | 497 | 65 | watch ▶ |
| Claude Opus 5.5, Medallion Mirage, seed 2 | 2 | 20/20 | 695 | 64 | watch ▶ |
| Qwen3.8-27B, Medallion Mirage, seed 0 | 0 | 0/20 | 300 | 32 | watch ▶ |
| Qwen3.8-27B, Medallion Mirage, seed 1 | 1 | 1/20 | 519 | 56 | watch ▶ |
| Qwen3.8-27B, Medallion Mirage, seed 2 | 2 | 1/20 | 484 | 54 | watch ▶ |
Bastion Leaguesdemo game · 20 levels| model | seed | levels beaten | actions | LLM calls | Replay |
|---|---|---|---|---|---|
| Claude Opus 5.5, Bastion Leagues, seed 0 | 0 | 2/20 | 344 | 44 | watch ▶ |
| Claude Opus 5.5, Bastion Leagues, seed 1 | 1 | 3/20 | 620 | 67 | watch ▶ |
| Claude Opus 5.5, Bastion Leagues, seed 2 | 2 | 2/20 | 386 | 45 | watch ▶ |
| Qwen3.8-27B, Bastion Leagues, seed 0 | 0 | 3/20 | 726 | 77 | watch ▶ |
| Qwen3.8-27B, Bastion Leagues, seed 1 | 1 | 4/20 | 532 | 65 | watch ▶ |
| Qwen3.8-27B, Bastion Leagues, seed 2 | 2 | 1/20 | 336 | 37 | watch ▶ |
Play
Play the two demo games yourself, in your browser. Move with the arrow keys or WASD, confirm with Space or Enter, and press R to reset the level. Each game has 20 levels, and you can start at any of them.
Changelog
Each score belongs to one benchmark version.
v2.0Current
Oct 2026
26 games × 3 seeds, harness v2, models re-evaluated on Fleet Platform
- 26 unseen games × 3 seeds (was 17 games × 5 seeds). All 13 models were run on the new set.
- The leaderboard now reports cost, output tokens, LLM calls and actions next to the score.
- OAE-L20 replaces RHAE-L5. RHAE is scored against human play. OAE is scored against the shortest solution the game accepts, which is defined for every level, and it covers all 20 levels instead of the first 5.
Harness
Harness v2 mostly changes how the prompt presents what the agent already sees; none of the changes tells the agent anything about a game's rules.
- ConvenienceA rejected submission also shows the full submitted path, which the agent could already rebuild from its action list.
- ClearerThe cells ringed in red by a rejection are listed in one line instead of marked on the board; same cells.
- New infoA budget line shows each level's action limit (300) next to the action count the agent already saw, and the reflection calls used out of their limit (1000 per game). Neither limit was shown before, so the agent now knows how close it is to running out.
- DisclosureThe prompt says that an answer without a plan is followed by one random move, as the harness already did.
- RewordedSystem-prompt rules were reworded to match what the harness v2 does.
- CosmeticBoard column numbers are aligned over the dot columns. Column numbers were offset by one character in v1.
v1.0
Sep 2026
First release
- 17 unseen games × 5 seeds, 20 levels per game, 18 models (arXiv:2609.32208).