I tested Jev on 104 chess positions and used Stockfish 19 to grade its moves, then put it beside Maia — networks trained to play like humans at a given rating, which like Jev answer without calculating ahead. A 1100-rated beginner network beat it on every measure, a hundred times faster. Jev is clearly playing chess and not guessing, and its shortlist of candidate moves carries real signal. But four ways of asking and three ways of showing it the position did not change the answer: on its own, it does not play well.
Jev received the position and its legal moves, and returned one move, a probability for each option, and a confidence score. It never saw Stockfish's evaluation. Stockfish graded the move afterwards at depth 18, keeping five lines.
One of the 104 positions has a single legal move, which the code plays without asking anyone. Every model number below is over the other 103 decisions.
Top-1 means the move matched Stockfish's first choice. A different move can still be fine, so the more useful column is how much evaluation the move gave up: 100 centipawns is one pawn. A blunder here means losing 200 centipawns or more against the oracle's best.
Six ways of choosing a move. The names in brackets are what the raw result files call them.
How often the chosen move was among Stockfish's top 1, 3 and 5 lines.
Centipawn loss, median and 90th percentile. Lower is better; the axis is logarithmic because the tail spans three orders of magnitude.
The hybrid keeps Jev's top K moves and lets Stockfish search only those. To find out what Jev's ranking actually contributes, I ran two controls at the identical search budget: the same K drawn at random, and a search over every legal move.
Same search, same depth, same K. The only difference is who picked the candidates.
Everything above prunes only the first move. There is another thing you can build, and it is what the project set out to reach.
Jev is asked once. It narrows the first move to 5 candidates, and from then on the search expands every legal move as usual. Jev shapes one decision; Stockfish does the rest.
Jev is asked at every position in the tree. Only its 5 guesses are explored at each ply, so the tree stays narrow all the way down. Final positions are scored by counting material — no engine anywhere.
| Positions examined, 3 plies deep | ply 1 | ply 2 | ply 3 |
|---|---|---|---|
| No pruning at all | 39 | ~28 | ~30,600 |
| Prune the root only | 5 | ~28 | ~3,900 |
| Jev prunes every ply | 5 | 5 | 125 |
Counted on the Kasparov position below, which has 39 legal moves. The cost moves from search time to requests: one call per position examined.
Counting material at the leaves is the one piece of chess knowledge in that loop that is mine, not the model's. The obvious next step is to hand that job over as well — so I measured whether the model's position score is good enough to search on.
Stockfish searching every legal move, against the hybrid searching only Jev's shortlist, at the same depth. Green is the move Kasparov played.
Four reformulations of the question had failed by this point, and none of them had changed what the model actually sees. It gets a FEN — r1bqkbnr/pppp1ppp/... — and has to rebuild the board in its head before it can reason. The weakness is specifically positional, and position is exactly what a FEN hides.
Everything above compares Jev with Stockfish, which is not really a fair frame: Stockfish answers "what is best" by calculating, and Jev is being asked something closer to "what would a good player do". Maia answers that second question directly — networks trained on Lichess games at a given rating, run with search switched off. Same kind of object as Jev: look at the position, name a move.
The benchmark scores single moves in hand-picked positions. A game is different: mistakes compound, and the positions you reach are consequences of your own earlier moves.
How often each policy picked the same move as each human network. Not who is better — who plays alike.
I went looking for a signal I could prune or search on, and came back empty four times. But the four failures were not random, and lining them up says something the individual results do not.
Confidence is a statistic TypeSafe derives from the concentration of the option distribution. It is not a stated probability of agreeing with Stockfish, so what follows is the association between the two on this dataset — not a measure of the model's self-knowledge.
Each point is a confidence bucket. Vertical position is how often the move matched the engine's first choice; bubble area is how many decisions landed there. The dashed line is where the two would coincide exactly.
Almost everything sits in the lowest bucket. Any rule based on high confidence would fire very rarely here.
The same run sliced by position type. The categories come from deterministic features this project computes, never from the engine. Several slices are small — the count is on every row.
Ten games per matchup, colours alternating, adjudicated a draw after 160 plies. The hybrid searched at the same depth it was benchmarked at.
Read the Stockfish rows for what they are worth, which is not much. Losing every game to Stockfish places nothing: practically every human alive loses every game to Stockfish, and so do most engines. It tells you the opponent is strong, not how strong the policy is. The row that carries information is the one against a rated human network — that is a scale with people on it.
A projection, not a measurement: it assumes mistakes are independent across a game, which they are not.
Every game played in these matches, move by move, exactly as recorded in the PGN.
Every number on this page comes from a file written by a run. Nothing is recomputed here that an experiment did not already measure.
In the repository, each run leaves results/<run>/ holding config.json (git commit, seed, dataset hash, engine and model settings), summary.json (every metric above) and runs.csv (one row per position, with the confidence, the loss, and whether the policy was actually consulted). Matches also write games.pgn. The dataset is tests/fixtures/positions.json, and scripts/build_dashboard.py rebuilds this page from those files.