All configurations click a header to sort
Route = gold: how often the routed model is exactly the one the router picks from the human labels. Under / over: prompts sent to a cheaper (quality risk) or pricier (money wasted) model than gold. Cost: estimated model cost of routing the 80 prompts (gold ≈ $1.18 at November prices). Conf.: share of topic answers at or above the router's 0.8 threshold. Rows are from different evenings: compare each model with the Jev run of its own session (Jev moves 69–75% on route = gold from run to run).
VegaML's few-shot heads (65% topic, 42.5% gold routes with T=0.72) were scored on held-out halves and aren't in the raw files; see the VegaML post.
Topic accuracy vs routing quality
Topic accuracy by prompt group
jev-latest now points to 1.14Two runs each, same evening 9 Nov 2026 · typesafe/jev-1.13 vs jev-1.14
Confidence threshold sweep
| Threshold | 1.13 cost | 1.13 = gold | 1.14 cost | 1.14 = gold |
|---|---|---|---|---|
| 0.8 (current) | $1.31–1.35 | 71–74% | $1.47–1.48 | 70–71% |
| 0.7 | $1.32 | 75% | $1.44 | 70% |
| 0.6 | $1.28 | 76% | $1.43 | 72.5% |
Reading it
- Same model family: the versions agree on the topic for 95–96% of prompts and on the routed model for 84–86%.
- 1.14 is the better classifier (risk, calibration, a bit faster) but is confident less often, so the router plays safe more often.
- A lower threshold doesn't close the gap: 1.14 costs more at 0.7 and 0.6 too.
- Only 11–13 prompts per run route differently: a direction, not a measurement.
- Side finding: both versions score risk ~0.53 above the labels. A
risk_offsetof −0.5 would give ~80% risk accuracy on these 80 prompts, which is overfitting; left at −0.3.
Where the requests would go
Prompts routed differently by the two versions
KEV_TEMPERATURE=1.11 (Jev: 61%)Kev-4B, three ways to read its confidence 9 Nov 2026 · MLX bf16
Why three runs
As shipped, Kev applies T = 2.41 fitted on its own dev set: only 19% of topic answers clear 0.8 (all right), so the router over-provisions. T = 0.46 on the client (cross-validated log-loss fit) fixes the topic confidence, but the router's temperature only rescales choice answers: complexity and risk keep Kev's flat distributions, squeezed toward the middle, and 14 prompts go to a model cheaper than gold. KEV_TEMPERATURE=1.11 (2.41 × 0.46) makes Kev's server sharpen every answer: same topics, risk 61% → 74%, gold routes 61% → 67.5%.
Complexity and risk: label vs answer
Complexity (0 trivial → 3 expert)
Risk (0 → 2)
Decision latency per prompt log scale
The first run with KEV_TEMPERATURE=1.11 lost four prompts (three long) to the bench's 60 s timeout while the Mac was ~6 GB into swap; the bench now gives Kev-4B 180 s and the tables show the clean re-run. The 3 GB 4-bit MLX build (AbstractFramework/kev-4b-mlx-4bit) should avoid the swapping and hasn't been run yet.
A confidence temperature per provider
The router raises the quality floor when topic confidence is below 0.8, so a model that is right but unsure still over-provisions. providers.<name>.temperature divides the logits by T before the softmax: it never changes which answer wins, only how confident it sounds. T is fitted by minimising log-loss on half the prompts and measured on the other half (20 random 2-fold splits).
| Model | Fitted T | Confident ≥ 0.8 (held-out) | Route = gold | ECE (in-sample) |
|---|---|---|---|---|
| Kev-0.8B | 0.47 | 5% → 60% (89% right) | 22% → 39% | 0.349 → 0.059 |
| Kev-4B | 0.46 | 21% → 74% (94% right) | 39% → 61% (67.5% server-side) | 0.280 → 0.080 |
| Laya typed-decisions | 0.43 | 0% → 33% (92% right) | 12.5% → 25% | 0.509 → 0.072 |
| Von 1.3 | 2.47 | 75% → 26% (69% right) | 48% → 36% | 0.226 → 0.236 |
| Clef-flash 9B 4-bit | 0.64 | 56% → 81% (95% right) | 52.5% → 64% | 0.101 → 0.072 |
| Laya fine-tuned | 0.87 | 0% → 86% | 35% → 64% | 0.311 → 0.082 |
| VegaML 0.8B (zero-shot) | 0.60 | 1% → 16% (93% right) | 17.5% → 20% | 0.162 → 0.092 |
Confident answers and gold routes, before → after
A temperature fixes "right but shy" models (Kev, Clef-flash, the Laya fine-tune) and can't help models whose errors are confident (Von) or that are unsure because they're often wrong (VegaML zero-shot). With 80 prompts, treat fitted values as a starting point.
Prompt by prompt
red: different from the human label (topic: not in the acceptable set). Topic confidence in brackets; below the 0.8 threshold in grey. Gold: the model the router picks from the labels, at that evening's prices (OpenRouter prices changed on 11 Oct). Prompts longer than 220 characters are cut here.
The series, and what was checked but not run
- 23 Sep 2026Jev 1.13 vs Laya (English, multilingual, auto): the first 80-prompt benchmark. post
- 2–3 OctLaya 0.3.24, typed-decisions, Von 1.3, Kev-0.8B, a confidence temperature, and a Laya fine-tune on 480 labels. post
- 4 OctClef-flash 9B (Cloudflare, 4-bit MLX): first open model to match Jev on topic. post
- 11 OctLaya 0.4.2 (identical answers) and VegaML 0.8B: zero-shot, temperature, few-shot heads from 40 labels. post
- 9 NovJev 1.14 vs 1.13 (two runs each) and Kev-4B in bf16 with three temperature settings. post
- Laya 0.4.2 → 0.4.30 · 28 releasesWeights unchanged since 18–19 Sep; the only Hub commit adds a preprint citation. 0.3.6 → 0.3.24 → 0.4.2 gave identical decisions, so not re-run. Runtime now bundles VegaML (
Agent(vegaml=True)), hooks on vega, video fixes, stricter fit labels. - VegaML 0.8 → 0.16.1vega-U: one engine attached to any reader model, calibrated on your labels; audio, video, trained policies. Weak zero-shot here (51%): only the few-shot heads would be worth a re-run.
- Von · 19 OctArchived; the card now points to LiquidAI d1-omni-600M.
- LiquidAI d1 · 5 & 24 Octd1-2B, d1-Omni-2B, d1-3B, d1-omni-600M: multilingual, text/image/video, runs on MPS. LFM Open License v1.0 (not Apache-2.0), training data vs Jev not stated. Next candidate.
- vllm-sr d3 · Octd3-mini (4.5B) peaks at 14.8 GiB in bf16; d3-flash and d3-turbo are bigger. Not on 16 GB.
- Clef / Clef-flashNo new weights, only evaluation results on the cards.
- autotrust JEV-* and the floodMost-downloaded "System One" models, trained on Jev outputs (excluded: TypeSafe's terms forbid distillation). Dozens of community fine-tunes per day (zils-*, OneJev…).
What the benchmark does
80 labelled prompts, four questions
57 core dev tasks, 8 in other languages, 4 longer than 512 tokens, 7 tricky or ambiguous, 4 with private data. Each prompt gets the router's four questions in one request: topic (choice among 10), complexity (score 0–3), risk (score 0–2) and private data (boolean).
From the answers the router computes a quality floor, raises it when topic confidence is below 0.8, and picks the cheapest of five OpenRouter models above the floor: Qwen 3.7 Flash, DeepSeek V4.1 Flash, GPT-5.6 Luna, Claude Sonnet 5, Claude Opus 5.5.
No prompt is sent to a chat model
Only decisions are compared. The gold route is the model the router picks when fed the human labels. Metrics: topic/complexity/risk accuracy, confident share, ECE (distance between stated confidence and accuracy), route = gold, under/over-provisioned prompts, estimated cost, latency.
Code, cases and setup: docs/benchmark.md. Raw data on this page: the bench's own JSON output, one file per run.