Agent Model Benchmark skill and the Yalc Framework

The short version first. We ran Kimi K3 and Claude Fable 5 on 11 real tasks from a production GTM agent, 3 trials each, judged by both Opus and Kimi in both orders. Correctness was dead even, every objective check that Fable passed, Kimi passed too, on every trial. Fable showed a real but modest quality edge overall, a 71 percent win rate that clears significance and is not just longer answers. But the two judges agreed only 48 percent of the time, position bias flipped some verdicts 100 percent of the time on an order swap, and no single task had a clean consensus winner. Kimi K3 runs at roughly a third of Fable's list price. The honest verdict: on the work that has a right answer, the cheap model is as good, and even a careful benchmark disagrees with itself far more than anyone admits.

Kimi K3 is Moonshot AI's flagship model, kimi-k3 on the platform and kimi-for-coding on the Kimi Code membership plan. Claude Fable 5 is claude-fable-5, Anthropic's premium tier and the default brain of Claude Code. On list price Kimi runs at roughly a third of Fable per token, a 3x cost cut sitting on the table for anyone running agents all day. The question is what you lose when you take it. A leaderboard cannot answer that, because a leaderboard tests riddles and toy code, not your skills, your output formats, or your reliability bar. This kit answers it the boring way: run both models on the work your agent already does, and score it blind.

It works in five parts. The kit scans your .claude/skills folder and turns each skill you confirm into a self contained task with a realistic invented input, and a four task starter suite covering coding, extraction, tool use, and long context runs if the folder is empty. Each task runs through both models fully headless, as many trials as you ask for: claude -p on claude-fable-5 reports real measured cost and latency, kimi -p is timed by wall clock with cost estimated from visible tokens and always labeled as an estimate. Every pair is then scored by two blind judges, one from each model family, Opus and Kimi, and each pair is judged in both orders because judge models have position bias. A win only counts when the judges agree; disagreements become ties. Reliability is reported as pass@k and pass^k across trials, errored trials count as failures with their own error rate column, and a model call that returns nothing fails loud. The output is one results page plus model-routing.json and a paste ready CLAUDE.md snippet, so the verdict becomes live routing your agents actually follow. A sample report ships in the repo.

The run's biggest finding is about benchmarks themselves. The two judge families agreed on the winner only 48 percent of the time, and on some tasks the verdict flipped completely when we swapped which answer came first. Score once, with one judge, in one order, the way almost everyone does, and half your verdicts are noise. That is why the tool judges twice, with two families, across both orders, and calls a tie whenever the evidence does not clear significance.

So the play is patience, not a hasty switch. On correctness the cheap model is already even, so the money argument is real. But the quality gap is small and contested enough that the run's own routing policy routed nothing to Kimi and marked every category insufficient evidence. Move your objectively checkable, high volume work to the cheaper model where correctness parity is proven, keep customer facing writing on the premium model, and re-benchmark after any major model release rather than trusting one screenshot.

This is the v2 run: 11 tasks, 3 trials each, every answer pair judged by both Opus and Kimi in both orders, with significance testing. It is deliberately harder to fool than a single judge scoring once, and the results are humbler and more interesting than the first pass.

Correctness is dead even. On all five tasks with an objective check, regex, strict JSON, merge intervals, the bug fix, and the train reasoning problem, both models passed every single trial. Kimi K3 is exactly as correct as Fable 5 on checkable work.

The two judges agreed only 48 percent of the time. A Claude judge and a Kimi judge disagree on the winner more than half the time. That one number should end your trust in any benchmark that uses a single judge, including our own first run.

Position bias was severe. On some tasks the verdict flipped 100 percent of the time just by swapping which answer came first. The judge was picking by slot, not by content. Judge once in one order and you are reporting noise as a result.

Fable 5 does have a real but modest quality edge. Its win rate was 71 percent, the confidence interval clears a tie, and it is not a length artifact since Fable's answers were actually a touch shorter. But no single task produced a clean consensus winner once both judge families had to agree across both orders. Fable is reliably a little better on average, not a task by task champion.

And the honest kicker: the run's own routing policy routed nothing to Kimi. Every category came back insufficient evidence. The tool refused to recommend a switch on data this thin, which is the whole point.

What changed from the first pass: the clean "Fable swept writing five to zero" story from v1 was largely judge artifact. Make two model families agree, swap the answer order, and run it three times, and the sweep dissolves into a small real edge plus a big lesson about how much benchmarks lie.

Clone, preview, run. The dry run builds your task list and costs nothing. The real run needs only the claude and kimi logins you already have.