# Agent Model Benchmark

> Canonical: https://www.yalc.ai/skills/agent-model-benchmark/

Fable 5 vs Kimi K3 on the Claude Code skills you actually run. One command, a blind judge, and an honest verdict.

**Seo Title:** How to Benchmark Kimi K3 vs Claude Fable 5 on Your Own AI Agent | Yalc

## Categories

- intelligence

**Repo:** https://github.com/Othmane-Khadri/kimi-k3-vs-fable-5-benchmark

**Skill Dir:** .

**Booker Url:** https://app.iclosed.io/e/acquisition-and-consulting-llp/yalc-implementation-call

**Github Skill Url:** https://github.com/Othmane-Khadri/kimi-k3-vs-fable-5-benchmark

**Hook:** i pointed Kimi K3 at my Claude Code setup and benchmarked it against Fable 5 on my own GTM agent. i expected the cheap model to get embarrassed. it didn't.

**Before State:** Everyone argues Kimi vs Claude off a leaderboard. A leaderboard never runs your agent, your skills, your formats, or your reliability bar. So the model choice ends up a guess.

**After State:** One command runs both models on the Claude Code skills you already use, blind judges the outputs with Opus, and hands you quality, cost, speed, and reliability per task. No new API key.

## Gates

- **Wire Kimi into Claude Code**
  - N: 1
  - Body: SETUP.md gives you the three ways in. Run the kimi CLI next to Claude Code, point Claude Code itself at Kimi with the endpoint and model swap, or add one note to your CLAUDE.md so Claude Code can delegate a subtask with kimi -p. The kit uses the delegation path, so your existing logins are enough.
  - Tools: Kimi CLI,Claude Code,Cursor
- **It reads the skills you already run**
  - N: 2
  - Body: The runner scans ./.claude/skills, turns each skill into a self contained benchmark task, and asks you which ones to test. Empty folder? A four task starter suite covering coding, extraction, tool use, and long context runs instead.
  - Tools: Claude Code
- **Both models, same task, headless**
  - N: 3
  - Body: Each task runs through claude -p on claude-fable-5 and kimi -p on Kimi K3. Latency is measured on both sides, Fable cost is read from the CLI, and Kimi cost is estimated and always labeled as an estimate. A call that returns nothing fails loud instead of being scored as a loss.
  - Tools: Fable 5,Kimi K3
- **A blind judge scores every pair**
  - N: 4
  - Body: Opus judges both answers with the order randomized so it cannot favor a side, 1 to 10 with a written reason. Errored tasks are excluded from the win and cost math, never counted as losses.
  - Tools: Opus 4.8
- **One page verdict**
  - N: 5
  - Body: results.html shows the winner per task, quality scores, latency, cost with the Kimi estimate labeled, and a headline verdict. A sample report from a real run ships in the repo.
  - Tools: 

**Result Stat:** 11 tasks in the first real run. Correctness parity on every objective check. Kimi won the lean structured tasks at about a third of the price. Fable won all writing and research. Kimi ran 2 to 5x slower.

## Closer

The verdict from the first run. Kimi K3 for high volume correct answer work you clean up after. Fable 5 for the polished stuff you can't babysit.

One trap inside. kimi -p quietly prepends a bullet that breaks strict JSON output. The kit strips it and the setup guide explains it.

## Glance

- License: MIT (Yalc)
- Models: claude-fable-5 and kimi-k3 (kimi-for-coding on the membership plan)
- Keys needed: None beyond your existing claude and kimi logins

**Clone Command:** gh repo clone Othmane-Khadri/kimi-k3-vs-fable-5-benchmark && cd kimi-k3-vs-fable-5-benchmark && node scripts/run.mjs --dry-run

**Yalc Fit Score:** 8

**Yalc Verdict:** The only model comparison that runs on your own work instead of a leaderboard. Fail loud runs, a blind randomized judge, and an always labeled cost estimate keep it honest. Run it before you move any high volume workload to the cheaper model.

## Plain Description

The short version first. We ran Kimi K3 and Claude Fable 5 on 11 real tasks from a production GTM agent. They matched on correctness, every objective check Fable passed, Kimi passed too. Kimi won lean structured output at about a third of the price per token. Fable won every writing and research task. Kimi ran 2 to 5x slower and errored on 2 of 11 tasks. The verdict: Kimi K3 for high volume correct answer work you clean up after, Fable 5 for the customer facing polish you cannot babysit.

Kimi K3 is Moonshot AI's flagship model, kimi-k3 on the platform and kimi-for-coding on the Kimi Code membership plan. Claude Fable 5 is claude-fable-5, Anthropic's premium tier and the default brain of Claude Code. On list price Kimi runs at roughly a third of Fable per token, a 3x cost cut sitting on the table for anyone running agents all day. The question is what you lose when you take it. A leaderboard cannot answer that, because a leaderboard tests riddles and toy code, not your skills, your output formats, or your reliability bar. This kit answers it the boring way: run both models on the work your agent already does, and score it blind.

It works in five parts. The kit scans your .claude/skills folder and turns each skill you confirm into a self contained task with a realistic invented input, and a four task starter suite covering coding, extraction, tool use, and long context runs if the folder is empty. Each task then runs twice, fully headless: once through claude -p on claude-fable-5, which reports real measured cost and latency, and once through kimi -p, timed by wall clock with cost estimated from visible tokens and always labeled as an estimate. A blind Opus judge scores every pair 1 to 10 with a written reason, seeing Answer A and Answer B in randomized order and never the model names, because judge models have position bias and randomization is what makes the verdict defensible. A model call that returns nothing fails loud and gets excluded, never scored as a loss. The output is one results page: winner per task, both scores, the judge's reason, latency, cost, and a headline verdict, with a sample report from the real run shipping in the repo.

In our run the pattern was clean. Kimi took the regex task 10 to 8 because it returned only the regex while Fable added anchors nobody asked for, and it took strict JSON 9 to 8. Reasoning tied at 10 apiece. Coding tied or landed within a point, with both models producing correct code every time. Fable swept all five GTM writing and research tasks. The cheap model is not wrong more often, it is polished less, and on "give me exactly the answer" work it actually followed instructions better.

Price is only one of three budgets. Kimi wins dollars by roughly 3x on list price. Fable wins latency by 2 to 5x, the regex took Kimi 23.5 seconds against Fable's 2.5. And Fable finished 11 of 11 tasks while Kimi finished 9 of 11. So the play is routing, not switching: move regex, SQL, JSON extraction, classification, and internal summaries to Kimi first, and keep client facing copy, research, and anything unreviewed on Fable. Run the benchmark before moving any workload, after any major model release, and monthly if inference is a real line on your bill.

## Yalc Framework

**Workflow Step:** intelligence

### Workflow Narrative

The benchmark sits at the **intelligence** layer of the Yalc workflow. Every other skill produces work; this one tells you which model should be doing that work, using your own skills as the test set instead of a public leaderboard.

The honesty rules are the point. A model call that returns nothing fails loud instead of being scored as a loss. The judge is blind with the answer order randomized. Kimi's cost is always labeled as an estimate because its CLI hides reasoning tokens on stderr. If a benchmark cannot fail loudly, it lies confidently.

**Workflow Position:** Run it before moving any high volume workload to a cheaper model, and re run it monthly. The output is a routing decision that every other skill benefits from: which model handles the lean structured work, which one handles the customer facing polish.

### Trigger Phrases

- benchmark Kimi vs Fable
- which model is better for my agent
- is Kimi K3 good enough
- compare claude-fable-5 and kimi-for-coding on my skills
- run the model benchmark
- should i switch to the cheaper model

### Required Inputs

- An authenticated claude CLI (a Claude subscription is enough, no API credits required)
- An authenticated kimi CLI (installs to ~/.kimi-code/bin, OAuth login with a Kimi membership)
- A .claude/skills folder to benchmark, or nothing at all (the 4 task starter suite runs on an empty machine)

### Outputs

- results.json with per task quality scores (1 to 10 plus a written reason), winner, latency for both models, cost, and pass or fail
- A self contained results.html verdict page with a headline and the full per task table
- Fable cost measured from the claude CLI; Kimi cost estimated and always labeled as an estimate
- Errored tasks recorded and excluded from the win math, never counted as losses

### Chaining

**Upstream:** Your existing Claude Code skills become the benchmark tasks, no rewriting needed

**Downstream:** A routing decision. Move high volume correct answer workloads to the winner, keep customer facing work on the polish winner, re run monthly


### Anti Patterns

- Do not reuse a Kimi membership key inside a standalone script. It is ToS restricted to the official CLIs and can get your access restricted.
- Do not write "run the X skill" in a task prompt. An agentic CLI goes hunting for a skill file by that name and can error out.
- Do not read the headline verdict before the per task table. The split by workload matters more than the total.
- Do not treat the Kimi cost number as measured. It is an estimate that excludes hidden reasoning tokens.


## Pros

- Benchmarks on your own skills, not a public leaderboard
- Symmetric comparison, both models run headless through their own CLI on the identical prompt
- Blind Opus judge with A/B order randomized, so it cannot favor a side
- Fail loud design, an empty response can never be silently scored as a loss
- Honest cost labeling, measured for Fable, estimated and flagged for Kimi
- Works on an empty machine via the 4 task starter suite

## Cons

- Kimi cost is an estimate because its CLI hides reasoning tokens on stderr
- One run is one sample, re run before betting a production workload on the verdict
- The kimi CLI can error on complex agentic tasks (2 of 11 in our run), plan a retry budget
- A full run takes minutes, not seconds, since every task runs twice plus a judge pass

## Who For

- Operators deciding whether to move agent workloads to a cheaper model without gambling in production
- Founders running Claude Code agents daily who review their model bill monthly
- Anyone shipping high volume AI output where "correct and cheap" beats "polished and pricey" for part of the work

## Faq

- Q: Is Kimi K3 as good as Claude?
  - A: On correctness, in our 11 task run, yes. It passed every objective check Fable 5 passed and won the lean structured tasks like regex and strict JSON. On customer facing writing and research, no. Fable won all five of those tasks with more grounded, sharper output.
- Q: Is Kimi K3 good for coding?
  - A: Both models tied or landed within one point on every coding probe, and both produced correct code every time. For snippets you review anyway, Kimi is good enough at about a third of the price. For long agentic chains it errored on 2 of 11 tasks, so plan a retry budget.
- Q: How much cheaper is Kimi K3 than Claude Fable 5?
  - A: Roughly a third of the list price per token. Ignore per task estimates that look 100x cheaper, they exclude the reasoning tokens Kimi's CLI hides on stderr. The kit labels its Kimi cost as an estimate for exactly this reason.
- Q: Does Kimi work with Claude Code?
  - A: Yes, three ways. Run the kimi CLI side by side, point Claude Code itself at Kimi's endpoint with ANTHROPIC_BASE_URL and the kimi-for-coding model, or add one CLAUDE.md note so Claude Code can delegate subtasks with kimi -p. The kit uses the delegation path.
- Q: Do i need a new API key to run the benchmark?
  - A: No. A Claude subscription and a Kimi membership login are the only credentials used. The Opus judge uses ANTHROPIC_API_KEY when one with credits exists and falls back to your authed claude CLI when it does not.
- Q: I have no skills in .claude/skills yet. Is it useless?
  - A: No. A 4 task starter suite covering coding, extraction, tool use, and long context runs instead, so you still get a real verdict.
- Q: What happens when a model call fails mid run?
  - A: The task is recorded as errored and excluded from the win and cost math, and the rest of the run continues. A call that returns nothing can never be scored as a loss. Our first run scored a never launched Kimi as a 5 out of 5 Fable sweep, and that rule exists because of it.
- Q: Can i benchmark a different model pair?
  - A: Yes. The CLIs, model ids, and prices live in config/models.json and config/pricing.json. Any two models reachable through a headless CLI can be compared the same way.

**Reviewer:** Othmane Khadri