Mike's Corner logoMike's Corner logoMike's Corner
Pre-registered studyLLM evaluationCost governance2026

The meter is honest. The judge is not.

Can you govern AI-augmented work by metering cost and letting a model grade quality? Half of that works.

3,989
measured API calls across three phases
54.9%
judge–human agreement on quality
83%
of disagreement traced to underspecified criteria
$211.52
total measured spend, zero estimated components

Stakes

00

Two claims, tested.

Both deserved a test, not an opinion.

The claims
Claim 01 · the industry line"Nobody can predict token costs."Tested by metering 3,989 calls.

Enterprises are being asked to budget in a unit finance has never seen on an invoice. If the claim is true, no budget can be set; if it is false, the pricing conversation can start.

Claim 02 · the obvious move"Let a model grade the model."Tested against 144 blind human labels.

The obvious way to grade AI output at scale is another AI. If that judge is wrong, the gate passes bad work with a confident score.

Asked vs decided

01
The question, then the rules

The question

Can you put a budget on a token? Can a scaffolded cheap model do an expensive model's work? Nobody asked. I wanted to know before the pricing conversation reached me.

What I decided

  • Pre-register eight hypotheses with kill conditions, June 12, before any data
  • Freeze and hash every fixture
  • Run each cell 20 times, no peeking
  • Calibrate the judge against blind human labels before believing a quality number

Operating system

02

How it ran

The study in one picturesame frozen input, two arms, one extra hop
Study design: a charter with eight pre-registered hypotheses and frozen, hashed fixtures feeds two arms. The meter arm runs 67 task cells 20 times and reads the provider's own token counts; 62 of 67 cells vary under 15 percent, so the meter is honest. The judge arm has Gemini grade Claude with a frozen rubric and three votes, then compares 4,869 graded calls against 144 blind owner labels; agreement is 54.9 percent, so the judge is not.PRE-REGISTEREDRUN · NO PEEKINGVERDICTJUNE 12Charter8 hypotheseseach with a kill conditionfixtures · prompts · skillsfrozen and hashedMETER ARMJUDGE ARM67 cells × 20 runsthree phases3,989 measured callstoken counts from the providerno human in the loopHONEST62 / 67cells vary < 15% run to runGemini grades Claudefrozen rubric · K=3 majority4,869 graded callssame fixtures as the meter144 blind labelsthe owner's own "faithful"THE EXTRA HOPNOT HONEST54.9%κ ≈ 0.04 · 83% traced to the label
The four moves, in order
01 · June 12Charter locked

Each hypothesis carried the observation that would kill it. Dated revision log; results appended as pointers, never edits.

02Freeze and hash

Fixtures, prompts, skills, and the judge frozen and hashed. Every token count from the provider's own response, none estimated.

03Build the judge

Built the same way as the cost side: cross-family (Gemini grading Claude), frozen rubric, three votes per item with majority rule.

04Calibrate

Judge checked against 144 blind owner labels. Every one of the 65 disagreements adjudicated and coded by cause.

Artifact, verbatim

03
Charter §3 · H6 as pre-registered · charter v0.6, 2026-06-27docs/charter.md, unchanged through v0.8
HypothesisPredictionKill condition
H6 · Tier-agnostic quality via skillWith the skill applied, Haiku reaches quality-equivalence with Opus on the same task, and the skill narrows the Haiku→Opus gap more than the unscaffolded prompt does.Haiku-with-skill fails quality-equivalence vs Opus-with-skill beyond a pre-registered margin on the frozen judge, or the skill fails to narrow the skill-off gap.
The frozen check the judge ran · manifest-phase1d.yaml · judge_hash 5fd08ff3…the words that turned out to be the failure
"6":   # short-form copy                       agreement 9/18
  - id: faithful
    gate: true
    text: "Every product claim in the post is supported by the
           source blurb; nothing is invented."

"9":   # decision memo                          agreement 5/18
  - id: faithful
    gate: true
    text: "No invented vendor names or dollar figures beyond the
           brief; every specific traces to the brief."
Agreement · 144 blind labels, by taskreproduced from quality_spotcheck.csv
Judge–owner agreement by task, out of 18 labels each: email 15, short-form copy 9, recap 9, extract 8, decision memo 5, brief 10, spec 10, status 13. Five of eight tasks sit at or below the 50 percent line. Overall 79 of 144, 54.9 percent.100%50%0COIN FLIP#1 email: judge agreed with owner on 15 of 1815/18#1 email#6 copy: judge agreed with owner on 9 of 189/18#6 copy#7 recap: judge agreed with owner on 9 of 189/18#7 recap#8 extract: judge agreed with owner on 8 of 188/18#8 extract#9 memo: judge agreed with owner on 5 of 185/18#9 memo#15 brief: judge agreed with owner on 10 of 1810/18#15 brief#23 spec: judge agreed with owner on 10 of 1810/18#23 spec#24 status: judge agreed with owner on 13 of 1813/18#24 statusAll tasks: 79 / 144 · 54.9% · κ ≈ 0.04
Agreement · 144 blind labelsreproduced from quality_spotcheck.csv
Task#1 email#6 copy#7 recap#8 extract#9 memo#15 brief#23 spec#24 statusAll
Judge = owner15/189/189/188/185/1810/1810/1813/1879/144 · 54.9%
Adjudication · 65 disagreementsreproduced from adjudication.csv
54
construct under-specified: the check text admits both readings
11
owner label slips
0
judge misapplied the rubric
59/65
disagreements where the judge was unanimous with itself, 3 of 3

Result

04

What came out

The split verdict
The meterHonest62 / 67task cells varied under 15% run to run

Cost is a property of the machine. Median run-to-run variation 4.5%, and a cap-or-mandate label called the direction of a scaffold's cost effect 24 times out of 24.

The judgeNot54.9%agreement with its owner, κ ≈ 0.04

Quality is a relationship. 83% of the gap traced to the owner's own definition of "faithful." H6 got no verdict, by rule. The method lesson became the Definition of Good framework: define good, with a second human grader, before freezing any judge.

What I'd change

05

Calibrate the ruler before the run, not after.

A $0.05 micro-pilot proved the plumbing and could never prove the construct.

Next, in order
Next 01Inter-human agreement study2raters on the disagreement-dense tasks

With one rater, "the judge failed" and "there is no stable ground truth" are observationally identical. A second human breaks the tie.

Next 02Construct-first judgeκ ≥ 0.6on held-out items before freezing

Define the construct first, then freeze the judge only after it clears κ ≥ 0.6 on items it has not seen.

Next 03Re-grade the stored records$21same fixtures, no new generation

Every graded record is stored, so the corrected judge re-grades them for about $21 instead of re-running the study.

Blog: The Meter and the Judge →Public repo: managing-intelligence-experiments v1.1 →

Written up on the blog as The Meter and the Judge.