Same code, different score: from pinning the answer to voting on it
Lumorem audits a repo and returns performance findings ranked by user impact. The ranking is the product. So this, from two audits of the exact same commit, is a bug report against the whole idea:
| Audit | Finding | Severity | Score |
|---|---|---|---|
| July 1 | Sequential upserts in admin PATCH (N+1) | P1 | 1.3 |
| July 4 | Sequential upserts in admin PATCH (N+1) | P2 | 0 |
Same file, same code, same detector. The drift lived in one stage:
LLM review
judgment: confirm, context, magnitude
First fix: record the opinion
The LLM classifies how each file is used, an admin PATCH can reasonably be read two ways, and the two readings sit on different scoring thresholds:
July 1: INPUT_RESPONSE → 150ms > 100ms threshold → P1, score 1.3
July 4: LONG_OPERATION → 150ms < 1000ms threshold → P2, score 0So I recorded the opinion: the first classification of a route is persisted and pinned, overridable from the UI, and every candidate must come out of review as a finding or an explicit rejection. Audits of unchanged code became reproducible.
Reproducible is not right
Pinning the first classification means the product's opinion of a route is whatever the model said the first time it looked. If both July readings were reasonable, saving July 1 doesn't make it the good one. It makes it the lucky one.
So before rebuilding anything, I went reading. First question: is there even a "true severity" to recover? For human experts, not as a point value:
51%
of duplicate bug reports (same underlying problem, reported twice) carry different human-assigned severities
Tian, Ali, Lo, Hassan 2016
68%
of trained CVSS raters disagree with their own earlier score when re-rating the same vulnerability
Wunder et al. 2024
Second question: why does the model answer differently on byte-identical input, even at temperature 0?
1,000
identical prompts
temperature 0
co-batched on the GPU with other users' traffic
80
unique completions
Thinking Machines 2025
A single LLM call is one draw from a distribution, and no API parameter collapses the distribution. The old design drew one sample and archived it.
Decide by majority, explain once
The rebuilt review splits judging from writing:
STAGE A · DECIDE
batch of candidates
vote 1
vote 2
vote 3
majority per field
2/1 split? +2 votes, majority of 5
Only compact categorical fields come back: confirmed, degradation bucket, recommendation. Last audit: 3 batches of 24 escalated.
STAGE B · EXPLAIN
confirmed candidates only
one call per file: title, remediation, rationale
Stage A decisions are passed in as final. The prose writer cannot re-litigate the verdict it is explaining.
Two design choices came straight from the reading.
Buckets replaced free milliseconds. The model used to output degradation_ms: 150 and the same N+1 straddled a tier boundary run to run. Categorical outputs are the most stable format an LLM judge produces and free numbers the least, so the model now picks a bucket anchored to the thresholds the score already uses. For a click:
IMPERCEPTIBLE
0-50ms
WITHIN_TOLERANCE
50-100ms
NOTICEABLE
100-200ms
DEGRADED
200-500ms
SEVERE
500ms+
Lighthouse findings skip the bucket entirely and carry the measured value.
Confidence stopped being a field the model fills in. Verbalized confidence is systematically overconfident (Xiong et al.); agreement across samples is the signal that tracks reliability (Farquhar et al., in Nature). So confidence is computed from the tally and rendered as dots next to each finding:
unanimous
●●●●○
settled on escalation
●●○○○
the ensemble is not sure, in public
What five opinions cost
Triple the calls should triple the bill. It doesn't, because votes 2 and 3 are byte-identical to vote 1, and a cached read is billed at a tenth of the input price. The request is assembled in cache-stable blocks:
system prompt
business context
file content
decision task
vote 1 runs alone and writes the prefix to cache: 105,689 tokens across one audit's 24 batches.
votes 2 and 3
fire in parallel and read it back: 3 uncached input tokens per call against roughly 6,200 cached.
independent opinions per verdict
of input tokens served from cache
the single-call design's cost, not 3x
Receipts
Two full audits of the same codebase, one day apart:
| July 25 | July 26 | |
|---|---|---|
| Findings | 8 | the same 8 |
| Unanimous (3/3) | 7 | 7 |
| Split (3/5, low) | 1: context provider | the same finding |
| Batches escalated to 5 | 2 of 24 | 3 of 24 |
The split finding is the interesting row: a React context provider recreating its value object, a real pattern with an unmeasurable cost. The ensemble is consistently unsure about it, and says so both times. One borderline finding still moved across runs: the admin upserts crossed a bucket edge and went P2 in one audit, P1 in the other. The dots carry the warning; the edge is still an edge. Wunder's raters disagreed with themselves 68% of the time, so "same severity on every run, guaranteed" is not a bar humans clear either. What I can ship is bounded, measured uncertainty.
What I read
| Source | What it showed |
|---|---|
| Tian, Ali, Lo, Hassan 2016 (EMSE) | 51% of duplicate bug pairs carry different human-assigned severities |
| Wunder et al. 2024 (IEEE S&P) | 68% of CVSS users disagree with their own earlier score on the same vuln |
| Thinking Machines 2025 | 1,000 identical temperature-0 prompts produced 80 unique completions |
| Haldar & Hockenmaier 2025 (EMNLP Findings) | Categorical outputs are an LLM judge's most stable format, free numbers its least |
| Wang et al. 2023 (ICLR) | Self-consistency: majority vote over k samples; most of the gain by k=3-5 |
| Farquhar et al. 2024 (Nature) | Agreement across ~5 samples as an uncertainty signal |
| Xiong et al. 2024 (ICLR) | Verbalized confidence fields are systematically overconfident |
The shared move
Pinning the first answer turned one opinion into permanent state. Voting turns the spread of opinions into the state: the verdict is the majority, and the disagreement itself becomes the confidence the user sees. The model still judges. It just doesn't get to judge alone anymore, and when the judges split, the user sees the split.