Same code, different score: making LLM judgment reproducible

Lumorem audits a repo and returns performance findings ranked by user impact. The ranking is the product. So this, from two audits of the exact same commit, is a bug report against the whole idea:

Same file, same code, same detector. Three separate sources of drift, all in the same place:

AST detectors          ← deterministic
  → candidates
LLM review             ← judgment: confirm/reject, fix, context   ◄── all three leaks
  → verdicts + context signals
scoring                ← deterministic formula
  → ranked findings

Leak 1: context re-inferred on every audit

To score a finding, the LLM classifies how the file is used. An admin settings PATCH can reasonably be read as "a click awaiting response" or "an API call". The same model reads it differently on different days:

July 1:  INPUT_RESPONSE  → 150ms > 100ms threshold  → NEEDS_IMPROVEMENT → P1, score 1.3
July 4:  LONG_OPERATION  → 150ms < 1000ms threshold → GOOD              → P2, score 0

Fix: the first classification is persisted and pinned. Later audits receive it as ground truth instead of re-deriving it. The team can override any value from the UI; overrides pin the same way.

// pipeline.ts: all known routes are pinned, team-set AND previously-inferred
const knownRoutes = await prisma.route.findMany({ where: { serviceId } })
for (const r of knownRoutes) pinnedRoutesByPath.set(r.filePath, toSignals(r))
// inferred rows are created once, never refreshed.
// "Reset" deletes the row; the next audit re-infers.

Leak 2: one context, several kinds of finding

All Lighthouse measurements for a URL share one file-level context, which said PAGE_LOAD. Right for LCP, wrong for Total Blocking Time, which is an interactivity cost:

TBT 7543ms vs PAGE_LOAD thresholds (2.5s/4s)        → score 3.2   (wrong)
TBT 7543ms vs INPUT_RESPONSE thresholds (100/200ms) → score ~10   (outranks the LCP)

Fix: the detector already knows which family each metric belongs to, so it stamps it, and the stamp beats the shared inference:

{ auditId: 'total-blocking-time', category: 'poor_inp',
  interactionType: 'INPUT_RESPONSE', ... }   // per metric, in the mapping rule

Leak 3: a finding vanished without a trace

An audit emitted two Lighthouse candidates; the report showed one. No rejection logged, no error. The logs:

[lighthouse] emitted candidates        count=2
[llm-review] file done  /              confirmed=1
rejects for "/":                       none

The LLM returned a verdicts array covering one candidate out of two. Both findings share a root cause (too much JavaScript) and the prompt invites rejecting duplicates, so it apparently "merged" them, by omission. The loop iterated over what came back; nothing noticed.

Fix: accounting. Every candidate that goes in comes out, as a finding or as an explicit rejection:

const covered = new Set(verdicts.map((v) => v.index))
for (let i = 0; i < fileCandidates.length; i++) {
  if (covered.has(i)) continue
  await recordEvent({ kind: 'reject', ...,
    reason: 'Reviewer returned no verdict for this candidate' })
}

Plus one line in the prompt: every candidate gets a verdict; shared root cause means confirming one and rejecting the other naming it. The next audit returned both findings.

What still drifts

• Borderline verdicts flip: a zero-impact memoization nit gets confirmed one run, rejected the next, with well-argued reasons both times.
• Magnitude estimates wobble: the same N+1 was estimated 150ms one run, 100ms the next, straddling a tier boundary.

Two things bound the damage:

• Everything that flips sits at score zero. The findings that carry the ranking are measured and threshold-crossing, and those have been stable since the fixes.
• In production, an audit only re-judges files that changed since the previous one; unchanged files inherit their findings verbatim, no LLM involved. The full-wipe re-audits I run while developing are the maximum-drift scenario. Real users never hit it.

The shared move

All three fixes do the same thing: they turn a fresh LLM opinion into recorded state. The model still judges, it just judges once. The next audit reads the record and re-judges only what changed.