Your judge must not share ancestry with your author

I built a pipeline where one LLM call drafts a postmortem and a second judges and revises it. Everyone building one of these says the judge has to be independent. Independent of what, exactly? Making that mean something in code came down to three decisions, and to being blunt about what they didn’t solve.

TL;DR

• Independence has to be at the level of weights lineage. Vendor and model name don’t get you there.
• I enforce it as a hard fail in code, before any API call goes out.
• Published research on same-family self-preference bias says the same thing.
• Requiring a citation raises the floor and stops there. Two holes are still open in mine.
• Temperature 0 with a fixed seed didn’t reproduce, so I compare structurally and over N samples.

Two model names, one set of ancestors

The obvious move is to stop a model judging its own output and use a different one. That’s where most people stop.

My pipeline could call several models served through Groq, and those labels trace back to different weights: one lineage from Meta, another from OpenAI’s open-weight releases. A served model’s label tells you who’s hosting it, not whose weights it’s built from.

So my rule ignores names and vendors entirely: an explicit map from label to weights lineage, checked before every judged run. If judge and executor resolve to the same family it raises SystemExit before any API call goes out, rather than logging a warning and carrying on.

FAMILY = {
    "groq-llama70b": "meta",
    "groq-gpt-oss": "openai",
    # Placeholders: nothing routes here yet, but a lineage the map
    # doesn't know is a lineage the gate can't compare.
    "claude-*": "anthropic",
    "ollama-gemma*": "google",
}


def assert_cross_family(judge: str, executor: str) -> None:
    """Run before any API call goes out. An unlisted or mislabeled model
    breaks on the lookup rather than being caught by the comparison."""
    if FAMILY[judge] == FAMILY[executor]:
        raise SystemExit(
            f"judge {judge} and executor {executor} share a weights lineage"
        )

A check that only reads the string in your config file is looking at the wrong thing. My fix is unglamorous, a hand-maintained map where an unlisted model breaks the assertion rather than getting caught by it, and there’s a measured effect behind it: research on self-preference bias in LLM-as-a-judge finds models rate their own output higher than human evaluators do, and a follow-up on scoring bias finds the effect extends to family.

Where the evidence requirement runs out

Real independence doesn’t make the next problem go away.

These postmortems get read cold, months later, by someone who wasn’t in the room. Left unconstrained, an LLM will state confident, plausible “key decisions” it can’t source, and in a document whose whole value is being a reliable record, an invented decision does more damage than a missing one.

So sourcing is structural rather than requested. Both schemas require a non-empty evidence string, and validate_pm rejects any draft missing one. When the judge flags a decision as unsourced, the revision step removes it rather than backfilling a plausible source.

That raises a real floor. But a required citation field is not a correct citation, and two holes are still open in mine.

The frontier moved from “did this claim cite anything” to “does the citation actually say this,” and the second question is much harder to check automatically.

The comparison method you’d reach for first doesn’t work here

When you’re iterating on a prompt, the instinct is to A/B it the way you’d A/B deterministic code: same input, same seed, temperature 0, diff the outputs. I was already sending seed 42 on every call in anticipation of exactly that.

It doesn’t work. Same prompt and seed through Groq, materially different output. That matches what Thinking Machines Lab documents about nondeterminism in LLM inference: batch-size-dependent kernels introduce drift before sampling ever runs, and neither OpenAI’s nor Anthropic’s seed is documented as more than a best effort. Less a Groq quirk than the default behavior of hosted inference.

So I compare structurally instead. Does the output contain a lie, is a given field covered, over N samples read as a distribution. One platform tested, so it’s no proof the approach fails everywhere, but reason enough to test the assumption before building a methodology on it.

Putting the three together

None of it is dramatic alone, and the honest version is smaller than the marketing version. What holds it up is that the lineage check runs in code before anything else does, because a guideline in a doc doesn’t stop a misconfigured run.