Skip to content

calibrated_agent_evaluation.judges

calibrated_agent_evaluation.judges

Judge-level layer.

The three judge levels a crew is judged at -- workflow (whole trace), span (one agent's input+output), process (the span tree) -- with the "calibrate at the level you attribute at" discipline enforced, plus the deterministic process judge (expected-vs-actual agent set-difference + ordering, no LLM call).

Pure methodology plane: reads normalized Trace/Span and the crew Define artifact; no vendor type appears here. It does NOT execute LLM workflow/span judges -- that batch fan-out runtime is the judge-execution layer; this package owns the level model + the deterministic process judge only.

Public API re-exports from submodules.

JudgeLevel module-attribute

JudgeLevel = Literal['workflow', 'span', 'process']

JUDGES_SCHEMA_VERSION module-attribute

JUDGES_SCHEMA_VERSION = '1'

VerdictCache

VerdictCache()

An in-memory, round-trippable map from (judge version, target) to a cached Label.

Not a context manager: it owns no connection/handle (pure in-memory dict + explicit dump/load), so the conventions' context-manager rule does not apply.

Source code in src/calibrated_agent_evaluation/judges/cache.py
def __init__(self) -> None:
    self._entries: dict[str, Label] = {}

get

get(judge, trace_id, span_id, sample=0)

Return the cached Label for this judge version + target, or None on a miss.

Returns None when the judge's model/prompt_version/content differ from any stored entry, so a re-versioned judge never serves a stale verdict, and for a repeat number not judged yet.

Source code in src/calibrated_agent_evaluation/judges/cache.py
def get(
    self, judge: ExecutableJudge, trace_id: str, span_id: str | None, sample: int = 0
) -> Label | None:
    """Return the cached ``Label`` for this judge version + target, or ``None`` on a miss.

    Returns ``None`` when the judge's ``model``/``prompt_version``/content differ from any
    stored entry, so a re-versioned judge never serves a stale verdict, and for a repeat
    number not judged yet.
    """
    return self._entries.get(_key(judge, trace_id, span_id, sample))

put

put(judge, label, sample=0)

Store label keyed on this judge version, its (trace_id, span_id) and repeat.

Source code in src/calibrated_agent_evaluation/judges/cache.py
def put(self, judge: ExecutableJudge, label: Label, sample: int = 0) -> None:
    """Store ``label`` keyed on this judge version, its (trace_id, span_id) and repeat."""
    self._entries[_key(judge, label.trace_id, label.span_id, sample)] = label

dump

dump(path)

Write the cache to verdict_cache.json under path (lossless round-trip).

Raises JudgeRuntimeError on any filesystem failure (no raw OSError escapes).

Source code in src/calibrated_agent_evaluation/judges/cache.py
def dump(self, path: str | Path) -> None:
    """Write the cache to ``verdict_cache.json`` under ``path`` (lossless round-trip).

    Raises :class:`JudgeRuntimeError` on any filesystem failure (no raw ``OSError`` escapes).
    """
    out = Path(path)
    payload = {
        "schema_version": JUDGES_SCHEMA_VERSION,
        "entries": [
            {"key": key, "label": to_dict(label)} for key, label in self._entries.items()
        ],
    }
    try:
        out.mkdir(parents=True, exist_ok=True)
        atomic_write_text(out / _JSON_NAME, json.dumps(payload, indent=2, sort_keys=True))
    except OSError as exc:
        raise JudgeRuntimeError(f"failed to write verdict cache to {out}: {exc}") from exc

load classmethod

load(path)

Read a cache back from path; an absent file yields an empty cache.

Version-checks before returning: a stored schema_version != running raises JudgeRuntimeError (no coercion). Wraps filesystem/parse/malformed failures.

Source code in src/calibrated_agent_evaluation/judges/cache.py
@classmethod
def load(cls, path: str | Path) -> VerdictCache:
    """Read a cache back from ``path``; an absent file yields an empty cache.

    Version-checks before returning: a stored ``schema_version`` != running raises
    :class:`JudgeRuntimeError` (no coercion). Wraps filesystem/parse/malformed failures.
    """
    src = Path(path)
    file = src / _JSON_NAME
    cache = cls()
    if not file.exists():
        return cache
    try:
        payload = json.loads(file.read_text(encoding="utf-8"))
    except OSError as exc:
        raise JudgeRuntimeError(f"failed to read verdict cache from {src}: {exc}") from exc
    except (UnicodeDecodeError, json.JSONDecodeError) as exc:
        raise JudgeRuntimeError(
            f"verdict cache file in {src} is not valid JSON: {exc}"
        ) from exc
    if not isinstance(payload, dict):
        raise JudgeRuntimeError(
            f"verdict cache file in {src} must hold one JSON object, got "
            f"{type(payload).__name__}; refusing to load it"
        )

    stored = payload.get("schema_version", JUDGES_SCHEMA_VERSION)
    if stored != JUDGES_SCHEMA_VERSION:
        raise JudgeRuntimeError(
            f"verdict cache schema_version {stored!r} != running {JUDGES_SCHEMA_VERSION!r}; "
            "refusing to load (no coercion -)"
        )
    try:
        entries: dict[str, Label] = {
            entry["key"]: label_from_dict(entry["label"])
            for entry in payload.get("entries", [])
        }
    except (KeyError, TypeError, AttributeError, SchemaError) as exc:
        raise JudgeRuntimeError(
            f"malformed verdict cache in {src} (missing/mistyped field {exc}); "
            "refusing to load a partial cache"
        ) from exc
    cache._entries = entries
    return cache

JudgeLevelError

Bases: CalibratedAgentEvaluationDataError

A workflow-level judge was used for a per-agent rate, which it can't attribute.

JudgeRuntimeError

Bases: CalibratedAgentEvaluationDataError

A failure in the judge-execution runtime.

Covers a model call that failed after exhausting retries, a reply that carried neither a schema-valid structured output nor a valid forced-tool-call payload, and a structured payload that violated the verdict schema (empty critique / non-binary verdict). Wraps the underlying vendor/execute exception via chaining (raise JudgeRuntimeError(...) from exc) so no raw inspect_ai/boto3/stdlib exception escapes the judges/ boundary (conventions: "wrap external errors at the boundary"). The runtime never fabricates a verdict: on any of these it raises rather than persisting a Label.

SpanGoldError

Bases: CalibratedAgentEvaluationDataError

A span-level judge was calibrated on gold derived from the trace outcome, not the span.

ExecutableJudge dataclass

ExecutableJudge(
    judge_id,
    model,
    prompt_version,
    level,
    prompt_template,
    output_fields=("critique", "verdict"),
    temperature=None,
    failure_mode=None,
    agent=None,
)

A fully runnable binary judge.

model is the Inspect model string the runtime calls through the execute/ seam; prompt_version labels prompt_template. Calibration is pinned to (model, prompt_version) plus fingerprint, so a change to any of them invalidates the record (certifies). level is "workflow" (whole trace) or "span" (one agent's input+output); the deterministic "process" judge lives in the level layer and needs no runtime. output_fields is the ordered field list of the forced-structured output; a non-empty critique MUST come strictly before verdict, enforced when the schema is built (build_verdict_schema).

temperature is sent to the model only when set; None sends no sampling parameter and leaves the provider's default. Newer models reject the parameter (Claude 4.7 and later on Bedrock), so it is opt-in. It is part of fingerprint, so changing it voids the judge's records like any prompt edit; give the changed judge a new prompt_version.

failure_mode and agent name what the judge measures (a judge file always sets the mode, and agent for a span judge). They are part of fingerprint when set, so pointing a certified judge at another mode or agent voids its records: its error rates were measured on the old one.

judge_id instance-attribute

judge_id

model instance-attribute

model

prompt_version instance-attribute

prompt_version

level instance-attribute

level

prompt_template instance-attribute

prompt_template

output_fields class-attribute instance-attribute

output_fields = ('critique', 'verdict')

temperature class-attribute instance-attribute

temperature = None

failure_mode class-attribute instance-attribute

failure_mode = None

agent class-attribute instance-attribute

agent = None

fingerprint property

fingerprint

A hash of everything that decides this judge's verdicts.

Covers the model, level, prompt template, output fields, sampling parameters and, when set, the failure mode and agent. Records stamp it, so editing the prompt in place without bumping prompt_version still voids the judge's calibration and bias records.

sampling

sampling()

The sampling parameters this judge sends: {"temperature": t}, or none.

Source code in src/calibrated_agent_evaluation/judges/executable.py
def sampling(self) -> dict[str, float]:
    """The sampling parameters this judge sends: ``{"temperature": t}``, or none."""
    return {} if self.temperature is None else {"temperature": float(self.temperature)}

JudgeProbeRunner

JudgeProbeRunner(judge, calib_record, model_backend)

Run each probe input through run_judge for one fixed, certified judge.

The battery passes (judge_model, prompt_version) with every call and pins them on the record it returns, so a call naming a different pair than this runner's judge is refused: the record would otherwise claim a battery result for a judge that was never probed.

Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
def __init__(
    self,
    judge: ExecutableJudge,
    calib_record: CalibrationRecord,
    model_backend: JudgeModelBackend,
) -> None:
    self._judge = judge
    self._calib = calib_record
    self._backend = model_backend

model_backend property

model_backend

The injected model seam (exposed so a caller can confirm which model is live).

live classmethod

live(judge, calib_record, *, timeout_seconds=60.0)

A runner over an InspectJudgeModel.

Building it makes no network call; the first run does, and needs the Inspect extra and the provider's credentials.

Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
@classmethod
def live(
    cls,
    judge: ExecutableJudge,
    calib_record: CalibrationRecord,
    *,
    timeout_seconds: float = 60.0,
) -> JudgeProbeRunner:
    """A runner over an :class:`~calibrated_agent_evaluation.execute.InspectJudgeModel`.

    Building it makes no network call; the first :meth:`run` does, and needs the Inspect extra
    and the provider's credentials.
    """
    from calibrated_agent_evaluation.execute import InspectJudgeModel

    return cls(
        judge,
        calib_record,
        InspectJudgeModel(
            judge.model, timeout_seconds=timeout_seconds, temperature=judge.temperature
        ),
    )

run

run(judge_model, prompt_version, probe_input)

Judge one probe input and return its PASS/FAIL verdict.

Raises ProbeExecutionError on a pinned-judge mismatch, a pairwise probe (a single-verdict judge cannot pick a candidate), or any model / verdict failure.

Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
def run(self, judge_model: str, prompt_version: str, probe_input: ProbeInput) -> ProbeVerdict:
    """Judge one probe input and return its PASS/FAIL verdict.

    Raises :class:`ProbeExecutionError` on a pinned-judge mismatch, a pairwise probe (a
    single-verdict judge cannot pick a candidate), or any model / verdict failure.
    """
    if (judge_model, prompt_version) != (self._judge.model, self._judge.prompt_version):
        raise ProbeExecutionError(
            f"battery asked for {judge_model}@{prompt_version} but this runner is pinned to "
            f"{self._judge.model}@{self._judge.prompt_version}; refusing to record one judge's "
            "probes under another's identity"
        )
    if probe_input.bias == "positional" or "better" in probe_input.payload:
        raise ProbeExecutionError(
            f"probe {probe_input.probe_id!r} is pairwise (a candidate pick); "
            "JudgeProbeRunner drives a single-verdict judge"
        )
    target = JudgeTarget(
        trace_id=probe_input.probe_id,
        span_id=None,
        content=render_probe_input(probe_input),
        sample=probe_input.repeat,
    )
    try:
        result = run_judge(
            self._judge, [target], calib_record=self._calib, model_backend=self._backend
        )
    except (ExecutionError, JudgeRuntimeError) as exc:
        raise ProbeExecutionError(
            f"judge probe {probe_input.probe_id!r} failed to execute: {exc}"
        ) from exc
    label = result.labels[0]
    return ProbeVerdict(label=label.outcome, detail=label.critique)

JudgeRunRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

One judge run's reproducibility record.

model/prompt_version are the frozen judge version; calib_id is the CalibrationRecord this run cleared. trace_ids and span_ids are aligned: span_ids[i] is the span scored on trace_ids[i] (None ⇒ trace-level). n_cached/n_computed count verdicts served from cache vs computed by a model call. A record with an empty calib_id is refused at construction; a run with no cleared calibration is not reproducible and must not be recorded as if it were.

Config:

  • frozen: True
  • extra: forbid

Fields:

Validators:

  • _check_invariants

run_id pydantic-field

run_id

judge_id pydantic-field

judge_id

model pydantic-field

model

prompt_version pydantic-field

prompt_version

calib_id pydantic-field

calib_id

level pydantic-field

level

trace_ids pydantic-field

trace_ids

span_ids pydantic-field

span_ids

n_cached pydantic-field

n_cached = 0

n_computed pydantic-field

n_computed = 0

schema_version pydantic-field

schema_version = JUDGES_SCHEMA_VERSION

judge_fingerprint pydantic-field

judge_fingerprint = None

ledger_head pydantic-field

ledger_head = None

ledger_entries pydantic-field

ledger_entries = None

JudgeRunResult dataclass

JudgeRunResult(labels, record)

The output of a judge run: the verdict labels + reproducibility record.

labels instance-attribute

labels

record instance-attribute

record

JudgeTarget dataclass

JudgeTarget(trace_id, span_id, content, sample=0)

One trace (or span) to judge.

content is the already-rendered text the judge examines; the caller renders the trace/span into the judge's prompt (the runtime stays decoupled from how a trace becomes a prompt). span_id is None for a whole-trace (workflow) judgment. sample numbers a repeat of the same target (0 is the first judgment), so a repeat is a fresh model draw.

trace_id instance-attribute

trace_id

span_id instance-attribute

span_id

content instance-attribute

content

sample class-attribute instance-attribute

sample = 0

JudgeDefinition dataclass

JudgeDefinition(judge_id, level, model, prompt_version)

The minimal level-carrying judge identity the level guards read.

level is what a judge attributes at; a "span" judge is calibrated over one agent's input+output, a "workflow" judge over whole traces; a workflow judge may not compute a per-agent rate. model/prompt_version mirror the calibration Judge protocol so a definition can be handed straight to the calibration gate.

judge_id instance-attribute

judge_id

level instance-attribute

level

model instance-attribute

model

prompt_version instance-attribute

prompt_version

ProcessVerdict dataclass

ProcessVerdict(
    skipped_agents,
    unexpected_agents,
    ordering_undetermined,
    rule_violations,
    undecided_rules,
    passed,
)

The deterministic process-judge output.

Computed by expected-vs-actual agent set-difference (no LLM call). rule_violations lists the rule ids whose deterministic clause failed; undecided_rules lists the non-deterministic rules, which nothing here can decide; reported as undecided, never a silent deterministic pass. passed is True only when nothing was skipped/unexpected and no rule was violated.

ordering_undetermined is True because a supervisor-fanout crew's pipeline order is not in a normalized trace (siblings under the supervisor; seq is emission order, not the logical route), so the deterministic judge does not assert an ordering violation; it records that order could not be verified. A mis-route is still caught by the set differences + the rule clauses.

skipped_agents instance-attribute

skipped_agents

unexpected_agents instance-attribute

unexpected_agents

ordering_undetermined instance-attribute

ordering_undetermined

rule_violations instance-attribute

rule_violations

undecided_rules instance-attribute

undecided_rules

passed instance-attribute

passed

assert_not_workflow_for_agent_rate

assert_not_workflow_for_agent_rate(judge)

Refuse a workflow-level judge used to compute a per-agent rate.

Raises JudgeLevelError if judge.level == "workflow". A workflow judge scores the whole trace and cannot attribute per agent; per-agent rates require a span-level judge.

Source code in src/calibrated_agent_evaluation/judges/levels.py
def assert_not_workflow_for_agent_rate(judge: JudgeDefinition) -> None:
    """Refuse a workflow-level judge used to compute a per-agent rate.

    Raises :class:`JudgeLevelError` if ``judge.level == "workflow"``. A workflow judge scores the
    whole trace and cannot attribute per agent; per-agent rates require a span-level judge.
    """
    if judge.level == "workflow":
        raise JudgeLevelError(
            f"judge {judge.judge_id!r} is level='workflow' and cannot compute a per-agent rate; "
            f"per-agent attribution requires a span-level judge."
        )

assert_span_gold_human

assert_span_gold_human(gold)

Refuse span-judge calibration gold that is not per-span human gold.

Raises SpanGoldError if any label lacks a span_id (it is trace-outcome gold, not a per-span judgment) or is not kind="human" (an LLM/code verdict cannot be gold). The fail hunted for a span judge is the span fail, so its gold is keyed on a span, not on outcome.

Source code in src/calibrated_agent_evaluation/judges/levels.py
def assert_span_gold_human(gold: Sequence[Label]) -> None:
    """Refuse span-judge calibration gold that is not per-span human gold.

    Raises :class:`SpanGoldError` if any label lacks a ``span_id`` (it is trace-outcome gold, not a
    per-span judgment) or is not ``kind="human"`` (an LLM/code verdict cannot be gold). The
    fail hunted for a span judge is the *span* fail, so its gold is keyed on a span, not on outcome.
    """
    for label in gold:
        if label.span_id is None:
            raise SpanGoldError(
                f"span-level calibration gold for trace {label.trace_id!r} has span_id=None; span "
                f"gold must be per-span human gold, not derived from the trace outcome. "
                f"Label the span itself."
            )
        if label.kind != "human":
            raise SpanGoldError(
                f"span-level gold for {label.span_id!r} is kind={label.kind!r}; calibration gold "
                f"must be human."
            )

render_probe_input

render_probe_input(probe_input)

Render one probe input as the judge's user prompt.

Shows the question, any context the bias varies (a sympathetic framing, an authoritative assertion, a claimed author), and the answer. Never shows the bias name, probe id, or control keys such as variant. An unrecognised payload key raises ProbeExecutionError rather than being shown, because it may be a control label the renderer does not know about.

Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
def render_probe_input(probe_input: ProbeInput) -> str:
    """Render one probe input as the judge's user prompt.

    Shows the question, any context the bias varies (a sympathetic framing, an authoritative
    assertion, a claimed author), and the answer. Never shows the bias name, probe id, or control
    keys such as ``variant``. An unrecognised payload key raises :class:`ProbeExecutionError`
    rather than being shown, because it may be a control label the renderer does not know about.
    """
    payload = probe_input.payload
    unknown = sorted(set(payload) - _CONTROL_KEYS - _CONTENT_KEYS)
    if unknown:
        raise ProbeExecutionError(
            f"probe {probe_input.probe_id!r} carries payload key(s) {unknown} the renderer does "
            "not know; refusing to show a possible control label to the judge"
        )
    if "answer" not in payload:
        raise ProbeExecutionError(f"probe {probe_input.probe_id!r} has no 'answer' to judge")
    parts: list[str] = []
    if "question" in payload:
        parts.append(str(payload["question"]))
    if "framing" in payload:
        parts.append(str(payload["framing"]))
    if "assertion" in payload:
        parts.append(str(payload["assertion"]))
    header = "Response"
    if "attributed_model" in payload:
        header = f"Response (written by {payload['attributed_model']})"
    parts.append(f"{header}:\n{payload['answer']}")
    return "\n\n".join(parts) + "\n"

process_judge

process_judge(trace, crew)

Deterministically judge a crew trace's structure against its scenario.

Resolves the scenario from trace.taxonomy_cell["scenario"] and reports the skipped agents (expected minus actual) and unexpected agents (actual minus expected) -- the order-independent set differences a normalized trace reliably supports. Each deterministic rule is checked by agent presence; each non-deterministic rule is reported as undecided. passed is True only when nothing was skipped or unexpected and no rule was violated.

Ordering is NOT asserted deterministically. A supervisor-fanout crew emits every specialist as a sibling of the supervisor, so the pipeline order (intake -> policy-checker -> ...) is not in the span tree; and the normalized seq is span emission order (the specialists start ~1 ms of each other), not the logical route -- the trace simply does not encode it (verified on the ClaimsCrew Strands Graph capture: no edge attributes, near-simultaneous starts). A seq-based ordering assertion therefore false-positives on real traces, so ordering_undetermined=True records that order could not be verified rather than claiming a violation. A mis-route is still caught by the set differences + the rule clauses.

Raises CrewDefinitionError if the scenario declares no expected process.

Source code in src/calibrated_agent_evaluation/judges/process.py
def process_judge(trace: Trace, crew: CrewDefinition) -> ProcessVerdict:
    """Deterministically judge a crew trace's structure against its scenario.

    Resolves the scenario from ``trace.taxonomy_cell["scenario"]`` and reports the skipped agents
    (expected minus actual) and unexpected agents (actual minus expected) -- the order-independent
    set differences a normalized trace reliably supports. Each deterministic rule is checked by
    agent presence; each non-deterministic rule is reported as undecided. ``passed`` is True only
    when nothing was skipped or unexpected and no rule was violated.

    **Ordering is NOT asserted deterministically.** A supervisor-fanout crew emits every specialist
    as a sibling of the supervisor, so the pipeline order (intake -> policy-checker -> ...) is not
    in the span tree; and the normalized ``seq`` is span *emission* order (the specialists start
    ~1 ms of each other), not the logical route -- the trace simply does not encode it (verified on
    the ClaimsCrew Strands ``Graph`` capture: no edge attributes, near-simultaneous starts). A
    ``seq``-based ordering assertion therefore false-positives on real traces, so
    ``ordering_undetermined=True`` records that order could not be verified rather than claiming a
    violation. A mis-route is still caught by the set differences + the rule clauses.

    Raises :class:`~calibrated_agent_evaluation.attribution.errors.CrewDefinitionError` if the
    scenario declares no expected process.
    """
    scenario = trace.taxonomy_cell.get(_SCENARIO_KEY, "")
    expected = require_expected_process(crew, scenario)

    actual_order = [s.agent for s in sorted(trace.spans, key=lambda s: s.seq)]
    actual_set = set(actual_order)
    expected_set = set(expected.expected_agents)

    skipped = tuple(a for a in expected.expected_agents if a not in actual_set)
    unexpected = tuple(a for a in actual_order if a not in expected_set)

    rule_violations: list[str] = []
    undecided_rules: list[str] = []
    for rule in crew.rules:
        _evaluate_rule(rule, actual_set, rule_violations, undecided_rules)

    passed = not skipped and not unexpected and not rule_violations
    return ProcessVerdict(
        skipped_agents=skipped,
        unexpected_agents=unexpected,
        ordering_undetermined=True,
        rule_violations=tuple(rule_violations),
        undecided_rules=tuple(undecided_rules),
        passed=passed,
    )

dump_run_records

dump_run_records(records, path, fmt='csv')

Write records to judge_run_record.{csv,json} under path.

Parameters:

Name Type Description Default
records Sequence[JudgeRunRecord]

The judge run records to persist.

required
path str | Path

The directory to write into.

required
fmt str

Which file a later load_run_records reads back (both are always written).

'csv'

Raises:

Type Description
JudgeRuntimeError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/judges/run_record.py
def dump_run_records(records: Sequence[JudgeRunRecord], path: str | Path, fmt: str = "csv") -> None:
    """Write ``records`` to ``judge_run_record.{csv,json}`` under ``path``.

    Args:
        records: The judge run records to persist.
        path: The directory to write into.
        fmt: Which file a later :func:`load_run_records` reads back (both are always written).

    Raises:
        JudgeRuntimeError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

load_run_records

load_run_records(path, fmt='csv')

Read judge run records back from path; a missing file is an empty store.

Parameters:

Name Type Description Default
path str | Path

The directory dump_run_records wrote.

required
fmt str

"csv" or "json".

'csv'

Returns:

Type Description
list[JudgeRunRecord]

The stored records.

Raises:

Type Description
JudgeRuntimeError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/judges/run_record.py
def load_run_records(path: str | Path, fmt: str = "csv") -> list[JudgeRunRecord]:
    """Read judge run records back from ``path``; a missing file is an empty store.

    Args:
        path: The directory :func:`dump_run_records` wrote.
        fmt: ``"csv"`` or ``"json"``.

    Returns:
        The stored records.

    Raises:
        JudgeRuntimeError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)

locked_test_set

locked_test_set(
    judge, gold, *, set_id, model_backend, max_retries=2
)

Run judge over its locked test and pair each verdict with its human gold.

The one judge run that needs no calibration record, because it is the run calibration is made from: run_judge refuses an uncertified judge, so it cannot produce the verdicts certify consumes. The verdicts here are not scores. Nothing is written and no verdict is returned on its own; they come back only inside the locked_test CalibrationSet, whose one use is certify.

gold pairs each JudgeTarget with the human label it is graded against, keyed on the same (trace_id, span_id). Refused before any model call: an empty set, a gold label that is not kind="human", a target that is not its gold's, a target graded more than once (DuplicateGoldError), and for a span judge any gold without a span_id (calibrate at the level you attribute at).

Source code in src/calibrated_agent_evaluation/judges/runtime.py
def locked_test_set(
    judge: ExecutableJudge,
    gold: Sequence[tuple[JudgeTarget, Label]],
    *,
    set_id: str,
    model_backend: JudgeModelBackend,
    max_retries: int = 2,
) -> CalibrationSet:
    """Run ``judge`` over its locked test and pair each verdict with its human gold.

    The one judge run that needs no calibration record, because it is the run calibration is made
    from: :func:`run_judge` refuses an uncertified judge, so it cannot produce the verdicts
    :func:`calibrated_agent_evaluation.calibrate.certify` consumes. The verdicts here are not
    scores. Nothing is written and no verdict is returned on its own; they come back only inside the
    ``locked_test`` :class:`CalibrationSet`, whose one use is ``certify``.

    ``gold`` pairs each :class:`JudgeTarget` with the human label it is graded against, keyed on
    the same ``(trace_id, span_id)``. Refused before any model call: an empty set, a gold label
    that is not ``kind="human"``, a target that is not its gold's, a target graded more than once
    (:class:`~calibrated_agent_evaluation.calibrate.errors.DuplicateGoldError`), and for a span
    judge any gold without a ``span_id`` (calibrate at the level you attribute at).
    """
    if not gold:
        raise UnpairedObservationError(
            f"judge {judge.judge_id!r}: no gold to calibrate against; a locked test needs at least "
            "one human-labeled target"
        )
    labels = [g for _, g in gold]
    if judge.level == "span":
        assert_span_gold_human(labels)
    for target, label in gold:
        if label.kind != "human":
            raise UnratifiedGoldError(
                f"gold label for {label.trace_id} has kind={label.kind!r}, not 'human'; a judge is "
                "certified only against human gold"
            )
        if (target.trace_id, target.span_id) != (label.trace_id, label.span_id):
            raise UnpairedObservationError(
                f"target {(target.trace_id, target.span_id)} is graded against gold "
                f"{(label.trace_id, label.span_id)}; a target and its gold must name the same "
                "trace and span"
            )
    refuse_duplicate_targets((label.trace_id, label.span_id) for label in labels)
    schema = build_verdict_schema(judge)
    pairs = tuple(
        LabelPair(gold=label, verdict=_judge_one(judge, target, schema, model_backend, max_retries))
        for target, label in gold
    )
    return CalibrationSet(pairs=pairs, split="locked_test", set_id=set_id)

run_judge

run_judge(
    judge,
    targets,
    *,
    calib_record,
    model_backend,
    store=None,
    cache=None,
    max_retries=2,
    per_agent_rate=False,
    run_id=None,
)

Run judge over targets and return the verdict labels + run record.

Refuses (UncalibratedScoreError) before any model call if calib_record does not certify judge at its current version. If per_agent_rate is requested, refuses a workflow-level judge (JudgeLevelError). For each target: a cache hit is returned without a model call; a miss forces a structured verdict via model_backend with up to max_retries retries on a transient model failure, maps it to a validated Label, caches it, and (if store is given) writes it back via put_score. A cache hit is NOT written back (it is already stored). A backend BackendError on write-back propagates un-swallowed.

Parameters:

Name Type Description Default
judge ExecutableJudge

The judge to run.

required
targets list[JudgeTarget]

The traces or spans to score.

required
calib_record CalibrationRecord

The calibration record that certifies judge.

required
model_backend JudgeModelBackend

The model the judge's prompts are sent to.

required
store TraceStoreBackend | None

Where to write the verdicts back; nothing is written when omitted.

None
cache VerdictCache | None

Verdicts already computed, reused without a model call.

None
max_retries int

Retries on a transient model failure.

2
per_agent_rate bool

The verdicts will feed a per-agent rate.

False
run_id str | None

The run id; derived from the inputs when omitted.

None

Returns:

Type Description
JudgeRunResult

The verdict labels and the run record.

Raises:

Type Description
UncalibratedScoreError

calib_record doesn't certify judge at its current version.

JudgeLevelError

A per-agent rate was requested from a workflow-level judge.

JudgeRuntimeError

The model failed after retries, or returned an invalid verdict.

BackendError

The store is unreachable on write-back.

Source code in src/calibrated_agent_evaluation/judges/runtime.py
def run_judge(
    judge: ExecutableJudge,
    targets: list[JudgeTarget],
    *,
    calib_record: CalibrationRecord,
    model_backend: JudgeModelBackend,
    store: TraceStoreBackend | None = None,
    cache: VerdictCache | None = None,
    max_retries: int = 2,
    per_agent_rate: bool = False,
    run_id: str | None = None,
) -> JudgeRunResult:
    """Run ``judge`` over ``targets`` and return the verdict labels + run record.

    Refuses (``UncalibratedScoreError``) before any model call if ``calib_record`` does not certify
    ``judge`` at its current version. If ``per_agent_rate`` is requested, refuses a
    workflow-level judge (``JudgeLevelError``). For each target: a cache hit is returned
    without a model call; a miss forces a structured verdict via ``model_backend`` with up
    to ``max_retries`` retries on a transient model failure, maps it to a validated
    ``Label``, caches it, and (if ``store`` is given) writes it back via ``put_score``.
    A cache hit is NOT written back (it is already stored). A backend ``BackendError`` on write-back
    propagates un-swallowed.

    Args:
        judge: The judge to run.
        targets: The traces or spans to score.
        calib_record: The calibration record that certifies ``judge``.
        model_backend: The model the judge's prompts are sent to.
        store: Where to write the verdicts back; nothing is written when omitted.
        cache: Verdicts already computed, reused without a model call.
        max_retries: Retries on a transient model failure.
        per_agent_rate: The verdicts will feed a per-agent rate.
        run_id: The run id; derived from the inputs when omitted.

    Returns:
        The verdict labels and the run record.

    Raises:
        UncalibratedScoreError: ``calib_record`` doesn't certify ``judge`` at its current
            version.
        JudgeLevelError: A per-agent rate was requested from a workflow-level judge.
        JudgeRuntimeError: The model failed after retries, or returned an invalid verdict.
        BackendError: The store is unreachable on write-back.
    """
    # Read the certification (never recompute). Refuses before any
    # model call, so an uncertified judge cannot spend a token or write a Label.
    enforce_score_gate(calib_record, judge)

    # A workflow judge cannot attribute per agent. Reuse the level guard via the
    # level-carrying JudgeDefinition it reads (single source of the refusal rule).
    if per_agent_rate:
        assert_not_workflow_for_agent_rate(
            JudgeDefinition(
                judge_id=judge.judge_id,
                level=judge.level,
                model=judge.model,
                prompt_version=judge.prompt_version,
            )
        )

    schema = build_verdict_schema(judge)
    labels: list[Label] = []
    n_cached = 0
    n_computed = 0

    for target in targets:
        cached = (
            cache.get(judge, target.trace_id, target.span_id, target.sample)
            if cache is not None
            else None
        )
        if cached is not None:
            labels.append(cached)
            n_cached += 1
        else:
            label = _judge_one(judge, target, schema, model_backend, max_retries)
            if cache is not None:
                cache.put(judge, label, target.sample)
            labels.append(label)
            n_computed += 1
        # Write-back per COMPUTED verdict only. A cache hit was already written durably on the run
        # that computed it, so re-writing it added a second row to an append-only score store: a
        # warm-cache re-run doubled every label it touched and moved every rate computed over them.
        # Read-side dedup (`serde.dedup_labels`) makes that harmless, but not writing is the honest
        # form -- "this verdict is already stored" is exactly what a cache hit means.
        #
        # A BackendError (e.g. the Langfuse adapter's synchronous reachability guard firing)
        # propagates un-swallowed: the runtime does NOT catch or wrap it, so an unreachable backend
        # is never reported as a durable write.
        if store is not None and cached is None:
            store.put_score(labels[-1])

    record = JudgeRunRecord(
        run_id=run_id or _derive_run_id(judge, targets),
        judge_id=judge.judge_id,
        model=judge.model,
        prompt_version=judge.prompt_version,
        calib_id=calib_record.calib_id,
        level=judge.level,
        trace_ids=tuple(t.trace_id for t in targets),
        span_ids=tuple(t.span_id for t in targets),
        n_cached=n_cached,
        n_computed=n_computed,
        judge_fingerprint=judge.fingerprint,
    )
    return JudgeRunResult(labels=tuple(labels), record=record)

ancestor_path

ancestor_path(trace, span)

The spans from the root down to span's parent, in that order.

Raises JudgeLevelError when span is not in trace, or a parent link names a span the trace does not hold or loops back on itself: the path would be partial, and a span judged on part of its context is a different judgment.

Source code in src/calibrated_agent_evaluation/judges/targets.py
def ancestor_path(trace: Trace, span: Span) -> list[Span]:
    """The spans from the root down to ``span``'s parent, in that order.

    Raises :class:`JudgeLevelError` when ``span`` is not in ``trace``, or a parent link names a
    span the trace does not hold or loops back on itself: the path would be partial, and a span
    judged on part of its context is a different judgment.
    """
    by_id = {s.span_id: s for s in trace.spans}
    if by_id.get(span.span_id) != span:
        raise JudgeLevelError(f"span {span.span_id!r} is not in trace {trace.trace_id!r}")
    path: list[Span] = []
    seen = {span.span_id}
    parent = span.parent
    while parent is not None:
        node = by_id.get(parent)
        if node is None:
            raise JudgeLevelError(
                f"span {span.span_id!r} in trace {trace.trace_id!r} has an ancestor {parent!r} the "
                "trace does not hold; refusing to judge it on a partial path"
            )
        if node.span_id in seen:
            raise JudgeLevelError(
                f"span {span.span_id!r} in trace {trace.trace_id!r} has a parent cycle at "
                f"{node.span_id!r}"
            )
        seen.add(node.span_id)
        path.append(node)
        parent = node.parent
    return list(reversed(path))

render_span

render_span(trace, span)

The judge-facing text for span: the request, its ancestor path, then the span itself.

Each ancestor shows what it received and the tools it called, never what it emitted: an orchestrator's output is written after its children ran and can state the judged span's result, which would hand the judge the answer.

Source code in src/calibrated_agent_evaluation/judges/targets.py
def render_span(trace: Trace, span: Span) -> str:
    """The judge-facing text for ``span``: the request, its ancestor path, then the span itself.

    Each ancestor shows what it received and the tools it called, never what it emitted:
    an orchestrator's output is written after its children ran and can state the
    judged span's result, which would hand the judge the answer.
    """
    lines = [f"Customer request: {_show(trace.input)}"]
    path = ancestor_path(trace, span)
    if path:
        lines.append("Steps leading to the step being judged, from the root (outputs withheld):")
        for depth, node in enumerate(path, start=1):
            lines.append(f"  [{depth}] agent {node.agent} ({node.role})")
            lines.extend(_step_lines(node, "      ", with_output=False))
    else:
        lines.append("The step being judged is the root of the run.")
    lines.append(f"Step being judged: agent {span.agent} ({span.role})")
    lines.extend(_step_lines(span, "  "))
    return "\n".join(lines) + "\n"

render_trace

render_trace(trace)

The judge-facing text for the whole of trace.

Parameters:

Name Type Description Default
trace Trace

The trace to render.

required

Returns:

Type Description
str

The trace's input, each span's agent, tool calls and output, and the final output, as text.

Source code in src/calibrated_agent_evaluation/judges/targets.py
def render_trace(trace: Trace) -> str:
    """The judge-facing text for the whole of ``trace``.

    Args:
        trace: The trace to render.

    Returns:
        The trace's input, each span's agent, tool calls and output, and the final output, as text.
    """
    tool_lines = [
        _tool_line("  ", tc.name, tc.args, tc.result)
        for span in trace.spans
        for tc in span.tool_calls
    ]
    tools = "\n".join(tool_lines) if tool_lines else "  (none)"
    return (
        f"Customer request: {_show(trace.input)}\n"
        f"Agent tool calls:\n{tools}\n"
        f"Agent final answer: {_show(trace.output)}\n"
    )

span_target

span_target(trace, span)

A span-level JudgeTarget for span, keyed on (trace_id, span_id).

Source code in src/calibrated_agent_evaluation/judges/targets.py
def span_target(trace: Trace, span: Span) -> JudgeTarget:
    """A span-level :class:`JudgeTarget` for ``span``, keyed on ``(trace_id, span_id)``."""
    return JudgeTarget(
        trace_id=trace.trace_id, span_id=span.span_id, content=render_span(trace, span)
    )

workflow_target

workflow_target(trace)

A whole-trace JudgeTarget for trace.

Source code in src/calibrated_agent_evaluation/judges/targets.py
def workflow_target(trace: Trace) -> JudgeTarget:
    """A whole-trace :class:`JudgeTarget` for ``trace``."""
    return JudgeTarget(trace_id=trace.trace_id, span_id=None, content=render_trace(trace))

reply_to_label

reply_to_label(reply, judge, *, trace_id, span_id)

Build a validated kind="llm-judge" Label from reply.

Raises JudgeRuntimeError if the structured payload is missing critique/verdict, has an empty/whitespace critique, or a non-binary verdict; the runtime refuses to persist a malformed verdict. (The Label dataclass + schema.validate also reject an empty critique, but we raise the typed judge error here so the boundary is the judge family.)

Source code in src/calibrated_agent_evaluation/judges/verdict.py
def reply_to_label(
    reply: StructuredReply,
    judge: ExecutableJudge,
    *,
    trace_id: str,
    span_id: str | None,
) -> Label:
    """Build a validated ``kind="llm-judge"`` ``Label`` from ``reply``.

    Raises :class:`JudgeRuntimeError` if the structured payload is missing ``critique``/``verdict``,
    has an empty/whitespace critique, or a non-binary verdict; the runtime refuses to
    persist a malformed verdict. (The ``Label`` dataclass + ``schema.validate`` also reject an empty
    critique, but we raise the typed judge error here so the boundary is the judge family.)
    """
    fields = reply.fields
    critique = fields.get("critique")
    verdict = fields.get("verdict")

    if not isinstance(critique, str) or not critique.strip():
        raise JudgeRuntimeError(
            f"judge {judge.judge_id!r} on trace {trace_id!r}: structured reply had an empty or "
            f"non-string 'critique' (via={reply.via}); refusing to persist a verdict"
        )
    if not isinstance(verdict, str) or verdict.lower() not in _VERDICT_TO_OUTCOME:
        raise JudgeRuntimeError(
            f"judge {judge.judge_id!r} on trace {trace_id!r}: structured reply 'verdict' was "
            f"{verdict!r}, not one of {sorted(_VERDICT_TO_OUTCOME)} (via={reply.via}); "
            "refusing to persist a non-binary verdict"
        )

    return Label(
        trace_id=trace_id,
        span_id=span_id,
        kind="llm-judge",
        outcome=_VERDICT_TO_OUTCOME[verdict.lower()],  # type: ignore[arg-type]
        critique=critique,
        rater_id=verdict_rater_id(judge),
    )

verdict_rater_id

verdict_rater_id(judge)

The rater_id a verdict from judge carries: "<judge_id>@<fingerprint>".

A stored verdict says which judge, as it is now, produced it, so a rate is corrected only with the calibration of the judge that made the verdicts. The fingerprint is 16 hex digits, so the id is read back by splitting on the last @.

Source code in src/calibrated_agent_evaluation/judges/verdict.py
def verdict_rater_id(judge: _Fingerprinted) -> str:
    """The ``rater_id`` a verdict from ``judge`` carries: ``"<judge_id>@<fingerprint>"``.

    A stored verdict says which judge, as it is now, produced it, so a rate is corrected only with
    the calibration of the judge that made the verdicts. The fingerprint is 16 hex digits, so the
    id is read back by splitting on the last ``@``.
    """
    return f"{judge.judge_id}@{judge.fingerprint}"

build_verdict_schema

build_verdict_schema(judge)

Return the JSON schema for judge's forced-structured verdict.

Guarantees the critique-before-verdict ordering by reusing critique_before_verdict: a judge whose output_fields place verdict before critique (or omit either) is rejected with a JudgeRuntimeError rather than emitting a schema; a mis-ordered judge must never reach the model. The emitted schema requires a non-empty critique (minLength: 1) and a binary verdict enum, so a blank critique or non-binary verdict fails schema-side too.

Source code in src/calibrated_agent_evaluation/judges/verdict_schema.py
def build_verdict_schema(judge: ExecutableJudge) -> dict[str, Any]:
    """Return the JSON schema for ``judge``'s forced-structured verdict.

    Guarantees the critique-before-verdict ordering by reusing ``critique_before_verdict``: a judge
    whose ``output_fields`` place ``verdict`` before ``critique`` (or omit either) is rejected with
    a :class:`JudgeRuntimeError` rather than emitting a schema; a mis-ordered judge must never
    reach the model. The emitted schema requires a non-empty ``critique`` (``minLength: 1``) and a
    binary ``verdict`` enum, so a blank critique or non-binary verdict fails schema-side too.
    """
    # Reuse the shared predicate as the single source of the ordering rule. axes=(); this
    # shape carries only the output_fields the predicate reads; the axis/lint checks live upstream.
    lint_shape = LintJudgeDefinition(
        judge_id=judge.judge_id, axes=(), output_fields=judge.output_fields
    )
    if not critique_before_verdict(lint_shape):
        raise JudgeRuntimeError(
            f"judge {judge.judge_id!r} output_fields {judge.output_fields!r} must place a "
            "non-empty 'critique' strictly before 'verdict'; refusing to build "
            "a verdict schema"
        )
    properties: dict[str, Any] = {
        "critique": {
            "type": "string",
            "minLength": 1,
            "description": (
                "The judge's reasoning, written before deciding the verdict "
                "(P-003)."  # allow-id: model-facing text that recorded cassettes key on
            ),
        },
        "verdict": {
            "type": "string",
            "enum": _VERDICT_VALUES,
            "description": "The binary verdict: 'pass' or 'fail'.",
        },
    }
    return {
        "type": "object",
        "properties": properties,
        "required": ["critique", "verdict"],
        "additionalProperties": False,
    }