calibrated_agent_evaluation.judges¶
calibrated_agent_evaluation.judges ¶
Judge-level layer.
The three judge levels a crew is judged at -- workflow (whole trace), span (one agent's input+output), process (the span tree) -- with the "calibrate at the level you attribute at" discipline enforced, plus the deterministic process judge (expected-vs-actual agent set-difference + ordering, no LLM call).
Pure methodology plane: reads normalized Trace/Span and the crew Define artifact; no vendor
type appears here. It does NOT execute LLM workflow/span judges -- that batch fan-out
runtime is the judge-execution layer; this package owns the level model + the deterministic
process judge only.
Public API re-exports from submodules.
VerdictCache ¶
An in-memory, round-trippable map from (judge version, target) to a cached Label.
Not a context manager: it owns no connection/handle (pure in-memory dict + explicit
dump/load), so the conventions' context-manager rule does not apply.
Source code in src/calibrated_agent_evaluation/judges/cache.py
get ¶
Return the cached Label for this judge version + target, or None on a miss.
Returns None when the judge's model/prompt_version/content differ from any
stored entry, so a re-versioned judge never serves a stale verdict, and for a repeat
number not judged yet.
Source code in src/calibrated_agent_evaluation/judges/cache.py
put ¶
Store label keyed on this judge version, its (trace_id, span_id) and repeat.
Source code in src/calibrated_agent_evaluation/judges/cache.py
dump ¶
Write the cache to verdict_cache.json under path (lossless round-trip).
Raises JudgeRuntimeError on any filesystem failure (no raw OSError escapes).
Source code in src/calibrated_agent_evaluation/judges/cache.py
load
classmethod
¶
Read a cache back from path; an absent file yields an empty cache.
Version-checks before returning: a stored schema_version != running raises
JudgeRuntimeError (no coercion). Wraps filesystem/parse/malformed failures.
Source code in src/calibrated_agent_evaluation/judges/cache.py
JudgeLevelError ¶
Bases: CalibratedAgentEvaluationDataError
A workflow-level judge was used for a per-agent rate, which it can't attribute.
JudgeRuntimeError ¶
Bases: CalibratedAgentEvaluationDataError
A failure in the judge-execution runtime.
Covers a model call that failed after exhausting retries, a reply that carried neither a
schema-valid structured output nor a valid forced-tool-call payload, and a structured payload
that violated the verdict schema (empty critique / non-binary verdict). Wraps the
underlying vendor/execute exception via chaining (raise JudgeRuntimeError(...) from exc)
so no raw inspect_ai/boto3/stdlib exception escapes the judges/ boundary
(conventions: "wrap external errors at the boundary"). The runtime never fabricates a verdict:
on any of these it raises rather than persisting a Label.
SpanGoldError ¶
Bases: CalibratedAgentEvaluationDataError
A span-level judge was calibrated on gold derived from the trace outcome, not the span.
ExecutableJudge
dataclass
¶
ExecutableJudge(
judge_id,
model,
prompt_version,
level,
prompt_template,
output_fields=("critique", "verdict"),
temperature=None,
failure_mode=None,
agent=None,
)
A fully runnable binary judge.
model is the Inspect model string the runtime calls through the execute/ seam;
prompt_version labels prompt_template. Calibration is pinned to (model,
prompt_version) plus fingerprint, so a change to any of them invalidates the record
(certifies). level is "workflow" (whole trace) or "span" (one agent's
input+output); the deterministic "process" judge lives in the level layer and needs no
runtime. output_fields is the ordered field list of the forced-structured output; a
non-empty critique MUST come strictly before verdict, enforced when the schema is built
(build_verdict_schema).
temperature is sent to the model only when set; None sends no sampling parameter and
leaves the provider's default. Newer models reject the parameter (Claude 4.7 and later on
Bedrock), so it is opt-in. It is part of fingerprint, so changing it voids the
judge's records like any prompt edit; give the changed judge a new prompt_version.
failure_mode and agent name what the judge measures (a judge file always sets the
mode, and agent for a span judge). They are part of fingerprint when set, so
pointing a certified judge at another mode or agent voids its records: its error rates were
measured on the old one.
fingerprint
property
¶
A hash of everything that decides this judge's verdicts.
Covers the model, level, prompt template, output fields, sampling parameters and, when
set, the failure mode and agent. Records stamp it, so editing the prompt in place without
bumping prompt_version still voids the judge's calibration and bias records.
sampling ¶
The sampling parameters this judge sends: {"temperature": t}, or none.
JudgeProbeRunner ¶
Run each probe input through run_judge for one fixed, certified judge.
The battery passes (judge_model, prompt_version) with every call and pins them on the
record it returns, so a call naming a different pair than this runner's judge is refused: the
record would otherwise claim a battery result for a judge that was never probed.
Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
model_backend
property
¶
The injected model seam (exposed so a caller can confirm which model is live).
live
classmethod
¶
A runner over an InspectJudgeModel.
Building it makes no network call; the first run does, and needs the Inspect extra
and the provider's credentials.
Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
run ¶
Judge one probe input and return its PASS/FAIL verdict.
Raises ProbeExecutionError on a pinned-judge mismatch, a pairwise probe (a
single-verdict judge cannot pick a candidate), or any model / verdict failure.
Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
JudgeRunRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One judge run's reproducibility record.
model/prompt_version are the frozen judge version; calib_id is the
CalibrationRecord this run cleared.
trace_ids and span_ids are aligned: span_ids[i] is the span scored on
trace_ids[i] (None ⇒ trace-level). n_cached/n_computed count verdicts served
from cache vs computed by a model call. A record with an empty calib_id is refused at
construction; a run with no cleared calibration is not reproducible and must not be recorded as
if it were.
Config:
frozen:Trueextra:forbid
Fields:
-
run_id(str) -
judge_id(str) -
model(str) -
prompt_version(str) -
calib_id(str) -
level(JudgeLevel | Literal['session']) -
trace_ids(tuple[str, ...]) -
span_ids(tuple[str | None, ...]) -
n_cached(int) -
n_computed(int) -
schema_version(str) -
judge_fingerprint(str | None) -
ledger_head(str | None) -
ledger_entries(int | None)
Validators:
-
_check_invariants
JudgeRunResult
dataclass
¶
JudgeTarget
dataclass
¶
One trace (or span) to judge.
content is the already-rendered text the judge examines; the caller renders the trace/span
into the judge's prompt (the runtime stays decoupled from how a trace becomes a prompt).
span_id is None for a whole-trace (workflow) judgment. sample numbers a repeat of
the same target (0 is the first judgment), so a repeat is a fresh model draw.
JudgeDefinition
dataclass
¶
The minimal level-carrying judge identity the level guards read.
level is what a judge attributes at; a "span" judge is calibrated over one agent's
input+output, a "workflow" judge over whole traces; a workflow judge may not compute a
per-agent rate. model/prompt_version mirror the calibration Judge protocol so a
definition can be handed straight to the calibration gate.
ProcessVerdict
dataclass
¶
ProcessVerdict(
skipped_agents,
unexpected_agents,
ordering_undetermined,
rule_violations,
undecided_rules,
passed,
)
The deterministic process-judge output.
Computed by expected-vs-actual agent set-difference (no LLM call). rule_violations lists the
rule ids whose deterministic clause failed; undecided_rules lists the non-deterministic
rules, which nothing here can decide; reported as undecided, never a silent deterministic pass.
passed is True only when nothing was skipped/unexpected and no rule was violated.
ordering_undetermined is True because a supervisor-fanout crew's pipeline order is not in a
normalized trace (siblings under the supervisor; seq is emission order, not the logical
route), so the deterministic judge does not assert an ordering violation; it records that order
could not be verified. A mis-route is still caught by the set differences + the rule clauses.
assert_not_workflow_for_agent_rate ¶
Refuse a workflow-level judge used to compute a per-agent rate.
Raises JudgeLevelError if judge.level == "workflow". A workflow judge scores the
whole trace and cannot attribute per agent; per-agent rates require a span-level judge.
Source code in src/calibrated_agent_evaluation/judges/levels.py
assert_span_gold_human ¶
Refuse span-judge calibration gold that is not per-span human gold.
Raises SpanGoldError if any label lacks a span_id (it is trace-outcome gold, not a
per-span judgment) or is not kind="human" (an LLM/code verdict cannot be gold). The
fail hunted for a span judge is the span fail, so its gold is keyed on a span, not on outcome.
Source code in src/calibrated_agent_evaluation/judges/levels.py
render_probe_input ¶
Render one probe input as the judge's user prompt.
Shows the question, any context the bias varies (a sympathetic framing, an authoritative
assertion, a claimed author), and the answer. Never shows the bias name, probe id, or control
keys such as variant. An unrecognised payload key raises ProbeExecutionError
rather than being shown, because it may be a control label the renderer does not know about.
Source code in src/calibrated_agent_evaluation/judges/probe_runner.py
process_judge ¶
Deterministically judge a crew trace's structure against its scenario.
Resolves the scenario from trace.taxonomy_cell["scenario"] and reports the skipped agents
(expected minus actual) and unexpected agents (actual minus expected) -- the order-independent
set differences a normalized trace reliably supports. Each deterministic rule is checked by
agent presence; each non-deterministic rule is reported as undecided. passed is True only
when nothing was skipped or unexpected and no rule was violated.
Ordering is NOT asserted deterministically. A supervisor-fanout crew emits every specialist
as a sibling of the supervisor, so the pipeline order (intake -> policy-checker -> ...) is not
in the span tree; and the normalized seq is span emission order (the specialists start
~1 ms of each other), not the logical route -- the trace simply does not encode it (verified on
the ClaimsCrew Strands Graph capture: no edge attributes, near-simultaneous starts). A
seq-based ordering assertion therefore false-positives on real traces, so
ordering_undetermined=True records that order could not be verified rather than claiming a
violation. A mis-route is still caught by the set differences + the rule clauses.
Raises CrewDefinitionError if the
scenario declares no expected process.
Source code in src/calibrated_agent_evaluation/judges/process.py
dump_run_records ¶
Write records to judge_run_record.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
records
|
Sequence[JudgeRunRecord]
|
The judge run records to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
JudgeRuntimeError
|
A bad |
Source code in src/calibrated_agent_evaluation/judges/run_record.py
load_run_records ¶
Read judge run records back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[JudgeRunRecord]
|
The stored records. |
Raises:
| Type | Description |
|---|---|
JudgeRuntimeError
|
A schema-version mismatch, or any read, parse or validation failure. |
Source code in src/calibrated_agent_evaluation/judges/run_record.py
locked_test_set ¶
Run judge over its locked test and pair each verdict with its human gold.
The one judge run that needs no calibration record, because it is the run calibration is made
from: run_judge refuses an uncertified judge, so it cannot produce the verdicts
certify consumes. The verdicts here are not
scores. Nothing is written and no verdict is returned on its own; they come back only inside the
locked_test CalibrationSet, whose one use is certify.
gold pairs each JudgeTarget with the human label it is graded against, keyed on
the same (trace_id, span_id). Refused before any model call: an empty set, a gold label
that is not kind="human", a target that is not its gold's, a target graded more than once
(DuplicateGoldError), and for a span
judge any gold without a span_id (calibrate at the level you attribute at).
Source code in src/calibrated_agent_evaluation/judges/runtime.py
run_judge ¶
run_judge(
judge,
targets,
*,
calib_record,
model_backend,
store=None,
cache=None,
max_retries=2,
per_agent_rate=False,
run_id=None,
)
Run judge over targets and return the verdict labels + run record.
Refuses (UncalibratedScoreError) before any model call if calib_record does not certify
judge at its current version. If per_agent_rate is requested, refuses a
workflow-level judge (JudgeLevelError). For each target: a cache hit is returned
without a model call; a miss forces a structured verdict via model_backend with up
to max_retries retries on a transient model failure, maps it to a validated
Label, caches it, and (if store is given) writes it back via put_score.
A cache hit is NOT written back (it is already stored). A backend BackendError on write-back
propagates un-swallowed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
judge
|
ExecutableJudge
|
The judge to run. |
required |
targets
|
list[JudgeTarget]
|
The traces or spans to score. |
required |
calib_record
|
CalibrationRecord
|
The calibration record that certifies |
required |
model_backend
|
JudgeModelBackend
|
The model the judge's prompts are sent to. |
required |
store
|
TraceStoreBackend | None
|
Where to write the verdicts back; nothing is written when omitted. |
None
|
cache
|
VerdictCache | None
|
Verdicts already computed, reused without a model call. |
None
|
max_retries
|
int
|
Retries on a transient model failure. |
2
|
per_agent_rate
|
bool
|
The verdicts will feed a per-agent rate. |
False
|
run_id
|
str | None
|
The run id; derived from the inputs when omitted. |
None
|
Returns:
| Type | Description |
|---|---|
JudgeRunResult
|
The verdict labels and the run record. |
Raises:
| Type | Description |
|---|---|
UncalibratedScoreError
|
|
JudgeLevelError
|
A per-agent rate was requested from a workflow-level judge. |
JudgeRuntimeError
|
The model failed after retries, or returned an invalid verdict. |
BackendError
|
The store is unreachable on write-back. |
Source code in src/calibrated_agent_evaluation/judges/runtime.py
91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 | |
ancestor_path ¶
The spans from the root down to span's parent, in that order.
Raises JudgeLevelError when span is not in trace, or a parent link names a
span the trace does not hold or loops back on itself: the path would be partial, and a span
judged on part of its context is a different judgment.
Source code in src/calibrated_agent_evaluation/judges/targets.py
render_span ¶
The judge-facing text for span: the request, its ancestor path, then the span itself.
Each ancestor shows what it received and the tools it called, never what it emitted: an orchestrator's output is written after its children ran and can state the judged span's result, which would hand the judge the answer.
Source code in src/calibrated_agent_evaluation/judges/targets.py
render_trace ¶
The judge-facing text for the whole of trace.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
trace
|
Trace
|
The trace to render. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The trace's input, each span's agent, tool calls and output, and the final output, as text. |
Source code in src/calibrated_agent_evaluation/judges/targets.py
span_target ¶
A span-level JudgeTarget for span, keyed on (trace_id, span_id).
Source code in src/calibrated_agent_evaluation/judges/targets.py
workflow_target ¶
A whole-trace JudgeTarget for trace.
reply_to_label ¶
Build a validated kind="llm-judge" Label from reply.
Raises JudgeRuntimeError if the structured payload is missing critique/verdict,
has an empty/whitespace critique, or a non-binary verdict; the runtime refuses to
persist a malformed verdict. (The Label dataclass + schema.validate also reject an empty
critique, but we raise the typed judge error here so the boundary is the judge family.)
Source code in src/calibrated_agent_evaluation/judges/verdict.py
verdict_rater_id ¶
The rater_id a verdict from judge carries: "<judge_id>@<fingerprint>".
A stored verdict says which judge, as it is now, produced it, so a rate is corrected only with
the calibration of the judge that made the verdicts. The fingerprint is 16 hex digits, so the
id is read back by splitting on the last @.
Source code in src/calibrated_agent_evaluation/judges/verdict.py
build_verdict_schema ¶
Return the JSON schema for judge's forced-structured verdict.
Guarantees the critique-before-verdict ordering by reusing critique_before_verdict: a judge
whose output_fields place verdict before critique (or omit either) is rejected with
a JudgeRuntimeError rather than emitting a schema; a mis-ordered judge must never
reach the model. The emitted schema requires a non-empty critique (minLength: 1) and a
binary verdict enum, so a blank critique or non-binary verdict fails schema-side too.