calibrated_agent_evaluation.scorers¶
calibrated_agent_evaluation.scorers ¶
Structured-target scorers: deterministic evaluation of agent outputs.
The public surface: Scorer (the protocol), concrete scorer classes (ExactMatch, Levenshtein,
etc.), ScorerLane (CI_GATE vs EXPLORATION), score_traces() (batch helper), and the typed
error family (ScorerError, ScorerLaneError). The autoevals dependency is lazy-imported
so import calibrated_agent_evaluation.scorers succeeds without the [scorers] extra.
Scorer ¶
Bases: Protocol
Score an agent output against an expected value and return a Label.
Every scorer carries a lane attribute declaring its CI-gate eligibility.
The score() method returns a Label with kind="code", binary outcome, and
a non-empty critique.
score ¶
Score output against expected and return a Label.
Raises ScorerError on invalid input or vendor failure.
Source code in src/calibrated_agent_evaluation/scorers/base.py
ScorerLane ¶
ScorerError ¶
Bases: Exception
A typed error from the scorers module.
Wraps vendor/input-validation exceptions via chaining (raise ScorerError(...) from exc).
ScorerLaneError ¶
Bases: Exception
Raised when an EXPLORATION scorer is used in a CI-gate context.
The two-lane rule: generic generation-quality metrics are banned from CI; only structured-target
scorers with lane=CI_GATE may gate deployments.
CodeExecutionScorer ¶
Score an output's code by executing the task's tests against it in a sandbox.
The candidate code is the trace's output (output["code"], output["language"] defaulting
to "python"). The tests are the oracle and come from expected (expected["tests"] --
from --expected), not from the output: the thing under test must not supply the assertions
it is graded by. The scorer wraps both in a harness (see _HARNESS_TEMPLATE) and runs it
through the sandbox, so the program is identical on the local and AgentCore backends. The
verdict is read from the harness's result marker, never from the process exit code.
The tests run in the same interpreter as the candidate, so a determined candidate can in
principle subvert them (see _RUNNER_SRC). The harness closes the cheap forgeries: the tests
and a per-run nonce reach the runner over stdin before the candidate runs, and only one
nonce-tagged record counts. Run untrusted model output on the AgentCore sandbox so that what
it does cannot reach this machine.
Source code in src/calibrated_agent_evaluation/scorers/execution.py
score ¶
Run the oracle tests against the output's code and return a PASS/FAIL Label.
Raises:
| Type | Description |
|---|---|
ScorerError
|
the output lacks a string |
Source code in src/calibrated_agent_evaluation/scorers/execution.py
SqlExecutionScorer ¶
Score a candidate SQL query by executing it against a seeded SQLite database.
output carries {"sql": <candidate>}; expected carries
{"sql": <gold>, "schema_sql": <DDL>, "seed_sql": <inserts>}. PASS iff the candidate's result
set equals the gold's. Comparison is order-insensitive unless the gold query has an ORDER BY
(then row order must match too). SQL runs locally on stdlib sqlite3 regardless of any code
sandbox selection. The candidate query runs under a read-only authorizer, so it cannot
ATTACH a database (which would create a file on the host) or write to the database.
Each query runs under a wall-clock deadline (timeout_s) and may return at most max_rows
rows. A candidate past either limit is a FAIL; a gold query past either is a broken oracle and
raises ScorerError.
Source code in src/calibrated_agent_evaluation/scorers/execution.py
score ¶
Execute the candidate and gold queries and compare result sets.
Raises:
| Type | Description |
|---|---|
ScorerError
|
inputs are missing, or the GOLD query errors (a broken oracle, not a candidate failure). |
Source code in src/calibrated_agent_evaluation/scorers/execution.py
ExactMatchScorer ¶
Binary exact-match check. PASS if output == expected, FAIL otherwise.
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
score ¶
Score output against expected as PASS or FAIL.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output
|
dict[str, Any]
|
The agent's output. |
required |
expected
|
dict[str, Any] | None
|
The expected output; required. |
required |
trace_id
|
str
|
The trace the label is written for. |
required |
Returns:
| Type | Description |
|---|---|
Label
|
A label carrying the verdict. |
Raises:
| Type | Description |
|---|---|
ScorerError
|
|
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
JSONDiffScorer ¶
Structural JSON comparison. Configurable threshold.
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
score ¶
Score output against expected as PASS or FAIL.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output
|
dict[str, Any]
|
The agent's output. |
required |
expected
|
dict[str, Any] | None
|
The expected output; required. |
required |
trace_id
|
str
|
The trace the label is written for. |
required |
Returns:
| Type | Description |
|---|---|
Label
|
A label carrying the verdict. |
Raises:
| Type | Description |
|---|---|
ScorerError
|
|
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
LevenshteinScorer ¶
Edit-distance similarity check. Configurable threshold.
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
score ¶
Score output against expected as PASS or FAIL.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output
|
dict[str, Any]
|
The agent's output. |
required |
expected
|
dict[str, Any] | None
|
The expected output; required. |
required |
trace_id
|
str
|
The trace the label is written for. |
required |
Returns:
| Type | Description |
|---|---|
Label
|
A label carrying the verdict. |
Raises:
| Type | Description |
|---|---|
ScorerError
|
|
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
NumericDiffScorer ¶
Numeric closeness check. Expects numeric values.
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
score ¶
Score output against expected as PASS or FAIL.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output
|
dict[str, Any]
|
The agent's output. |
required |
expected
|
dict[str, Any] | None
|
The expected output; required. |
required |
trace_id
|
str
|
The trace the label is written for. |
required |
Returns:
| Type | Description |
|---|---|
Label
|
A label carrying the verdict. |
Raises:
| Type | Description |
|---|---|
ScorerError
|
|
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
ValidJSONScorer ¶
JSON validity check. Does not require expected value.
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
score ¶
Score output as PASS when it is valid JSON, FAIL otherwise.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
output
|
dict[str, Any]
|
The agent's output. |
required |
expected
|
dict[str, Any] | None
|
Ignored. |
None
|
trace_id
|
str
|
The trace the label is written for. |
required |
Returns:
| Type | Description |
|---|---|
Label
|
A label carrying the verdict. |
Raises:
| Type | Description |
|---|---|
ScorerError
|
The value to check can't be read from |
Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
enforce_ci_gate ¶
Raise ScorerLaneError if scorer is not permitted as a CI gate.
Call this before running a scorer in CI mode. Scorers with lane=EXPLORATION are
structurally banned from CI.
Source code in src/calibrated_agent_evaluation/scorers/lane.py
score_traces ¶
Score a batch of traces against expected values and return Labels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
traces
|
list[Trace]
|
the traces to score. |
required |
scorers
|
list[Scorer]
|
the scorers to apply to each trace. |
required |
expected_map
|
Mapping[str, dict[str, Any] | None]
|
mapping from trace_id → expected dict for scoring. |
required |
ci_gate_only
|
bool
|
if True, enforce that all scorers have lane=CI_GATE. |
False
|
Returns:
| Type | Description |
|---|---|
list[Label]
|
One Label per (trace, scorer) pair. |
Raises:
| Type | Description |
|---|---|
ScorerLaneError
|
if ci_gate_only=True and a scorer has lane=EXPLORATION. |
ScorerError
|
if a scorer fails on a trace (propagates, not swallowed). |
Source code in src/calibrated_agent_evaluation/scorers/batch.py
pass_at_k ¶
The unbiased pass@k over samples (one inner sequence per task) with a bootstrap interval.
Each inner sequence is a task's per-sample PASS (True) / FAIL (False) results. The point
is the mean over tasks of 1 - C(n-c, k) / C(n, k); the interval is the two-sided 95%
bootstrap percentile bound over a nested resample (tasks, then samples within task).
Raises:
| Type | Description |
|---|---|
ScorerError
|
there are no tasks, a task is empty, or |