calibrated-agent-evaluation score¶
grade traces with a deterministic code scorer (exact, levenshtein, numeric, ...)
usage: calibrated-agent-evaluation score [-h] [--trace TRACE_ID]
--scorer {exact,levenshtein,numeric,json-diff,valid-json,code,sql}
--mode NAME [--key FIELD] [--threshold X]
[--expected FILE]
[--frame {production-representative,discovery}]
[--sandbox {local,agentcore}] [--region REGION]
[--profile PROFILE] [--role-arn ARN]
[--max-billed-calls N] [--backend {local,langfuse}]
[--root PATH] [--explain]
Compare each trace's output with its expected value using a code scorer and store the PASS/FAIL
verdict for --mode, stamped with the scorer and its parameters. Expected values come from the
human FAIL labels for the mode, or --expected FILE. Gate them with `calibrated-agent-evaluation
run --verdicts scorer`.
options:
-h, --help show this help message and exit
--trace TRACE_ID operate on one stored trace (default: every stored trace); an unknown id
is refused
--scorer {exact,levenshtein,numeric,json-diff,valid-json,code,sql}
the deterministic scorer. Compare-to-expected: exact match, Levenshtein
similarity, numeric closeness, JSON structural similarity, JSON validity
(needs the scorers extra). Execution: code (run the trace's code against
the --expected tests in a sandbox) or sql (execute the trace's query and
the --expected gold query on a seeded SQLite database)
--mode NAME the failure mode the verdicts measure (a FAIL is stamped with it)
--key FIELD compare this field of the output and the expected value (default: the
whole value)
--threshold X similarity at or above which a trace passes, for levenshtein (default
0.9), numeric (0.99) and json-diff (1.0)
--expected FILE JSONL of {"trace_id": ..., "expected": {...}} (default: the expected value
a human recorded on a FAIL label for the mode)
--frame {production-representative,discovery}
score only the traces in this frame
--sandbox {local,agentcore}
where --scorer code runs the code: local (default, stdlib subprocess, no
extra, no AWS; best-effort isolation, not a security boundary) or
agentcore (the Bedrock AgentCore Code Interpreter, needs [agentcore]; use
it for untrusted model output)
--region REGION AWS region for --sandbox agentcore (default: the profile's)
--profile PROFILE AWS profile for --sandbox agentcore (default: AWS_PROFILE / the default
chain)
--role-arn ARN require --sandbox agentcore to run as this IAM role, refusing before any
call if the caller is someone else (default: no identity check)
--max-billed-calls N cap on billed Code Interpreter calls for --sandbox agentcore; the call
past the cap is refused before it is sent (default: one per scored trace;
CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS also sets it)
--backend {local,langfuse}
trace-store backend (default: local, offline)
--root PATH local root for the trace store and the stage records (default:
$CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
evaluation; quickstart defaults to a temp dir)
--explain after the plain summary, print the statistics behind it (bounds, methods,
thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
on for every command