Skip to content

calibrated-agent-evaluation score

grade traces with a deterministic code scorer (exact, levenshtein, numeric, ...)

usage: calibrated-agent-evaluation score [-h] [--trace TRACE_ID]
                                         --scorer {exact,levenshtein,numeric,json-diff,valid-json,code,sql}
                                         --mode NAME [--key FIELD] [--threshold X]
                                         [--expected FILE]
                                         [--frame {production-representative,discovery}]
                                         [--sandbox {local,agentcore}] [--region REGION]
                                         [--profile PROFILE] [--role-arn ARN]
                                         [--max-billed-calls N] [--backend {local,langfuse}]
                                         [--root PATH] [--explain]

Compare each trace's output with its expected value using a code scorer and store the PASS/FAIL
verdict for --mode, stamped with the scorer and its parameters. Expected values come from the
human FAIL labels for the mode, or --expected FILE. Gate them with `calibrated-agent-evaluation
run --verdicts scorer`.

options:
  -h, --help            show this help message and exit
  --trace TRACE_ID      operate on one stored trace (default: every stored trace); an unknown id
                        is refused
  --scorer {exact,levenshtein,numeric,json-diff,valid-json,code,sql}
                        the deterministic scorer. Compare-to-expected: exact match, Levenshtein
                        similarity, numeric closeness, JSON structural similarity, JSON validity
                        (needs the scorers extra). Execution: code (run the trace's code against
                        the --expected tests in a sandbox) or sql (execute the trace's query and
                        the --expected gold query on a seeded SQLite database)
  --mode NAME           the failure mode the verdicts measure (a FAIL is stamped with it)
  --key FIELD           compare this field of the output and the expected value (default: the
                        whole value)
  --threshold X         similarity at or above which a trace passes, for levenshtein (default
                        0.9), numeric (0.99) and json-diff (1.0)
  --expected FILE       JSONL of {"trace_id": ..., "expected": {...}} (default: the expected value
                        a human recorded on a FAIL label for the mode)
  --frame {production-representative,discovery}
                        score only the traces in this frame
  --sandbox {local,agentcore}
                        where --scorer code runs the code: local (default, stdlib subprocess, no
                        extra, no AWS; best-effort isolation, not a security boundary) or
                        agentcore (the Bedrock AgentCore Code Interpreter, needs [agentcore]; use
                        it for untrusted model output)
  --region REGION       AWS region for --sandbox agentcore (default: the profile's)
  --profile PROFILE     AWS profile for --sandbox agentcore (default: AWS_PROFILE / the default
                        chain)
  --role-arn ARN        require --sandbox agentcore to run as this IAM role, refusing before any
                        call if the caller is someone else (default: no identity check)
  --max-billed-calls N  cap on billed Code Interpreter calls for --sandbox agentcore; the call
                        past the cap is refused before it is sent (default: one per scored trace;
                        CALIBRATED_AGENT_EVALUATION_AGENTCORE_MAX_BILLED_CALLS also sets it)
  --backend {local,langfuse}
                        trace-store backend (default: local, offline)
  --root PATH           local root for the trace store and the stage records (default:
                        $CALIBRATED_AGENT_EVALUATION_LOCAL_ROOT, else ./.calibrated-agent-
                        evaluation; quickstart defaults to a temp dir)
  --explain             after the plain summary, print the statistics behind it (bounds, methods,
                        thresholds, record paths); CALIBRATED_AGENT_EVALUATION_EXPLAIN=1 turns it
                        on for every command