API reference¶
The calibrated_agent_evaluation package, generated from docstrings. The library is split into planes: the data plane stores and normalizes traces, the execution plane runs agents and models, and the methodology plane holds the statistics and gates. The methodology plane never imports a trace-store or executor vendor type.
Each row names a package's main entry points. Each package has one page listing every name in its __all__: that is the public surface. Anything not listed there is internal and may change without notice.
Data plane¶
| Package | What it is for | Start with |
|---|---|---|
calibrated_agent_evaluation.schema |
The normalized Trace, Span and Label types every stage reads. |
Trace, Span, Label, validate |
calibrated_agent_evaluation.normalize |
Turn Inspect logs, Strands sessions and OTel spans into a Trace. |
normalize, OTelSpans, InspectEvalLog, StrandsSession |
calibrated_agent_evaluation.ingest |
Ingest trace files or Langfuse traces, refusing anything that would lose data. | run_ingest, capture_and_ingest, ingest_langfuse, verify_no_silent_loss |
calibrated_agent_evaluation.backends |
Trace stores: the offline local store and Langfuse. | LocalBackend, LangfuseBackend, get_backend, TraceStoreBackend |
calibrated_agent_evaluation.dumpload |
Dump and load any Calibrated Agent Evaluation record to CSV or JSON. | dump, load, results_table |
Execution plane¶
| Package | What it is for | Start with |
|---|---|---|
calibrated_agent_evaluation.execute |
Run agents and judge models through Inspect or Strands, with cassette replay. | InspectExecutor, StrandsExecutor, Cassette, BedrockRenderer |
calibrated_agent_evaluation.synth |
Generate test inputs from a taxonomy and optionally run them. | generate, generate_and_run, dedup |
calibrated_agent_evaluation.probe |
Evaluate a black-box endpoint you are authorized to test. | probe, run_battery, require_authorized, pass_hat_k |
calibrated_agent_evaluation.agentcore |
Opt-in AWS Bedrock AgentCore plug-in: the Runtime executor behind calibrated-agent-evaluation generate --run --executor agentcore, the read-only account check behind calibrated-agent-evaluation doctor --agentcore, the window ingest behind calibrated-agent-evaluation ingest --source agentcore, publishing and pulling evaluation datasets, and the strands-agents-evals failure detector with its failure vocabulary and a draft namer for failure-mode clusters. |
AgentCoreRuntimeExecutor, RuntimeTarget, AgentCoreRuntimeTask, run_checks, DoctorTarget, require_role, ReadOnlyGuard, ingest_agentcore, publish, pull, failure_vocabulary, FailureDetector, DetectorDraftNamer |
Methodology plane¶
| Package | What it is for | Start with |
|---|---|---|
calibrated_agent_evaluation.sampling |
Build discovery and production-representative sampling frames. | discovery_frame, production_frame, kish_deff |
calibrated_agent_evaluation.review |
The trace review surface that records gold labels. | ReviewSession, build_label, build_work_list |
calibrated_agent_evaluation.cluster |
Group failures into modes and check saturation and coverage. | ratify, saturation_check, coverage_report, chao1_interval |
calibrated_agent_evaluation.calibrate |
Certify a judge against gold labels and correct its measured rate. | plan_calibration, certify, corrected_rate, CalibrationRecord |
calibrated_agent_evaluation.bias |
The bias battery a judge passes before promotion. | run_battery, enforce_bias_gate, BiasBatteryRecord |
calibrated_agent_evaluation.judges |
Run a certified judge over traces or spans and record its scores. | run_judge, render_trace, process_judge, JudgeDefinition |
calibrated_agent_evaluation.scorers |
Deterministic scorers for structured outputs. | ExactMatchScorer, JSONDiffScorer, score_traces |
calibrated_agent_evaluation.scenarios |
Test scenarios built from caught failures and an answer key, grouped into slices. | build_slice, starter_key, DatasetSlice, PredefinedScenario |
calibrated_agent_evaluation.subjective |
Pairwise preference ratings for outcomes with no single right answer. | bradley_terry, record_preference, compute_ratings |
calibrated_agent_evaluation.attribution |
Find the culprit and manifestation spans in a multi-agent trace. | validate_walkback, per_agent_rates, relocation_check, CrewDefinition |
calibrated_agent_evaluation.run |
The CI gate, trend correction and drift checks. | run_ci_gate, trend_report, input_drift, judge_drift |
calibrated_agent_evaluation.improve |
Decide whether a fix worked, with pre-registration and guard slices. | evaluate_improvement, register_prereg, ImproveRequest, GuardSlice |
calibrated_agent_evaluation.enforce |
Lint judge definitions and refuse scores that skip calibration. | lint_judge_definition, enforce_score_gate, agreement_display |
calibrated_agent_evaluation.stats |
The statistics the gates use: bounds, corrections, paired tests. | wilson_lower, rogan_gladen, rg_interval, mcnemar_paired |
Harness¶
| Package | What it is for | Start with |
|---|---|---|
calibrated_agent_evaluation.cli |
The calibrated-agent-evaluation command. See the CLI reference for every subcommand. |
main |
calibrated_agent_evaluation.flagship |
The end-to-end proof that runs every stage against both reference agents. | prove, run_agent_pipeline, FlagshipReport |