Skip to content

API reference

The calibrated_agent_evaluation package, generated from docstrings. The library is split into planes: the data plane stores and normalizes traces, the execution plane runs agents and models, and the methodology plane holds the statistics and gates. The methodology plane never imports a trace-store or executor vendor type.

Each row names a package's main entry points. Each package has one page listing every name in its __all__: that is the public surface. Anything not listed there is internal and may change without notice.

Data plane

Package What it is for Start with
calibrated_agent_evaluation.schema The normalized Trace, Span and Label types every stage reads. Trace, Span, Label, validate
calibrated_agent_evaluation.normalize Turn Inspect logs, Strands sessions and OTel spans into a Trace. normalize, OTelSpans, InspectEvalLog, StrandsSession
calibrated_agent_evaluation.ingest Ingest trace files or Langfuse traces, refusing anything that would lose data. run_ingest, capture_and_ingest, ingest_langfuse, verify_no_silent_loss
calibrated_agent_evaluation.backends Trace stores: the offline local store and Langfuse. LocalBackend, LangfuseBackend, get_backend, TraceStoreBackend
calibrated_agent_evaluation.dumpload Dump and load any Calibrated Agent Evaluation record to CSV or JSON. dump, load, results_table

Execution plane

Package What it is for Start with
calibrated_agent_evaluation.execute Run agents and judge models through Inspect or Strands, with cassette replay. InspectExecutor, StrandsExecutor, Cassette, BedrockRenderer
calibrated_agent_evaluation.synth Generate test inputs from a taxonomy and optionally run them. generate, generate_and_run, dedup
calibrated_agent_evaluation.probe Evaluate a black-box endpoint you are authorized to test. probe, run_battery, require_authorized, pass_hat_k
calibrated_agent_evaluation.agentcore Opt-in AWS Bedrock AgentCore plug-in: the Runtime executor behind calibrated-agent-evaluation generate --run --executor agentcore, the read-only account check behind calibrated-agent-evaluation doctor --agentcore, the window ingest behind calibrated-agent-evaluation ingest --source agentcore, publishing and pulling evaluation datasets, and the strands-agents-evals failure detector with its failure vocabulary and a draft namer for failure-mode clusters. AgentCoreRuntimeExecutor, RuntimeTarget, AgentCoreRuntimeTask, run_checks, DoctorTarget, require_role, ReadOnlyGuard, ingest_agentcore, publish, pull, failure_vocabulary, FailureDetector, DetectorDraftNamer

Methodology plane

Package What it is for Start with
calibrated_agent_evaluation.sampling Build discovery and production-representative sampling frames. discovery_frame, production_frame, kish_deff
calibrated_agent_evaluation.review The trace review surface that records gold labels. ReviewSession, build_label, build_work_list
calibrated_agent_evaluation.cluster Group failures into modes and check saturation and coverage. ratify, saturation_check, coverage_report, chao1_interval
calibrated_agent_evaluation.calibrate Certify a judge against gold labels and correct its measured rate. plan_calibration, certify, corrected_rate, CalibrationRecord
calibrated_agent_evaluation.bias The bias battery a judge passes before promotion. run_battery, enforce_bias_gate, BiasBatteryRecord
calibrated_agent_evaluation.judges Run a certified judge over traces or spans and record its scores. run_judge, render_trace, process_judge, JudgeDefinition
calibrated_agent_evaluation.scorers Deterministic scorers for structured outputs. ExactMatchScorer, JSONDiffScorer, score_traces
calibrated_agent_evaluation.scenarios Test scenarios built from caught failures and an answer key, grouped into slices. build_slice, starter_key, DatasetSlice, PredefinedScenario
calibrated_agent_evaluation.subjective Pairwise preference ratings for outcomes with no single right answer. bradley_terry, record_preference, compute_ratings
calibrated_agent_evaluation.attribution Find the culprit and manifestation spans in a multi-agent trace. validate_walkback, per_agent_rates, relocation_check, CrewDefinition
calibrated_agent_evaluation.run The CI gate, trend correction and drift checks. run_ci_gate, trend_report, input_drift, judge_drift
calibrated_agent_evaluation.improve Decide whether a fix worked, with pre-registration and guard slices. evaluate_improvement, register_prereg, ImproveRequest, GuardSlice
calibrated_agent_evaluation.enforce Lint judge definitions and refuse scores that skip calibration. lint_judge_definition, enforce_score_gate, agreement_display
calibrated_agent_evaluation.stats The statistics the gates use: bounds, corrections, paired tests. wilson_lower, rogan_gladen, rg_interval, mcnemar_paired

Harness

Package What it is for Start with
calibrated_agent_evaluation.cli The calibrated-agent-evaluation command. See the CLI reference for every subcommand. main
calibrated_agent_evaluation.flagship The end-to-end proof that runs every stage against both reference agents. prove, run_agent_pipeline, FlagshipReport