Command-line interface for Calibrated Agent Evaluation (calibrated-agent-evaluation).
The calibrated-agent-evaluation CLI runs the whole
probe → define → generate → label → cluster → judge → run → improve pipeline (plus
dump/load) over the code each feature already ships; it reuses the methodology plane, it
never re-implements it. The package layout:
cli.parser ; the argparse subcommand surface (all ten stages).
cli.config ; backend/model/extra resolution + graceful degradation.
cli.dispatch: maps a subcommand to its feature call; owns exit codes + the
top-level typed-error boundary.
cli.errors ; CliError + the (non-uniform) error-boundary family tuple.
cli.stages.*: one thin module per stage that needs more than a direct dispatch line.
The default install carries one runtime dependency, pydantic; every stage
that needs an optional extra degrades with a typed install-hint rather than an ImportError.
The console entry point is calibrated-agent-evaluation = calibrated_agent_evaluation.cli:main.
main
Entry point for calibrated-agent-evaluation. Returns a process exit code.
Source code in src/calibrated_agent_evaluation/cli/dispatch.py
| def main(argv: Sequence[str] | None = None) -> int:
"""Entry point for ``calibrated-agent-evaluation``. Returns a process exit code."""
_tolerate_unencodable_output()
args = parse_args(argv)
command: str | None = getattr(args, "command", None)
if command is None:
# No subcommand: print help and exit 0 (a bare `calibrated-agent-evaluation` is not an
# error).
from calibrated_agent_evaluation.cli.parser import build_parser
build_parser().print_help()
return EXIT_OK
if command not in _COMMANDS: # pragma: no cover - argparse restricts this, defensive only
print(f"calibrated-agent-evaluation: unknown command {command!r}", file=sys.stderr)
return EXIT_USAGE
presenter = Presenter(
out=sys.stdout,
err=sys.stderr,
explain=bool(getattr(args, "explain", False)) or _explain_from_env(),
explainers=default_error_explainers(),
)
try:
handler = _load_handler(command)
outcome = handler(args)
except UsageError as exc:
presenter.error(command, exc)
return EXIT_USAGE
except DOMAIN_ERRORS as exc:
# The one place the CLI catches the (non-uniform) typed-error family: a plain diagnostic,
# no traceback. Anything outside DOMAIN_ERRORS is a genuine bug; it propagates.
presenter.error(command, exc)
return EXIT_DOMAIN_ERROR
except ModuleNotFoundError as exc:
# The backstop for a stage that reached an optional engine before checking for it: a
# module an extra provides becomes that extra's install command. Any other missing
# module is a bug and propagates.
missing = _missing_extra(command, exc)
if missing is None:
raise
presenter.error(command, missing)
return EXIT_DOMAIN_ERROR
presenter.outcome(command, outcome, ok=outcome.exit_code == EXIT_OK)
return outcome.exit_code
|