Skip to content

calibrated_agent_evaluation.cli

calibrated_agent_evaluation.cli

Command-line interface for Calibrated Agent Evaluation (calibrated-agent-evaluation).

The calibrated-agent-evaluation CLI runs the whole probe → define → generate → label → cluster → judge → run → improve pipeline (plus dump/load) over the code each feature already ships; it reuses the methodology plane, it never re-implements it. The package layout:

  • cli.parser ; the argparse subcommand surface (all ten stages).
  • cli.config ; backend/model/extra resolution + graceful degradation.
  • cli.dispatch: maps a subcommand to its feature call; owns exit codes + the top-level typed-error boundary.
  • cli.errors ; CliError + the (non-uniform) error-boundary family tuple.
  • cli.stages.*: one thin module per stage that needs more than a direct dispatch line.

The default install carries one runtime dependency, pydantic; every stage that needs an optional extra degrades with a typed install-hint rather than an ImportError. The console entry point is calibrated-agent-evaluation = calibrated_agent_evaluation.cli:main.

main

main(argv=None)

Entry point for calibrated-agent-evaluation. Returns a process exit code.

Source code in src/calibrated_agent_evaluation/cli/dispatch.py
def main(argv: Sequence[str] | None = None) -> int:
    """Entry point for ``calibrated-agent-evaluation``. Returns a process exit code."""
    _tolerate_unencodable_output()
    args = parse_args(argv)

    command: str | None = getattr(args, "command", None)
    if command is None:
        # No subcommand: print help and exit 0 (a bare `calibrated-agent-evaluation` is not an
        # error).
        from calibrated_agent_evaluation.cli.parser import build_parser

        build_parser().print_help()
        return EXIT_OK

    if command not in _COMMANDS:  # pragma: no cover - argparse restricts this, defensive only
        print(f"calibrated-agent-evaluation: unknown command {command!r}", file=sys.stderr)
        return EXIT_USAGE

    presenter = Presenter(
        out=sys.stdout,
        err=sys.stderr,
        explain=bool(getattr(args, "explain", False)) or _explain_from_env(),
        explainers=default_error_explainers(),
    )
    try:
        handler = _load_handler(command)
        outcome = handler(args)
    except UsageError as exc:
        presenter.error(command, exc)
        return EXIT_USAGE
    except DOMAIN_ERRORS as exc:
        # The one place the CLI catches the (non-uniform) typed-error family: a plain diagnostic,
        # no traceback. Anything outside DOMAIN_ERRORS is a genuine bug; it propagates.
        presenter.error(command, exc)
        return EXIT_DOMAIN_ERROR
    except ModuleNotFoundError as exc:
        # The backstop for a stage that reached an optional engine before checking for it: a
        # module an extra provides becomes that extra's install command. Any other missing
        # module is a bug and propagates.
        missing = _missing_extra(command, exc)
        if missing is None:
            raise
        presenter.error(command, missing)
        return EXIT_DOMAIN_ERROR

    presenter.outcome(command, outcome, ok=outcome.exit_code == EXIT_OK)
    return outcome.exit_code