Skip to content

calibrated_agent_evaluation.scorers

calibrated_agent_evaluation.scorers

Structured-target scorers: deterministic evaluation of agent outputs.

The public surface: Scorer (the protocol), concrete scorer classes (ExactMatch, Levenshtein, etc.), ScorerLane (CI_GATE vs EXPLORATION), score_traces() (batch helper), and the typed error family (ScorerError, ScorerLaneError). The autoevals dependency is lazy-imported so import calibrated_agent_evaluation.scorers succeeds without the [scorers] extra.

Scorer

Bases: Protocol

Score an agent output against an expected value and return a Label.

Every scorer carries a lane attribute declaring its CI-gate eligibility. The score() method returns a Label with kind="code", binary outcome, and a non-empty critique.

lane instance-attribute

lane

score

score(output, expected, *, trace_id)

Score output against expected and return a Label.

Raises ScorerError on invalid input or vendor failure.

Source code in src/calibrated_agent_evaluation/scorers/base.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None,
    *,
    trace_id: str,
) -> Label:
    """Score ``output`` against ``expected`` and return a Label.

    Raises ``ScorerError`` on invalid input or vendor failure.
    """
    ...

ScorerLane

Bases: StrEnum

Declares whether a scorer may be used as a CI gate.

CI_GATE class-attribute instance-attribute

CI_GATE = 'ci_gate'

EXPLORATION class-attribute instance-attribute

EXPLORATION = 'exploration'

ScorerError

Bases: Exception

A typed error from the scorers module.

Wraps vendor/input-validation exceptions via chaining (raise ScorerError(...) from exc).

ScorerLaneError

Bases: Exception

Raised when an EXPLORATION scorer is used in a CI-gate context.

The two-lane rule: generic generation-quality metrics are banned from CI; only structured-target scorers with lane=CI_GATE may gate deployments.

CodeExecutionScorer

CodeExecutionScorer(
    sandbox=None, *, timeout_s=_DEFAULT_TIMEOUT_S
)

Score an output's code by executing the task's tests against it in a sandbox.

The candidate code is the trace's output (output["code"], output["language"] defaulting to "python"). The tests are the oracle and come from expected (expected["tests"] -- from --expected), not from the output: the thing under test must not supply the assertions it is graded by. The scorer wraps both in a harness (see _HARNESS_TEMPLATE) and runs it through the sandbox, so the program is identical on the local and AgentCore backends. The verdict is read from the harness's result marker, never from the process exit code.

The tests run in the same interpreter as the candidate, so a determined candidate can in principle subvert them (see _RUNNER_SRC). The harness closes the cheap forgeries: the tests and a per-run nonce reach the runner over stdin before the candidate runs, and only one nonce-tagged record counts. Run untrusted model output on the AgentCore sandbox so that what it does cannot reach this machine.

Source code in src/calibrated_agent_evaluation/scorers/execution.py
def __init__(
    self, sandbox: SandboxBackend | None = None, *, timeout_s: float = _DEFAULT_TIMEOUT_S
) -> None:
    # Default to the local sandbox so the common path needs no backend wiring and no extra.
    self._sandbox: SandboxBackend = sandbox if sandbox is not None else LocalSubprocessSandbox()
    self._timeout_s = timeout_s

lane class-attribute instance-attribute

lane = ScorerLane.CI_GATE

score

score(output, expected, *, trace_id)

Run the oracle tests against the output's code and return a PASS/FAIL Label.

Raises:

Type Description
ScorerError

the output lacks a string code, the language is not Python, the expected value does not carry the string tests oracle, or the sandbox could not run the program (an infrastructure failure, not a test failure).

Source code in src/calibrated_agent_evaluation/scorers/execution.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None,
    *,
    trace_id: str,
) -> Label:
    """Run the oracle tests against the output's code and return a PASS/FAIL Label.

    Raises:
        ScorerError: the output lacks a string ``code``, the language is not Python, the
            expected value does not carry the string ``tests`` oracle, or the sandbox could not
            run the program (an infrastructure failure, not a test failure).
    """
    code = output.get("code")
    if not isinstance(code, str):
        raise ScorerError(
            f"CodeExecutionScorer needs the output to carry a string 'code' (got keys "
            f"{sorted(output)})"
        )
    language = output.get("language", "python")
    if language != "python":
        raise ScorerError(f"the code scorer runs Python only in v1, got language={language!r}")
    tests = expected.get("tests") if isinstance(expected, dict) else None
    if not isinstance(tests, str):
        raise ScorerError(
            "CodeExecutionScorer needs the tests in the expected value (pass --expected "
            'carrying {"tests": <assertions>}); the output under test must not supply the '
            "tests it is graded by"
        )
    harness = _build_code_harness(code, tests, self._timeout_s)
    try:
        result = self._sandbox.run_code(
            harness, language="python", timeout_s=self._timeout_s + _TIMEOUT_GRACE_S
        )
    except ExecutionError as exc:
        # Infrastructure failure (sandbox/ backend), not a test result: no verdict.
        raise ScorerError(f"code execution could not be run: {exc}") from exc
    return self._verdict(result, trace_id)

SqlExecutionScorer

SqlExecutionScorer(
    *,
    timeout_s=_DEFAULT_SQL_TIMEOUT_S,
    max_rows=_DEFAULT_SQL_MAX_ROWS,
)

Score a candidate SQL query by executing it against a seeded SQLite database.

output carries {"sql": <candidate>}; expected carries {"sql": <gold>, "schema_sql": <DDL>, "seed_sql": <inserts>}. PASS iff the candidate's result set equals the gold's. Comparison is order-insensitive unless the gold query has an ORDER BY (then row order must match too). SQL runs locally on stdlib sqlite3 regardless of any code sandbox selection. The candidate query runs under a read-only authorizer, so it cannot ATTACH a database (which would create a file on the host) or write to the database.

Each query runs under a wall-clock deadline (timeout_s) and may return at most max_rows rows. A candidate past either limit is a FAIL; a gold query past either is a broken oracle and raises ScorerError.

Source code in src/calibrated_agent_evaluation/scorers/execution.py
def __init__(
    self, *, timeout_s: float = _DEFAULT_SQL_TIMEOUT_S, max_rows: int = _DEFAULT_SQL_MAX_ROWS
) -> None:
    if not timeout_s > 0:
        raise ValueError(f"timeout_s must be positive, got {timeout_s!r}")
    if max_rows < 1:
        raise ValueError(f"max_rows must be at least 1, got {max_rows!r}")
    self._timeout_s = timeout_s
    self._max_rows = max_rows

lane class-attribute instance-attribute

lane = ScorerLane.CI_GATE

score

score(output, expected, *, trace_id)

Execute the candidate and gold queries and compare result sets.

Raises:

Type Description
ScorerError

inputs are missing, or the GOLD query errors (a broken oracle, not a candidate failure).

Source code in src/calibrated_agent_evaluation/scorers/execution.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None,
    *,
    trace_id: str,
) -> Label:
    """Execute the candidate and gold queries and compare result sets.

    Raises:
        ScorerError: inputs are missing, or the GOLD query errors (a broken oracle, not a
            candidate failure).
    """
    if expected is None:
        raise ScorerError("SqlExecutionScorer requires an expected value with the gold query")
    candidate = output.get("sql")
    gold = expected.get("sql")
    schema_sql = expected.get("schema_sql", "")
    seed_sql = expected.get("seed_sql", "")
    if not isinstance(candidate, str) or not isinstance(gold, str):
        raise ScorerError("SqlExecutionScorer needs a string 'sql' in both output and expected")

    with closing(sqlite3.connect(":memory:")) as conn:
        try:
            if schema_sql:
                conn.executescript(schema_sql)
            if seed_sql:
                conn.executescript(seed_sql)
        except sqlite3.Error as exc:
            raise ScorerError(f"SQL fixture schema/seed failed to load: {exc}") from exc
        try:
            gold_rows = self._bounded_fetch(conn, gold)
        except _SqlLimitExceeded as exc:
            raise ScorerError(
                f"gold SQL query {exc} (broken oracle, or raise the limit)"
            ) from exc
        except sqlite3.Error as exc:
            # A broken oracle is not a candidate failure; refuse rather than score against it.
            raise ScorerError(f"gold SQL query errored (broken oracle): {exc}") from exc
        # From here on the UNTRUSTED candidate runs: restrict it to reads so it cannot ATTACH a
        # file (which would write to the host), or write/alter the database. The fixture and the
        # trusted gold query ran above, before the restriction is installed.
        conn.set_authorizer(_read_only_authorizer)
        try:
            candidate_rows = self._bounded_fetch(conn, candidate)
        except _SqlLimitExceeded as exc:
            return _label(trace_id, "FAIL", f"SQL execution: candidate query {exc}")
        except sqlite3.Error as exc:
            return _label(trace_id, "FAIL", f"SQL execution: candidate query errored: {exc}")

    ordered = _gold_is_ordered(gold)
    if ordered:
        matched = candidate_rows == gold_rows
    else:
        matched = Counter(map(tuple, candidate_rows)) == Counter(map(tuple, gold_rows))
    if matched:
        how = "ordered" if ordered else "as a multiset"
        return _label(trace_id, "PASS", f"SQL execution: result set matches the gold ({how})")
    critique = (
        f"SQL execution: result set differs from gold "
        f"(candidate {len(candidate_rows)} rows, gold {len(gold_rows)} rows)"
    )
    return _label(trace_id, "FAIL", critique)

ExactMatchScorer

ExactMatchScorer(*, key=None)

Binary exact-match check. PASS if output == expected, FAIL otherwise.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def __init__(self, *, key: str | None = None) -> None:
    self._key = key

lane class-attribute instance-attribute

lane = ScorerLane.CI_GATE

score

score(output, expected, *, trace_id)

Score output against expected as PASS or FAIL.

Parameters:

Name Type Description Default
output dict[str, Any]

The agent's output.

required
expected dict[str, Any] | None

The expected output; required.

required
trace_id str

The trace the label is written for.

required

Returns:

Type Description
Label

A label carrying the verdict.

Raises:

Type Description
ScorerError

expected is missing or a value can't be compared.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None,
    *,
    trace_id: str,
) -> Label:
    """Score ``output`` against ``expected`` as PASS or FAIL.

    Args:
        output: The agent's output.
        expected: The expected output; required.
        trace_id: The trace the label is written for.

    Returns:
        A label carrying the verdict.

    Raises:
        ScorerError: ``expected`` is missing or a value can't be compared.
    """
    if expected is None:
        raise ScorerError("ExactMatchScorer requires an expected value")
    ae = _load_autoevals()
    out_val = _extract_value(output, self._key)
    exp_val = _extract_value(expected, self._key, side="expected")
    try:
        result = ae.ExactMatch().eval(output=str(out_val), expected=str(exp_val))
    except Exception as exc:
        raise ScorerError(f"ExactMatchScorer failed: {exc}") from exc
    outcome = "PASS" if result.score == 1.0 else "FAIL"
    if outcome == "PASS":
        critique = f"ExactMatch: output matches expected ('{exp_val}')"
    else:
        critique = f"ExactMatch: got '{out_val}', expected '{exp_val}'"
    return Label(
        trace_id=trace_id, span_id=None, kind="code", outcome=outcome, critique=critique
    )

JSONDiffScorer

JSONDiffScorer(*, threshold=1.0, key=None)

Structural JSON comparison. Configurable threshold.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def __init__(self, *, threshold: float = 1.0, key: str | None = None) -> None:
    self._threshold = threshold
    self._key = key

lane class-attribute instance-attribute

lane = ScorerLane.CI_GATE

score

score(output, expected, *, trace_id)

Score output against expected as PASS or FAIL.

Parameters:

Name Type Description Default
output dict[str, Any]

The agent's output.

required
expected dict[str, Any] | None

The expected output; required.

required
trace_id str

The trace the label is written for.

required

Returns:

Type Description
Label

A label carrying the verdict.

Raises:

Type Description
ScorerError

expected is missing or a value can't be compared.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None,
    *,
    trace_id: str,
) -> Label:
    """Score ``output`` against ``expected`` as PASS or FAIL.

    Args:
        output: The agent's output.
        expected: The expected output; required.
        trace_id: The trace the label is written for.

    Returns:
        A label carrying the verdict.

    Raises:
        ScorerError: ``expected`` is missing or a value can't be compared.
    """
    if expected is None:
        raise ScorerError("JSONDiffScorer requires an expected value")
    import json

    out_val = _extract_value(output, self._key)
    exp_val = _extract_value(expected, self._key, side="expected")
    out_str = json.dumps(out_val, sort_keys=True) if not isinstance(out_val, str) else out_val
    exp_str = json.dumps(exp_val, sort_keys=True) if not isinstance(exp_val, str) else exp_val
    ae = _load_autoevals()
    try:
        result = ae.JSONDiff().eval(output=out_str, expected=exp_str)
    except Exception as exc:
        raise ScorerError(f"JSONDiffScorer failed: {exc}") from exc
    score_val = result.score if result.score is not None else 0.0
    outcome = "PASS" if score_val >= self._threshold else "FAIL"
    critique = (
        f"JSONDiff: similarity {score_val:.3f} "
        f"{'≥' if outcome == 'PASS' else '<'} threshold {self._threshold}"
    )
    return Label(
        trace_id=trace_id, span_id=None, kind="code", outcome=outcome, critique=critique
    )

LevenshteinScorer

LevenshteinScorer(*, threshold=0.9, key=None)

Edit-distance similarity check. Configurable threshold.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def __init__(self, *, threshold: float = 0.9, key: str | None = None) -> None:
    self._threshold = threshold
    self._key = key

lane class-attribute instance-attribute

lane = ScorerLane.CI_GATE

score

score(output, expected, *, trace_id)

Score output against expected as PASS or FAIL.

Parameters:

Name Type Description Default
output dict[str, Any]

The agent's output.

required
expected dict[str, Any] | None

The expected output; required.

required
trace_id str

The trace the label is written for.

required

Returns:

Type Description
Label

A label carrying the verdict.

Raises:

Type Description
ScorerError

expected is missing or a value can't be compared.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None,
    *,
    trace_id: str,
) -> Label:
    """Score ``output`` against ``expected`` as PASS or FAIL.

    Args:
        output: The agent's output.
        expected: The expected output; required.
        trace_id: The trace the label is written for.

    Returns:
        A label carrying the verdict.

    Raises:
        ScorerError: ``expected`` is missing or a value can't be compared.
    """
    if expected is None:
        raise ScorerError("LevenshteinScorer requires an expected value")
    ae = _load_autoevals()
    out_val = _extract_value(output, self._key)
    exp_val = _extract_value(expected, self._key, side="expected")
    try:
        result = ae.LevenshteinScorer().eval(output=str(out_val), expected=str(exp_val))
    except Exception as exc:
        raise ScorerError(f"LevenshteinScorer failed: {exc}") from exc
    score_val = result.score if result.score is not None else 0.0
    outcome = "PASS" if score_val >= self._threshold else "FAIL"
    critique = (
        f"Levenshtein: similarity {score_val:.3f} "
        f"{'≥' if outcome == 'PASS' else '<'} threshold {self._threshold}"
    )
    return Label(
        trace_id=trace_id, span_id=None, kind="code", outcome=outcome, critique=critique
    )

NumericDiffScorer

NumericDiffScorer(*, threshold=0.99, key=None)

Numeric closeness check. Expects numeric values.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def __init__(self, *, threshold: float = 0.99, key: str | None = None) -> None:
    self._threshold = threshold
    self._key = key

lane class-attribute instance-attribute

lane = ScorerLane.CI_GATE

score

score(output, expected, *, trace_id)

Score output against expected as PASS or FAIL.

Parameters:

Name Type Description Default
output dict[str, Any]

The agent's output.

required
expected dict[str, Any] | None

The expected output; required.

required
trace_id str

The trace the label is written for.

required

Returns:

Type Description
Label

A label carrying the verdict.

Raises:

Type Description
ScorerError

expected is missing or a value can't be compared.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None,
    *,
    trace_id: str,
) -> Label:
    """Score ``output`` against ``expected`` as PASS or FAIL.

    Args:
        output: The agent's output.
        expected: The expected output; required.
        trace_id: The trace the label is written for.

    Returns:
        A label carrying the verdict.

    Raises:
        ScorerError: ``expected`` is missing or a value can't be compared.
    """
    if expected is None:
        raise ScorerError("NumericDiffScorer requires an expected value")
    out_val = _extract_value(output, self._key)
    exp_val = _extract_value(expected, self._key, side="expected")
    try:
        out_num = float(out_val)
        exp_num = float(exp_val)
    except (TypeError, ValueError) as exc:
        raise ScorerError(
            f"NumericDiffScorer: expected numeric values, got output={out_val!r}, "
            f"expected={exp_val!r}"
        ) from exc
    ae = _load_autoevals()
    try:
        result = ae.NumericDiff().eval(output=out_num, expected=exp_num)
    except Exception as exc:
        raise ScorerError(f"NumericDiffScorer failed: {exc}") from exc
    score_val = result.score if result.score is not None else 0.0
    outcome = "PASS" if score_val >= self._threshold else "FAIL"
    critique = (
        f"NumericDiff: closeness {score_val:.4f} "
        f"{'≥' if outcome == 'PASS' else '<'} threshold {self._threshold} "
        f"(output={out_num}, expected={exp_num})"
    )
    return Label(
        trace_id=trace_id, span_id=None, kind="code", outcome=outcome, critique=critique
    )

ValidJSONScorer

ValidJSONScorer(*, key=None)

JSON validity check. Does not require expected value.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def __init__(self, *, key: str | None = None) -> None:
    self._key = key

lane class-attribute instance-attribute

lane = ScorerLane.CI_GATE

score

score(output, expected=None, *, trace_id)

Score output as PASS when it is valid JSON, FAIL otherwise.

Parameters:

Name Type Description Default
output dict[str, Any]

The agent's output.

required
expected dict[str, Any] | None

Ignored.

None
trace_id str

The trace the label is written for.

required

Returns:

Type Description
Label

A label carrying the verdict.

Raises:

Type Description
ScorerError

The value to check can't be read from output.

Source code in src/calibrated_agent_evaluation/scorers/heuristic.py
def score(
    self,
    output: dict[str, Any],
    expected: dict[str, Any] | None = None,  # noqa: ARG002
    *,
    trace_id: str,
) -> Label:
    """Score ``output`` as PASS when it is valid JSON, FAIL otherwise.

    Args:
        output: The agent's output.
        expected: Ignored.
        trace_id: The trace the label is written for.

    Returns:
        A label carrying the verdict.

    Raises:
        ScorerError: The value to check can't be read from ``output``.
    """
    ae = _load_autoevals()
    out_val = _extract_value(output, self._key)
    out_str = str(out_val) if not isinstance(out_val, str) else out_val
    try:
        result = ae.ValidJSON().eval(output=out_str, expected=None)
    except Exception as exc:
        raise ScorerError(f"ValidJSONScorer failed: {exc}") from exc
    outcome = "PASS" if result.score == 1 else "FAIL"
    if outcome == "PASS":
        critique = "ValidJSON: output is valid JSON"
    else:
        critique = f"ValidJSON: output is not valid JSON ('{out_str[:50]}...')"
    return Label(
        trace_id=trace_id, span_id=None, kind="code", outcome=outcome, critique=critique
    )

enforce_ci_gate

enforce_ci_gate(scorer)

Raise ScorerLaneError if scorer is not permitted as a CI gate.

Call this before running a scorer in CI mode. Scorers with lane=EXPLORATION are structurally banned from CI.

Source code in src/calibrated_agent_evaluation/scorers/lane.py
def enforce_ci_gate(scorer: Scorer) -> None:
    """Raise ``ScorerLaneError`` if ``scorer`` is not permitted as a CI gate.

    Call this before running a scorer in CI mode. Scorers with ``lane=EXPLORATION`` are
    structurally banned from CI.
    """
    if scorer.lane != ScorerLane.CI_GATE:
        raise ScorerLaneError(
            f"scorer '{type(scorer).__name__}' has lane={scorer.lane.value} and cannot be "
            f"used as a CI gate. Only scorers with lane=CI_GATE are permitted in CI mode "
            f"(generic generation-quality metrics are exploration-only)."
        )

score_traces

score_traces(
    traces, scorers, expected_map, *, ci_gate_only=False
)

Score a batch of traces against expected values and return Labels.

Parameters:

Name Type Description Default
traces list[Trace]

the traces to score.

required
scorers list[Scorer]

the scorers to apply to each trace.

required
expected_map Mapping[str, dict[str, Any] | None]

mapping from trace_id → expected dict for scoring.

required
ci_gate_only bool

if True, enforce that all scorers have lane=CI_GATE.

False

Returns:

Type Description
list[Label]

One Label per (trace, scorer) pair.

Raises:

Type Description
ScorerLaneError

if ci_gate_only=True and a scorer has lane=EXPLORATION.

ScorerError

if a scorer fails on a trace (propagates, not swallowed).

Source code in src/calibrated_agent_evaluation/scorers/batch.py
def score_traces(
    traces: list[Trace],
    scorers: list[Scorer],
    expected_map: Mapping[str, dict[str, Any] | None],
    *,
    ci_gate_only: bool = False,
) -> list[Label]:
    """Score a batch of traces against expected values and return Labels.

    Args:
        traces: the traces to score.
        scorers: the scorers to apply to each trace.
        expected_map: mapping from trace_id → expected dict for scoring.
        ci_gate_only: if True, enforce that all scorers have lane=CI_GATE.

    Returns:
        One Label per (trace, scorer) pair.

    Raises:
        ScorerLaneError: if ci_gate_only=True and a scorer has lane=EXPLORATION.
        ScorerError: if a scorer fails on a trace (propagates, not swallowed).
    """
    if ci_gate_only:
        for scorer in scorers:
            enforce_ci_gate(scorer)

    labels: list[Label] = []
    for trace in traces:
        expected = expected_map.get(trace.trace_id)
        for scorer in scorers:
            label = scorer.score(output=trace.output, expected=expected, trace_id=trace.trace_id)
            labels.append(label)
    return labels

pass_at_k

pass_at_k(
    samples, k, *, resamples=_DEFAULT_RESAMPLES, seed=0
)

The unbiased pass@k over samples (one inner sequence per task) with a bootstrap interval.

Each inner sequence is a task's per-sample PASS (True) / FAIL (False) results. The point is the mean over tasks of 1 - C(n-c, k) / C(n, k); the interval is the two-sided 95% bootstrap percentile bound over a nested resample (tasks, then samples within task).

Raises:

Type Description
ScorerError

there are no tasks, a task is empty, or k is outside 1..n for any task (k samples cannot be drawn from fewer than k).

Source code in src/calibrated_agent_evaluation/scorers/passk.py
def pass_at_k(
    samples: Sequence[Sequence[bool]],
    k: int,
    *,
    resamples: int = _DEFAULT_RESAMPLES,
    seed: int = 0,
) -> PassAtK:
    """The unbiased pass@k over ``samples`` (one inner sequence per task) with a bootstrap interval.

    Each inner sequence is a task's per-sample PASS (``True``) / FAIL (``False``) results. The point
    is the mean over tasks of ``1 - C(n-c, k) / C(n, k)``; the interval is the two-sided 95%
    bootstrap percentile bound over a nested resample (tasks, then samples within task).

    Raises:
        ScorerError: there are no tasks, a task is empty, or ``k`` is outside ``1..n`` for any task
            (``k`` samples cannot be drawn from fewer than ``k``).
    """
    if not samples:
        raise ScorerError("pass_at_k: need at least one task")
    tasks: list[list[bool]] = []
    for i, task in enumerate(samples):
        n = len(task)
        if not 1 <= k <= n:
            raise ScorerError(
                f"pass_at_k: k={k} must lie in [1, n] for task {i}, which has n={n} samples"
            )
        tasks.append(list(task))

    point = sum(_pass_at_k_one(sum(t), len(t), k) for t in tasks) / len(tasks)
    interval = _bootstrap_interval(tasks, k, resamples=resamples, seed=seed)
    return PassAtK(point=point, interval=interval, k=k, n_tasks=len(tasks))