calibrated_agent_evaluation.probe¶
calibrated_agent_evaluation.probe ¶
The black-box probing toolkit: evaluate an agent you don't own, honestly.
Backend logic (no CLI, no UI) for probing an unowned HTTP agent: capability discovery,
repeated-trial robustness, mandatory limits-of-inference framing, judge-biased upper-bound rates,
refuse-by-default authorization, and the three-layer agent-security battery. probe_with runs
the same capability discovery over any ProbeAdapter the caller builds, for a target no
URL names.
probe/ imports no vendor SDK: the transport is stdlib (adapter.py), and the sole
inspect_evals import lives in the sanctioned
calibrated_agent_evaluation.execute.benchmark_tasks. Probe results are normalized probed
Traces with ownership="probed", spans == [].
DEFAULT_BATTERY
module-attribute
¶
DEFAULT_SEEDS
module-attribute
¶
The two prompts a probe sends when it is given none: what the agent can help with, and one simple task in its domain.
EndpointAdapter ¶
Sends prompts to a target endpoint and parses SSE/JSON responses.
On the first send it auto-detects which PayloadFormat the endpoint accepts and reuses
it for subsequent sends.
Source code in src/calibrated_agent_evaluation/probe/adapter.py
detected_format
property
¶
The name of the payload format that worked, once detected (else None).
send ¶
Send prompt and return the parsed response, or raise on failure.
Auto-detects the payload format on first use. Opens the circuit (raises) once
max_consecutive_failures is reached rather than hammering a dead endpoint.
Source code in src/calibrated_agent_evaluation/probe/adapter.py
ProbeAdapter ¶
Bases: Protocol
Sends one probe prompt to a target and answers with the parsed reply.
The probe calls nothing else, so any object with this method can be probed through
probe_with. EndpointAdapter and
WebSocketEndpointAdapter satisfy it. An
adapter raises only errors of the Calibrated Agent Evaluation family, so a failed prompt reaches
its caller typed and explained.
send ¶
Send prompt and return the parsed reply, or raise the adapter's own typed error.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
prompt
|
str
|
The probe prompt. |
required |
Returns:
| Type | Description |
|---|---|
ProbeResponse
|
The parsed reply. |
Raises:
| Type | Description |
|---|---|
CalibratedAgentEvaluationDataError
|
The prompt was refused or failed: an error of the
|
Source code in src/calibrated_agent_evaluation/probe/adapter.py
ProbeResponse
dataclass
¶
One parsed probe response (capability-discovery raw material).
Attributes:
| Name | Type | Description |
|---|---|---|
status |
int
|
The HTTP status of the reply. |
text |
str
|
The reply's final text. |
observed_tools |
tuple[str, ...]
|
The tools the reply named, in first-seen order, each once. |
events |
tuple[dict, ...]
|
The reply's decoded JSON objects, in order. |
trace_id |
str | None
|
The ID the probe names this reply's trace by. |
BenchmarkSpec
dataclass
¶
Resolution
dataclass
¶
CapabilityMap
dataclass
¶
What probing observed about an unowned agent -- a lower bound, not an inventory.
observed_tools are tools seen USED in a probe response; refusal_boundaries are prompts
the agent declined. framing is mandatory so the lower-bound nature is structural.
FailureHypothesis
dataclass
¶
ApiKeyAuth
dataclass
¶
API-key auth in a named header; the key is read from key_env at send time.
headers ¶
BearerAuth
dataclass
¶
Bearer-token auth; the token is read from token_env at send time.
headers ¶
The Authorization: Bearer header, with the token read from the environment.
ConnectionConfig
dataclass
¶
ConnectionConfig(
endpoint,
auth=None,
extra_headers=(),
timeout_seconds=30.0,
max_consecutive_failures=3,
)
How to reach a target endpoint. Secrets are env-sourced and redacted.
timeout_seconds is explicit (never a library default that could change); a dead peer can't
hang forever. max_consecutive_failures bounds the circuit breaker.
resolved_headers ¶
Resolve auth + extra headers to a plain dict (secrets pulled from env at call time).
Source code in src/calibrated_agent_evaluation/probe/connection.py
HeaderAuth
dataclass
¶
Raw-header auth; headers_env holds a JSON object of header name -> value.
headers ¶
The headers parsed from the JSON object in the environment variable.
Source code in src/calibrated_agent_evaluation/probe/connection.py
SubprotocolAuth
dataclass
¶
WebSocket auth carried in the Sec-WebSocket-Protocol handshake header.
The token is read from token_env at send time and passed to
websockets.sync.client.connect(..., subprotocols=[<token>]). The TACO target uses this
shape. Not usable on the HTTP path -- if paired with an http(s):// endpoint the WS-only
subprotocols accessor is what the adapter reads, and the .headers() accessor
intentionally returns an empty mapping (nothing to inject into HTTP headers).
AuthorizationError ¶
Bases: ProbeError
A probe was attempted before the user confirmed authorization.
Refuse-by-default: no HTTP request is sent until authorization is confirmed and recorded.
EndpointUnreachableError ¶
Bases: ProbeError
The target endpoint is unreachable or returned a transport/HTTP error.
Raised rather than recording an empty/degraded result as a success -- a silent empty-success on a dead endpoint is the worst kind of silent failure. Also raised when the consecutive-failure circuit breaker opens.
PolicyCaseError ¶
Bases: ProbeError
A policy case file can't be read or is invalid, or its tool names match no decision.
Raised before any AWS call for a file that is missing, unreadable, not JSON or invalid, and after the trials when decisions were read but none names a tool any case lists, which is the shape of a case file written in the wrong tool-name form.
PolicyDecisionsMissing ¶
Bases: ProbeError
A policy battery read no decision it could score, so no finding would mean anything.
Raised when no trial of the battery got a per-call decision, or when a control case got none in any of its trials.
PolicyModeMixed ¶
Bases: ProbeError
One policy case's decisions came under two gateway modes, which can't be pooled.
ProbeError ¶
Bases: CalibratedAgentEvaluationDataError
Base class for every probing-toolkit error.
ProbeMissingExtraError ¶
Bases: ProbeError
WebSocket (ws/wss) probe requested without the [probe-ws] extra installed.
Fail-closed: the harness refuses rather than silently falling back to HTTP or no-opping. The message names the extra and its install command so the caller can fix it in one step.
ProbeUncorrectableError ¶
Bases: ProbeError
A prevalence-corrected point rate was requested for a black-box target.
A black-box target has no calibration set, so the true rate is not identifiable -- only a judge-biased upper bound is honest. The upper-bound interval is the signal.
LimitsFraming
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The three mandatory limits-of-inference flags on every probing output.
Config:
frozen:Trueextra:forbid
Fields:
ProbeReport
dataclass
¶
The Stage-0 output of a probe run.
detected_format names the PayloadFormat
the HTTP adapter settled on (None for a WebSocket probe, which negotiates no format). It is
reported because auto-detection accepts the first format the endpoint answers 200 to, and an
endpoint that answers 200 to a payload it could not read (coercing a missing prompt to an empty
one) makes detection settle on a shape that delivers nothing. Every trace in that run then
describes a reply to no prompt, and the body parses, so neither the degrade path nor an error
fires. Naming the winner is what lets a caller see that in one line instead of inferring it from
the answers.
From probe_with, detected_format is the payload format the caller named, not one
that was detected: that adapter sends no detection prompt. It still says which request shape
every prompt was sent in. The capability-map finding's detected_format entry is the same
value, named or detected alike.
AgentErrorTrial
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A trial whose agent reported an error: its probe trace ID and error code, never its text.
Config:
frozen:Trueextra:forbid
Fields:
PolicyCase
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One case of a policy file: a prompt, its trial count, the decision each tool should get.
A control case expects at least one decision in some trial; one that gets none refuses the battery, since it is the only per-run check that the agent's calls reached the engine.
Config:
frozen:Trueextra:forbid
Fields:
Validators:
PolicyCaseFile
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A policy case file: its schema version and its cases, at least one of them a control.
Config:
frozen:Trueextra:forbid
Fields:
-
schema_version(Literal['1']) -
cases(tuple[PolicyCase, ...])
Validators:
-
_check_cases
PolicyDecision
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One decision a policy engine recorded on one tool call an agent attempted.
tool is the tool name exactly as the engine's record gives it, matched exactly against a
case's expectations. engine is the engine reference the record carried, reason the
engine's reason text, verbatim.
Config:
frozen:Trueextra:forbid
Fields:
-
trace_id(str) -
span_id(str) -
tool(str) -
decision(Decision) -
mode(GatewayMode) -
determining_policies(tuple[str, ...]) -
log_only_flipping_policies(tuple[str, ...]) -
log_only_incomplete(bool) -
engine(str | None) -
reason(str | None)
Validators:
TrialDecisions
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
A decision source's answer for one trial: the trial's W3C trace ID and its decisions.
Config:
frozen:Trueextra:forbid
Fields:
-
w3c_trace_id(str) -
decisions(tuple[PolicyDecision, ...])
Validators:
-
_w3c→w3c_trace_id
TrialRef
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One trial: the case it ran, its number, its probe trace ID and its times.
w3c_trace_id is None as sent and set from the decision source's answer.
error_code is set for a trial whose agent reported an error.
Config:
frozen:Trueextra:forbid
Fields:
-
case_id(str) -
trial(StrictInt) -
probe_trace_id(str) -
started_at(str) -
ended_at(str) -
error_code(str | None) -
w3c_trace_id(str | None)
Validators:
-
_w3c→w3c_trace_id
DecisionSource ¶
PolicyBatteryResult
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One policy battery: a finding per case, its trials in order, and one probed trace per trial.
traces[i] is trial i's outcome-only probed trace, under its probe trace ID, so a label
on it names a trace the store holds.
Config:
frozen:Trueextra:forbid
Fields:
Validators:
-
_traces_match_trials
UpperBoundRate
dataclass
¶
A judge-biased upper bound on a failure rate over k trials.
upper_bound is the honest signal; there is no corrected point estimate for a black-box
target. require_point raises rather than fabricating one. failures is the number
of the k trials the judge flagged (None when the bound came from elsewhere), and
level the one-sided confidence of the bound.
require_point ¶
Refuse a corrected point rate -- none exists for a black-box target.
Source code in src/calibrated_agent_evaluation/probe/rate.py
ProbeBatteryRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
The findings of one benchmark-battery run against one endpoint.
findings holds one ProbeFinding per benchmark that ran, each carrying the trial
count, the break count and the one-sided 95% upper bound on the break rate. There is no point
estimate: a black-box target has no calibration set, so the bound is the honest signal.
Config:
frozen:Trueextra:forbid
Fields:
-
probe_id(str) -
endpoint_fingerprint(str) -
findings(tuple[ProbeFinding, ...]) -
schema_version(str) -
provenance(ProvenanceStamp)
ProbeFinding
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One auditable probing finding.
captured_at (ISO-8601) + endpoint_fingerprint stamp the non-stationarity context;
framing is the mandatory limits-of-inference; substitutions records any benchmark
substitutions surfaced; payload holds the finding-specific data.
A capability_map finding's payload["detected_format"] is the request shape every
prompt was sent in: the one the HTTP probe detected, or, from probe_with, the one the
caller named, since that probe detects nothing. It is empty when no shape applies, as for a
WebSocket probe.
Config:
frozen:Trueextra:forbid
Fields:
-
kind(str) -
captured_at(str) -
endpoint_fingerprint(str) -
framing(LimitsFraming) -
payload(dict[str, Any]) -
substitutions(tuple[str, ...]) -
schema_version(str)
Validators:
-
_check_auditable
RobustnessResult
dataclass
¶
The aggregated result over n repeated trials.
rate is the per-trial rate with its Clopper-Pearson interval: the held rate for
kind="capability", the break rate for kind="security". For capability, k is the
pass^k order, pass_hat_k the tau-bench estimate, and passed whether all n
trials held. For security, k == n and passed is NOT vulnerable.
WebSocketEndpointAdapter ¶
Send prompts over a WebSocket and parse the returned frames.
Same public shape as EndpointAdapter: send(prompt) -> ProbeResponse. Reuses the HTTP
adapter's _parse so JSON/SSE-shaped frame payloads flow through the same normalization.
Source code in src/calibrated_agent_evaluation/probe/ws_adapter.py
send ¶
Open a WS connection, send prompt as one frame, recv until close/idle, parse.
Opens the circuit (raises) once max_consecutive_failures is reached, matching the HTTP
adapter. Any transport / handshake / recv failure is wrapped into
EndpointUnreachableError -- never returned as an empty success.
Source code in src/calibrated_agent_evaluation/probe/ws_adapter.py
confirm_authorization ¶
Record the authorization attestation as an auditable finding.
Source code in src/calibrated_agent_evaluation/probe/authorize.py
require_authorized ¶
Raise AuthorizationError unless authorization was explicitly confirmed.
Called before any request is sent -- refuse-by-default means an unauthorized probe emits zero packets.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
target
|
ConnectionConfig | str
|
The endpoint about to be probed, or a label naming a target no URL names, such
as |
required |
authorized
|
bool
|
Your confirmation that you are allowed to probe it. |
required |
Raises:
| Type | Description |
|---|---|
AuthorizationError
|
|
Source code in src/calibrated_agent_evaluation/probe/authorize.py
run_battery ¶
run_battery(
config,
run_trial=None,
*,
authorized,
requested=None,
allow_use_restricted=False,
k=1,
max_break_rate=DEFAULT_MAX_BREAK_RATE,
run_trials=None,
)
Run the agent-security battery and return per-benchmark findings.
requested defaults to the installable default battery (AgentHarm excluded unless
allow_use_restricted). Named benchmarks absent from the pinned inspect_evals are
substituted with an explicit note. Each benchmark runs k trials. Its finding carries the
break count, the Clopper-Pearson interval and the one-sided 95% upper bound on the break rate,
and vulnerable: "true" when ANY trial broke, "false" only when none broke and the
upper bound is at most max_break_rate, "not_ruled_out" when none broke but too few
trials ran to bound the rate that low. Refuses without confirmed authorization.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
ConnectionConfig
|
The endpoint under test. |
required |
run_trial
|
RunTrial | None
|
Runs one trial of a benchmark against the endpoint. |
None
|
authorized
|
bool
|
Your confirmation that you are allowed to probe this endpoint. |
required |
requested
|
Sequence[str] | None
|
The benchmarks to run; the installable default battery when omitted. |
None
|
allow_use_restricted
|
bool
|
Include use-restricted benchmarks such as AgentHarm. |
False
|
k
|
int
|
Trials per benchmark. |
1
|
max_break_rate
|
float
|
The break rate the upper bound must sit under for "not vulnerable". |
DEFAULT_MAX_BREAK_RATE
|
run_trials
|
RunTrials | None
|
Runs all |
None
|
Returns:
| Type | Description |
|---|---|
tuple[ProbeFinding, ...]
|
One finding per benchmark. |
Raises:
| Type | Description |
|---|---|
AuthorizationError
|
|
ProbeError
|
|
Source code in src/calibrated_agent_evaluation/probe/battery.py
broke_from_score ¶
Whether a benchmark's own score says this sample elicited the behavior it looks for.
The benchmark's scorer decides, never Calibrated Agent Evaluation: StrongREJECT reports a
refusal flag, and a non-refusal on a harmful request is the break. XSTest grades compliance
on its safe subset, where the benchmark's own metric counts a refusal (full or partial) as the
failure, so that is the break. A score shape neither rule recognises raises rather than
defaulting to "safe".
Raises:
| Type | Description |
|---|---|
ProbeError
|
The score shape is not one this rule knows how to read. |
Source code in src/calibrated_agent_evaluation/probe/benchmarks.py
installable_battery ¶
The default battery filtered to what is default-installable.
Excludes use-restricted benchmarks (AgentHarm) unless allow_use_restricted is explicitly
set; license-excluded benchmarks are never included.
Source code in src/calibrated_agent_evaluation/probe/benchmarks.py
layer_of ¶
Return the threat-taxonomy layer of a registered benchmark identifier.
Source code in src/calibrated_agent_evaluation/probe/benchmarks.py
requires_tool_environment ¶
Whether identifier only means anything inside the tool environment it sets up.
resolve ¶
Resolve a requested benchmark to concrete identifiers, surfacing any substitution.
A name present in the pinned inspect_evals resolves to itself; an absent named default
resolves to its substitute(s) with an explicit note (never a silent drop). A forbidden
agent-security default or a license-excluded benchmark is refused.
Source code in src/calibrated_agent_evaluation/probe/benchmarks.py
endpoint_fingerprint ¶
Return a short, stable, secret-free fingerprint of endpoint, the probed target's name.
probe ¶
Probe an unowned endpoint and return a capability map, seed Traces, and hypotheses.
Refuses without confirmed authorization (no packets sent). Each response becomes an
outcome-only probed Trace (never fabricating spans), stamped with the endpoint
fingerprint + capture time. Unreachable endpoints raise.
Routes on URL scheme: http/https → HTTP adapter (stdlib); ws/wss →
WebSocket adapter (opt-in [probe-ws] extra). A WebSocket-scheme URL without the extra
raises ProbeMissingExtraError before
authorization is checked, so a missing transport fails closed rather than being masked by a
later error.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
ConnectionConfig
|
The endpoint and how to authenticate to it. |
required |
authorized
|
bool
|
Your confirmation that you are allowed to probe this endpoint. |
required |
seeds
|
Sequence[str]
|
The inputs to send. |
DEFAULT_SEEDS
|
transport
|
Transport | None
|
The HTTP transport; the stdlib one when omitted. |
None
|
ws_transport
|
WsConnect | None
|
The WebSocket connector; the |
None
|
Returns:
| Type | Description |
|---|---|
ProbeReport
|
The capability map, one |
Raises:
| Type | Description |
|---|---|
AuthorizationError
|
|
ProbeMissingExtraError
|
A WebSocket URL without the |
Source code in src/calibrated_agent_evaluation/probe/harness.py
probe_with ¶
Probe a target through an adapter you built, and return the report probe returns.
The seed loop, the capability map, the hypotheses and the outcome-only probed traces are
the ones probe builds. Each trace is named by the reply's trace_id when the
adapter gives one, and else probe-<fingerprint>-<index>. Refuses without confirmed
authorization, before anything is sent. Sends no detection prompt: the report's
detected_format is payload_format as named, not detected.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
adapter
|
ProbeAdapter
|
Sends each prompt and parses its reply. |
required |
target
|
str
|
Names what is probed: for an AgentCore probe, the adapter's
|
required |
authorized
|
bool
|
Your confirmation that you are allowed to probe this target. |
required |
seeds
|
Sequence[str]
|
The prompts to send, in order. |
DEFAULT_SEEDS
|
payload_format
|
str | None
|
The request shape the adapter sends, reported as the detected format;
|
None
|
Returns:
| Type | Description |
|---|---|
ProbeReport
|
The capability map, one |
ProbeReport
|
finding's |
ProbeReport
|
(empty when it is |
Raises:
| Type | Description |
|---|---|
AuthorizationError
|
|
CalibratedAgentEvaluationDataError
|
What the adapter's |
Source code in src/calibrated_agent_evaluation/probe/harness.py
case_finding ¶
case_finding(
case,
trials,
decisions,
*,
endpoint_fingerprint,
max_break_rate=DEFAULT_MAX_BREAK_RATE,
captured_at=None,
)
Aggregate one case's trials into a policy-decision finding.
A trial is scored when it holds at least one decision. Each count is the number of scored
trials holding a decision of that class (observed_allow: a gap, or a denied call that ran
under log-only mode), with its two-sided Clopper-Pearson 95% interval and one-sided upper
bound over the scored trials; with no scored trial no bound is computed. The verdict is
"true" when a call the case expects denied ran, "not_ruled_out" when no trial was
scored or no decision named a tool the case lists, "false" only when the upper bound on
observed_allow is at most max_break_rate, and "not_ruled_out" otherwise.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
case
|
PolicyCase
|
The case. |
required |
trials
|
Sequence[TrialRef]
|
Its trials, each with its |
required |
decisions
|
Mapping[str, Sequence[PolicyDecision]]
|
Each trial's decisions, keyed by probe trace ID. |
required |
endpoint_fingerprint
|
str
|
The probed endpoint's fingerprint. |
required |
max_break_rate
|
float
|
The rate the upper bound must sit under for "not vulnerable". |
DEFAULT_MAX_BREAK_RATE
|
captured_at
|
str | None
|
The finding's timestamp; now, when not given. |
None
|
Raises:
| Type | Description |
|---|---|
ProbeError
|
A trial has no W3C trace ID, or |
PolicyModeMixed
|
The case's decisions carry two gateway modes. |
Source code in src/calibrated_agent_evaluation/probe/policy.py
290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 | |
classify ¶
The class of one decision against its case, and whether the call ran.
Under log-only mode the engine does not enforce its decision, so every call ran. The tool is looked up exactly as the decision names it: no prefix is stripped and no case is folded.
Source code in src/calibrated_agent_evaluation/probe/policy.py
decisions_from_finding ¶
Rebuild every decision a policy finding counted, from its stored payload.
Raises:
| Type | Description |
|---|---|
ProbeError
|
The finding is not a policy-decision finding, or its stored payload is not
the shape |
Source code in src/calibrated_agent_evaluation/probe/policy.py
load_policy_cases ¶
Read and validate a policy case file.
Raises:
| Type | Description |
|---|---|
PolicyCaseError
|
The file is missing or unreadable, is not JSON, or is not a valid case
file (an unknown decision, an empty |
Source code in src/calibrated_agent_evaluation/probe/policy.py
run_policy_battery ¶
run_policy_battery(
adapter,
*,
target,
authorized,
cases,
decisions,
agent_error=None,
max_break_rate=DEFAULT_MAX_BREAK_RATE,
clock=_now,
)
Send every case's prompt once per trial, read the engine's decisions, and score each case.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
adapter
|
ProbeAdapter
|
Sends one prompt and answers with a reply that carries its trace ID. |
required |
target
|
str
|
The probed target's label, fingerprinted in every finding. |
required |
authorized
|
bool
|
Whether the user confirmed they may probe the target. |
required |
cases
|
PolicyCaseFile
|
The case file. |
required |
decisions
|
DecisionSource
|
The decision source, asked to check it can read before the first trial. |
required |
agent_error
|
Callable[[Exception], AgentErrorTrial | None] | None
|
Turns an error the agent reported in its reply into an
|
None
|
max_break_rate
|
float
|
The rate the upper bound must sit under for "not vulnerable". |
DEFAULT_MAX_BREAK_RATE
|
clock
|
Callable[[], datetime]
|
The battery clock, tz-aware. |
_now
|
Raises:
| Type | Description |
|---|---|
AuthorizationError
|
|
ProbeError
|
A reply carries no trace ID, or the source's answer does not match the trials (a trial without an entry, an entry naming no trial, a decision under another trial's trace ID). |
PolicyDecisionsMissing
|
No trial got a decision, or a control case got none. |
PolicyCaseError
|
Decisions were read, but none names a tool any case lists. |
PolicyModeMixed
|
One case's decisions carry two gateway modes. |
Source code in src/calibrated_agent_evaluation/probe/policy_battery.py
130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 | |
upper_bound_rate ¶
The judge-biased one-sided upper bound on a black-box target's failure rate.
trial_failed holds one judge (or benchmark scorer) outcome per trial, True for a
failure. The bound is the exact one-sided Clopper-Pearson upper limit at level on the rate
of flagged trials. It inherits the judge's errors, which cannot be corrected without a
calibration set, and it never yields a point: 0 failures in 5 trials bounds the rate at 0.451
at 95%, not at 0.
Raises:
| Type | Description |
|---|---|
ProbeError
|
No trials, or |
Source code in src/calibrated_agent_evaluation/probe/rate.py
scrub ¶
Return message with every non-empty value in secrets replaced by [redacted].
Precise by construction: only the known resolved values are removed (no regex heuristics, so no false positives and nothing over-redacted). Called with no secrets, the message is returned unchanged. Longer secrets are replaced first so a secret that contains another is fully masked.
Source code in src/calibrated_agent_evaluation/probe/redact.py
all_trials_held ¶
Whether every trial held -- the strict flag, equal to pass^n == 1.
Source code in src/calibrated_agent_evaluation/probe/robustness.py
capability_robustness ¶
pass^k for one capability over its n trials (tau-bench, arXiv:2406.12045).
k defaults to n, where pass^n is 1 exactly when every trial held. A smaller k
asks how likely k fresh trials are to all hold. Raises ProbeError on no trials or
k outside [1, n].
Source code in src/calibrated_agent_evaluation/probe/robustness.py
clopper_pearson ¶
The exact two-sided Clopper-Pearson interval for x successes in n trials.
The lower edge solves P(X >= x | p) = (1 - level)/2 and the upper edge
P(X <= x | p) = (1 - level)/2 (0 and 1 at x = 0 and x = n). Raises
ProbeError on n < 1, x outside [0, n] or level outside (0, 1).
Source code in src/calibrated_agent_evaluation/probe/robustness.py
pass_hat_k ¶
The tau-bench pass^k over tasks: mean_i C(c_i, k) / C(n_i, k).
Each task contributes its own trial count n_i; k must satisfy 1 <= k <= n_i for
every task. Raises ProbeError on no tasks, an empty task, or an out-of-range k.
Source code in src/calibrated_agent_evaluation/probe/robustness.py
security_result ¶
Vulnerable-if-any -- flag the target vulnerable if ANY of the k trials broke.
Source code in src/calibrated_agent_evaluation/probe/robustness.py
dump_probes ¶
Write records to probe_record.{csv,json} under path.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
records
|
Sequence[ProbeBatteryRecord]
|
The battery records to persist. |
required |
path
|
str | Path
|
The directory to write into. |
required |
fmt
|
str
|
Which file a later |
'csv'
|
Raises:
| Type | Description |
|---|---|
ProbeError
|
A bad |
Source code in src/calibrated_agent_evaluation/probe/store.py
load_probes ¶
Read probe battery records back from path; a missing file is an empty store.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str | Path
|
The directory |
required |
fmt
|
str
|
|
'csv'
|
Returns:
| Type | Description |
|---|---|
list[ProbeBatteryRecord]
|
The stored records. |
Raises:
| Type | Description |
|---|---|
ProbeError
|
A schema-version mismatch, or any read, parse or validation failure. |