Write an improve request¶
Check a fix whose before-and-after results you measured outside this toolkit.
calibrated-agent-evaluation improve decides whether a fix really lowered a failure rate (see
Check that a fix worked). When the
before and after verdicts both live in this library's stores, calibrated-agent-evaluation improve
--evaluate pairs them for you. When you measured them elsewhere, write them to a request file and
pass it with --request.
Steps¶
1. Register the bar before the fix exists¶
This freezes the failure mode and the smallest drop in failure rate that counts as a fix. The bar is read from the stored record and not from the request, so seeing the results can't move it.
2. Run the same held-out traces before and after the fix¶
Write one pair per trace.
3. Evaluate the request¶
$ calibrated-agent-evaluation --root store improve --register trusted_broken_policy_tool --min-reduction 0.02
Pre-registered 'trusted_broken_policy_tool': the fix must cut its failure rate by at least 2 percentage points. The threshold is now frozen.
Next: after the fix, compare the run before it with the run after it: `calibrated-agent-evaluation improve --evaluate trusted_broken_policy_tool --before-root A --after-root B --development FILE`.
Run with --explain for the numbers behind this.
(exit 0)
$ calibrated-agent-evaluation --root store improve --request request.json
Fix accepted: the failure rate dropped by at least the 2 percentage points you pre-registered, and nothing it was checked against got worse.
Not checked: whether the target was registered before the after run began (a request file carries no run times).
Run with --explain for the numbers behind this.
(exit 0)
This is the request file it evaluated:
{
"target_metric": "trusted_broken_policy_tool",
"pairs": [
{
"trace_id": "ORD-conf-0",
"before_fail": true,
"after_fail": true,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-1",
"before_fail": true,
"after_fail": true,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-2",
"before_fail": true,
"after_fail": true,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-3",
"before_fail": true,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-4",
"before_fail": true,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-5",
"before_fail": true,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-6",
"before_fail": true,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-7",
"before_fail": true,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-8",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-9",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-10",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-11",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-12",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-13",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-14",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-15",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-16",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-17",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-18",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
},
{
"trace_id": "ORD-conf-19",
"before_fail": false,
"after_fail": false,
"labels_are_judge_produced": false
}
],
"development_ids": [
"ORD-1001",
"ORD-1002"
],
"slices": [
{
"name": "misread_amount",
"kind": "other",
"regression_p": 0.7,
"rate_before": {
"point": 0.06,
"lo": 0.01,
"hi": 0.11,
"method": "wilson",
"level": 0.95,
"sided": "two-sided",
"n": 100
},
"rate_after": {
"point": 0.06,
"lo": 0.01,
"hi": 0.11,
"method": "wilson",
"level": 0.95,
"sided": "two-sided",
"n": 100
}
}
]
}
The fields¶
| Field | What it holds |
|---|---|
target_metric |
The registered failure mode. |
pairs |
One object per held-out trace: trace_id, before_fail and after_fail (booleans), and labels_are_judge_produced. |
confirmation_ids |
The traces the decision is read from. Optional; defaults to every trace in pairs. |
development_ids |
Required. The traces you looked at while writing the fix. They must not appear among the confirmation traces. |
slices |
The guard panel: other things the fix must not make worse. At least one is required. |
Each slice has a name, a kind (target, agent, handoff or other), a regression_p,
and rate_before and rate_after. regression_p is your own test's answer to "did this slice
get worse?", as a probability: small means it probably did. Each rate is an object with the value
(point), its likely range (lo, hi), how that range was computed (method, level, and
sided, which is two-sided, lower or upper) and the number of traces behind it (n).
When labels_are_judge_produced is true, every pair where the verdict changed needs a human
label on that trace in the store, which settles it, so a judge doesn't grade its own fix.
The detail behind the verdict¶
$ calibrated-agent-evaluation --root store improve --register trusted_broken_policy_tool --min-reduction 0.02
Pre-registered 'trusted_broken_policy_tool': the fix must cut its failure rate by at least 2 percentage points. The threshold is now frozen.
Next: after the fix, compare the run before it with the run after it: `calibrated-agent-evaluation improve --evaluate trusted_broken_policy_tool --before-root A --after-root B --development FILE`.
Run with --explain for the numbers behind this.
(exit 0)
$ calibrated-agent-evaluation --root store improve --request request.json --explain
Fix accepted: the failure rate dropped by at least the 2 percentage points you pre-registered, and nothing it was checked against got worse.
Not checked: whether the target was registered before the after run began (a request file carries no run times).
improve 'trusted_broken_policy_tool': ACCEPTED
paired fail-rate change (after - before) -0.250, 95% CI [-0.449, -0.029] (newcombe_paired_hybrid_score, n=20); McNemar midp, 5 discordant pair(s), p=0.03125
pre-registered minimum reduction 0.02: the interval's upper bound must be at or below -0.020
guard panel: 1 slice(s), Holm; regressed: none
pre-registration frozen at <time>; not checked against the after run: a request file carries no run times
recorded improve-<id> in store/records/improve
(exit 0)
The fix is accepted when the whole range of the change in failure rate sits past the registered drop (here, the range ends at -0.029, past the -0.020 bar), and no guard slice got worse, allowing for the fact that several slices were checked at once.
Pitfalls¶
slices can't be empty¶
With nothing to check against, the request can't show that nothing got worse, so improve stops
and asks for at least one slice.
Registering after the fact¶
A metric registers once. Register before you start on the fix.
Overlapping development and confirmation traces¶
A trace in both lists stops the evaluation: a fix scored on the traces it was written against says nothing about new traffic.
Crews¶
A request file cannot carry what a crew check needs (which agent owned each failure before and
after), so --request does not accept a crew metric. Evaluate a crew fix from two stores with
--evaluate, --crew-file and --touched-agent;
Evaluate a multi-agent crew shows how.