Skip to content

Glossary

Terms the documentation uses with a specific meaning. Most of them show the same definition when you hover them on another page.

Everyday terms

blind spot
A taxonomy cell with no observed failure yet. Either the agent is safe there or nobody has tested it. More
capability map
What probing saw an endpoint do, such as the tools its replies mentioned. Always a lower bound, never a full list. More
cassette
A file of recorded model calls. Replaying it reruns an agent or judge with no network and no credentials. More
certified
The judge agreed with your gold labels well enough, on both failures and passes, to be trusted to score. More
certify
Check a judge against a locked test set of gold labels. It passes only if it catches enough of the real failures and passes enough of the good runs, judged cautiously for how many labels you have. More
CI gate
The pass or fail calibrated-agent-evaluation run --ci gives your build. Every failure mode has to clear the threshold on a cautious reading of the traces you have, or the build fails and names the mode. More
convention
The attribute naming an agent's OpenTelemetry spans follow, such as GenAI, OpenInference or OpenLLMetry. Ingest detects it and maps it to one trace shape. More
crew
Several agents working on one task and handing work to each other. Its trace has one span per agent step, linked by parent. More
culprit
In a multi-agent trace, the first step that produced a wrong output from the input it was given. More
discovery
Traces picked to find failures, covering many combinations of situations. Never used to measure a failure rate. More
failure mode
A named kind of failure, such as misreading an amount. Labels are grouped into failure modes, and rates are reported per mode. More
gold label
A pass or fail with a written critique, recorded by a human. Judges and rates are checked against gold labels; a model may pre-fill one but never replaces it. More
hunted
Picked on purpose to find failures, so the set holds far more of them than production does. Used to check a judge, never to measure a rate. More
in-cell
A trace inside the taxonomy cell a command was limited to with --cell. More
label
A pass or fail verdict on a trace, usually with a short critique. A gold label is one a human recorded. More
LLM judge
A model prompted to grade an agent's trace as pass or fail. Calibrated Agent Evaluation checks it against gold labels before trusting its verdicts. More
locked test set
The gold-labeled traces a judge is certified on. Each one can certify a judge once. More
manifestation
The step where a crew's failure becomes visible, often downstream of the culprit: a payout agent paying a claim that intake misread. More
owned
A trace from an agent you instrument. Its step-by-step spans let Calibrated Agent Evaluation point at the step that caused a failure. More
probed
A trace from an endpoint you can only call: input and output, no internal steps, so analysis stays at the outcome. More
production-representative
Traces picked to match real traffic. The only kind a failure rate is measured on. More
rate sample
A random sample of production traces, scored by the judge, whose flag rate gets corrected. Never the gold-labeled set. More
relocation
A fix that lowers one agent's failures while the crew's overall outcome does not improve: the failure moved instead of going away. calibrated-agent-evaluation improve rejects it. More
sampling frame
Which population a trace was drawn from: discovery (picked to find failures) or production-representative (picked to match real traffic). Rates are only measured on production-representative traces. More
span
One agent step inside a trace: what it received, the tool calls it made and what it handed on. More
taxonomy cell
One combination of the situations an agent handles, for example feature, persona and scenario. Each trace records the cell it exercises. More
trace
The record of one agent run: its input, its output, and the steps (spans) in between. More

Statistical terms

CALM
The set of judge biases Calibrated Agent Evaluation checks: length, self-preference and sycophancy must pass; distraction, bandwagon and compassion fade are recorded; positional applies to judges that compare two answers. More
coverage
How often a range contains the true value across repeated studies. A 95% range should contain it 95 times in 100. Good–Turing coverage, used when clustering, is a different quantity: the chance a new failure falls in a mode already seen. More
coverage floor
The fewest gold labels per class at which the corrected-rate range was measured to hold up, for a judge of a given accuracy: 35 at 0.90, 162 at 0.99. Certifying can need more: a 0.90 judge needs 55 per class to certify, so the plan for it is 55. A stronger rerun found a few gaps above the floor. More
intersection-union
A gate that passes only when every check passes, such as the CI gate across failure modes. It needs no correction for running many checks. More
placebo
In the bias check, the unchanged input judged again, to measure how often the judge flips its verdict when nothing changed. More
rate-safe
The judge was certified on enough gold labels, for the accuracy it showed, to be given a corrected rate. More
rate-unsafe
The judge is certified to score but was checked on too few gold labels, for the accuracy it showed, to be given a corrected rate. More
Rogan–Gladen
The correction that undoes a judge's known false alarms and misses to turn its flag rate into an estimate of the true failure rate. More
TNR
True-negative rate: of the traces that really passed, the share the judge left alone. More
TPR
True-positive rate: of the traces that really failed, the share the judge flagged. More
Wilson bound
The lowest value a rate could plausibly have, given how many traces it was measured on. Gates read this bound, so a small sample cannot pass on a lucky streak. More
Youden's J
TPR + TNR - 1, how much better than a coin the judge is. Below 0.5 the corrected rate is too unstable to report as a single number, so only its range is given. More