Glossary¶
Terms the documentation uses with a specific meaning. Most of them show the same definition when you hover them on another page.
Everyday terms¶
- blind spot
- A taxonomy cell with no observed failure yet. Either the agent is safe there or nobody has tested it. More
- capability map
- What probing saw an endpoint do, such as the tools its replies mentioned. Always a lower bound, never a full list. More
- cassette
- A file of recorded model calls. Replaying it reruns an agent or judge with no network and no credentials. More
- certified
- The judge agreed with your gold labels well enough, on both failures and passes, to be trusted to score. More
- certify
- Check a judge against a locked test set of gold labels. It passes only if it catches enough of the real failures and passes enough of the good runs, judged cautiously for how many labels you have. More
- CI gate
- The pass or fail calibrated-agent-evaluation run --ci gives your build. Every failure mode has to clear the threshold on a cautious reading of the traces you have, or the build fails and names the mode. More
- convention
- The attribute naming an agent's OpenTelemetry spans follow, such as GenAI, OpenInference or OpenLLMetry. Ingest detects it and maps it to one trace shape. More
- crew
- Several agents working on one task and handing work to each other. Its trace has one span per agent step, linked by parent. More
- culprit
- In a multi-agent trace, the first step that produced a wrong output from the input it was given. More
- discovery
- Traces picked to find failures, covering many combinations of situations. Never used to measure a failure rate. More
- failure mode
- A named kind of failure, such as misreading an amount. Labels are grouped into failure modes, and rates are reported per mode. More
- gold label
- A pass or fail with a written critique, recorded by a human. Judges and rates are checked against gold labels; a model may pre-fill one but never replaces it. More
- hunted
- Picked on purpose to find failures, so the set holds far more of them than production does. Used to check a judge, never to measure a rate. More
- in-cell
- A trace inside the taxonomy cell a command was limited to with --cell. More
- label
- A pass or fail verdict on a trace, usually with a short critique. A gold label is one a human recorded. More
- LLM judge
- A model prompted to grade an agent's trace as pass or fail. Calibrated Agent Evaluation checks it against gold labels before trusting its verdicts. More
- locked test set
- The gold-labeled traces a judge is certified on. Each one can certify a judge once. More
- manifestation
- The step where a crew's failure becomes visible, often downstream of the culprit: a payout agent paying a claim that intake misread. More
- owned
- A trace from an agent you instrument. Its step-by-step spans let Calibrated Agent Evaluation point at the step that caused a failure. More
- probed
- A trace from an endpoint you can only call: input and output, no internal steps, so analysis stays at the outcome. More
- production-representative
- Traces picked to match real traffic. The only kind a failure rate is measured on. More
- rate sample
- A random sample of production traces, scored by the judge, whose flag rate gets corrected. Never the gold-labeled set. More
- relocation
- A fix that lowers one agent's failures while the crew's overall outcome does not improve: the failure moved instead of going away. calibrated-agent-evaluation improve rejects it. More
- sampling frame
- Which population a trace was drawn from: discovery (picked to find failures) or production-representative (picked to match real traffic). Rates are only measured on production-representative traces. More
- span
- One agent step inside a trace: what it received, the tool calls it made and what it handed on. More
- taxonomy cell
- One combination of the situations an agent handles, for example feature, persona and scenario. Each trace records the cell it exercises. More
- trace
- The record of one agent run: its input, its output, and the steps (spans) in between. More
Statistical terms¶
- CALM
- The set of judge biases Calibrated Agent Evaluation checks: length, self-preference and sycophancy must pass; distraction, bandwagon and compassion fade are recorded; positional applies to judges that compare two answers. More
- coverage
- How often a range contains the true value across repeated studies. A 95% range should contain it 95 times in 100. Good–Turing coverage, used when clustering, is a different quantity: the chance a new failure falls in a mode already seen. More
- coverage floor
- The fewest gold labels per class at which the corrected-rate range was measured to hold up, for a judge of a given accuracy: 35 at 0.90, 162 at 0.99. Certifying can need more: a 0.90 judge needs 55 per class to certify, so the plan for it is 55. A stronger rerun found a few gaps above the floor. More
- intersection-union
- A gate that passes only when every check passes, such as the CI gate across failure modes. It needs no correction for running many checks. More
- placebo
- In the bias check, the unchanged input judged again, to measure how often the judge flips its verdict when nothing changed. More
- rate-safe
- The judge was certified on enough gold labels, for the accuracy it showed, to be given a corrected rate. More
- rate-unsafe
- The judge is certified to score but was checked on too few gold labels, for the accuracy it showed, to be given a corrected rate. More
- Rogan–Gladen
- The correction that undoes a judge's known false alarms and misses to turn its flag rate into an estimate of the true failure rate. More
- TNR
- True-negative rate: of the traces that really passed, the share the judge left alone. More
- TPR
- True-positive rate: of the traces that really failed, the share the judge flagged. More
- Wilson bound
- The lowest value a rate could plausibly have, given how many traces it was measured on. Gates read this bound, so a small sample cannot pass on a lucky streak. More
- Youden's J
- TPR + TNR - 1, how much better than a coin the judge is. Below 0.5 the corrected rate is too unstable to report as a single number, so only its range is given. More