Versioning and stability¶
The current release is 0.1.0, published on PyPI as calibrated-agent-evaluation.
Every release is listed in the changelog.
What pre-1.0 means here¶
- The project follows Semantic Versioning. Before 1.0, a minor release
(0.1 to 0.2) may break the Python API, the
calibrated-agent-evaluationcommand line, or a persisted record format, and there is no deprecation window yet. Patch releases do not break them. - Breaking changes are listed in the changelog. Read it before you upgrade if you depend on anything the library writes to disk.
- If you build on it now, pin the minor version you tested against
(
calibrated-agent-evaluation~=0.1.0).
Persisted formats¶
Every record the library writes carries a schema version, and every reader refuses a record whose version differs from the one it was built for. Nothing is converted silently, so an old store or dump fails loudly after an upgrade instead of being misread. The probe record is the exception: it is stamped with its version, but no reader checks it yet.
| Record | Defined in | Schema version |
|---|---|---|
Trace, Span, Label (the trace store and calibrated-agent-evaluation dump) |
calibrated_agent_evaluation.schema |
1 |
| Calibration record | calibrated_agent_evaluation.calibrate |
7 |
| Bias battery record | calibrated_agent_evaluation.bias |
4 |
| Run record | calibrated_agent_evaluation.run |
7 |
| Improve record | calibrated_agent_evaluation.improve |
6 |
| Flagship report | calibrated_agent_evaluation.flagship |
4 |
| Judge definitions and enforcement decisions | calibrated_agent_evaluation.enforce |
1 |
| Judge run records and the verdict cache | calibrated_agent_evaluation.judges |
1 |
| Probe record | calibrated_agent_evaluation.probe |
1 |
Dump manifest (dump-manifest.json) |
calibrated_agent_evaluation.dumpload |
1 |
These values are the *_SCHEMA_VERSION constants in the code at the time of writing:
calibrated_agent_evaluation.schema.SCHEMA_VERSION, calibrated_agent_evaluation.calibrate.record.CALIB_SCHEMA_VERSION, and so on.
The dump manifest's is calibrated_agent_evaluation.dumpload.io.DUMP_FORMAT_VERSION.
A store or dump written under an older version has to be regenerated with the release that
reads it. There is no migration tool: calibrated-agent-evaluation load refuses a dump whose version differs, as
Troubleshooting shows.
The fixture corpus under fixtures/ carries its own VERSION file, which must equal the trace
schema version the code runs.
Dependency pins¶
Most extras pin a floor at the version whose API was checked when the adapter was written, and cap
the next major version. datasets and hf keep older floors than the versions they were checked
against. Three are narrower:
| Extra | Pin | Why |
|---|---|---|
ui |
python-fasthtml>=0.14,<0.15 |
FastHTML is pre-1.0, so a minor release may break. The cap stops one arriving unreviewed. |
inspect |
inspect-ai>=0.3.242,<0.4 |
Same reason: a 0.x package, capped at the next minor. The bedrock extra carries the same pin. |
probe |
inspect_evals>=0.14.3,<0.15, inspect-ai>=0.3.242,<0.3.288 |
Also 0.x; the benchmark battery reads task names that a minor release can rename. inspect_evals 0.14 imports a symbol inspect-ai 0.3.288 removed. |
synth, agentcore |
strands-agents-evals>=1.6.0,<1.7 |
The AgentCore failure detectors are verified and recorded against the 1.6 detector prompt and failure vocabulary; 1.10 changed both. |
Raising any of these means re-checking the adapter against the new version first.
The checkout's uv.lock pins the whole development tree. The committed coverage figures under
fixtures/expected/coverage/ record the numpy, scipy and statsmodels versions they were measured
with, so uv sync --locked from the same commit reproduces the environment behind them.
The documentation¶
This site is built from the main branch and describes that version only. There is no version
selector yet.