calibrated_agent_evaluation.subjective¶
calibrated_agent_evaluation.subjective ¶
Subjective-outcome handling engine.
Public API re-exports; import from calibrated_agent_evaluation.subjective.
SubjectiveError ¶
Bases: CalibratedAgentEvaluationDataError
Invalid input or state in the subjective-outcome module.
SubjectiveGateError ¶
Bases: SubjectiveError
Attempt to gate a CI decision on a subjective dimension.
SubjectiveStoreError ¶
Bases: SubjectiveError
A preference record file could not be read or written, or holds a malformed record.
PreferenceRecord
pydantic-model
¶
Bases: CalibratedAgentEvaluationModel
One stored pairwise preference: trace A or trace B was better on one subjective dimension.
source is "file" for calibrated-agent-evaluation prefer --pairs and "review" for
the A/B page; source_ref locates it ("<file>:<line>", or the review session).
rater_id names the human who chose, when one did. Tracked, never gated: no CI decision reads
these records.
Config:
frozen:Trueextra:forbid
Fields:
-
preference_id(str) -
criterion(str) -
dimension(str) -
trace_id_a(str) -
trace_id_b(str) -
winner(Literal['a', 'b']) -
judge_kind(PairJudgeKind) -
rater_id(str | None) -
source(PreferenceSource) -
source_ref(str) -
schema_version(str) -
provenance(ProvenanceStamp)
pair ¶
The library pair this record stores.
Source code in src/calibrated_agent_evaluation/subjective/store.py
Atom
dataclass
¶
AtomSet
dataclass
¶
BradleyTerryRating
dataclass
¶
BradleyTerryRatings
dataclass
¶
EloRating
dataclass
¶
One trace's sequential-Elo display score for one dimension.
Display only: sequential Elo depends on the order the pairs were processed in and has no
interval. bradley_terry is the estimator of
record.
EloRatings
dataclass
¶
EloRatings(
dimension,
ratings,
n_total_comparisons,
treatment="tracked",
k_factor=32.0,
initial_rating=1500.0,
)
All sequential-Elo display ratings for one subjective dimension.
k_factor and initial_rating are the update hyperparameters, recorded so the numbers
can be reproduced given the same pairs in the same order.
PreferencePair
dataclass
¶
SubjectiveDimension
dataclass
¶
atom_set_from_dict ¶
Build an AtomSet from a decoded atom-set file.
Raises:
| Type | Description |
|---|---|
SubjectiveError
|
The document is not an object of the shape above, or the split is invalid. |
Source code in src/calibrated_agent_evaluation/subjective/atomfile.py
load_atom_set ¶
Read and check the atom-set file at path.
Raises:
| Type | Description |
|---|---|
SubjectiveError
|
The file cannot be read, is not JSON, or does not declare a valid split. |
Source code in src/calibrated_agent_evaluation/subjective/atomfile.py
define_atoms ¶
Define an AtomSet from objective atoms and subjective dimensions.
Each tuple is (name, description). Raises SubjectiveError if names overlap.
Source code in src/calibrated_agent_evaluation/subjective/classify.py
is_objective ¶
True if dimension is classified as objective in the AtomSet.
is_subjective ¶
True if dimension is classified as subjective in the AtomSet.
compute_ratings ¶
Compute Elo ratings from preference pairs.
Returns EloRatings with treatment="tracked" (never "gated"), carrying k_factor and
initial_rating. Returns empty ratings if pairs is empty. The result depends on the order
of pairs; bradley_terry does not.
Source code in src/calibrated_agent_evaluation/subjective/elo.py
validate_gateable ¶
Confirm a dimension is objective (gatable). Raises if subjective or unknown.
Passes silently if objective. Raises SubjectiveGateError if subjective. Raises SubjectiveError if unknown.
Source code in src/calibrated_agent_evaluation/subjective/gate.py
read_preference_file ¶
Every preference in the file at path with its 1-based line number, checked.
Raises:
| Type | Description |
|---|---|
SubjectiveError
|
The file cannot be read, holds no preference, or any line is not valid
JSON, lacks a field, carries an unknown one, or fails |
Source code in src/calibrated_agent_evaluation/subjective/pairfile.py
record_preference ¶
Validate a preference pair against the AtomSet and return it.
Raises SubjectiveError if: - winner is not 'a' or 'b' - trace_id_a or trace_id_b is empty - dimension is objective - dimension is unknown - trace_id_a == trace_id_b
Source code in src/calibrated_agent_evaluation/subjective/preference.py
dump_preference_records ¶
Write records to preference_record.{csv,json} under path.
Raises:
| Type | Description |
|---|---|
SubjectiveStoreError
|
A bad |
Source code in src/calibrated_agent_evaluation/subjective/store.py
load_preference_records ¶
Read preference records back from path; a missing file is an empty store.
Raises:
| Type | Description |
|---|---|
SubjectiveStoreError
|
A schema-version mismatch, or any read, parse or validation failure. |
Source code in src/calibrated_agent_evaluation/subjective/store.py
preference_record ¶
Validate pair against atom_set and return it as a record to store.
Raises:
| Type | Description |
|---|---|
SubjectiveError
|
The pair names an objective or undeclared dimension, compares a trace
with itself, or has a winner other than |
SubjectiveStoreError
|
|