Skip to content

calibrated_agent_evaluation.subjective

calibrated_agent_evaluation.subjective

Subjective-outcome handling engine.

Public API re-exports; import from calibrated_agent_evaluation.subjective.

SUBJECTIVE_SCHEMA_VERSION module-attribute

SUBJECTIVE_SCHEMA_VERSION = '1'

SubjectiveError

Bases: CalibratedAgentEvaluationDataError

Invalid input or state in the subjective-outcome module.

SubjectiveGateError

Bases: SubjectiveError

Attempt to gate a CI decision on a subjective dimension.

SubjectiveStoreError

Bases: SubjectiveError

A preference record file could not be read or written, or holds a malformed record.

PreferenceRecord pydantic-model

Bases: CalibratedAgentEvaluationModel

One stored pairwise preference: trace A or trace B was better on one subjective dimension.

source is "file" for calibrated-agent-evaluation prefer --pairs and "review" for the A/B page; source_ref locates it ("<file>:<line>", or the review session). rater_id names the human who chose, when one did. Tracked, never gated: no CI decision reads these records.

Config:

  • frozen: True
  • extra: forbid

Fields:

preference_id pydantic-field

preference_id

criterion pydantic-field

criterion

dimension pydantic-field

dimension

trace_id_a pydantic-field

trace_id_a

trace_id_b pydantic-field

trace_id_b

winner pydantic-field

winner

judge_kind pydantic-field

judge_kind

rater_id pydantic-field

rater_id = None

source pydantic-field

source

source_ref pydantic-field

source_ref

schema_version pydantic-field

schema_version = SUBJECTIVE_SCHEMA_VERSION

provenance pydantic-field

provenance

pair

pair()

The library pair this record stores.

Source code in src/calibrated_agent_evaluation/subjective/store.py
def pair(self) -> PreferencePair:
    """The library pair this record stores."""
    return PreferencePair(
        trace_id_a=self.trace_id_a,
        trace_id_b=self.trace_id_b,
        dimension=self.dimension,
        winner=self.winner,
        judge_kind=self.judge_kind,
    )

Atom dataclass

Atom(name, description='')

One objective binary dimension (gatable via the calibration gate).

name instance-attribute

name

description class-attribute instance-attribute

description = ''

AtomSet dataclass

AtomSet(criterion, objective_atoms, subjective_dimensions)

Declares the gate/track split for one evaluation criterion.

criterion instance-attribute

criterion

objective_atoms instance-attribute

objective_atoms

subjective_dimensions instance-attribute

subjective_dimensions

BradleyTerryRating dataclass

BradleyTerryRating(trace_id, rating, n_comparisons)

One trace's Bradley-Terry rating (Elo scale) with its 95% Wald interval.

trace_id instance-attribute

trace_id

rating instance-attribute

rating

n_comparisons instance-attribute

n_comparisons

BradleyTerryRatings dataclass

BradleyTerryRatings(
    dimension,
    ratings,
    n_total_comparisons,
    treatment="tracked",
)

All Bradley-Terry ratings for one subjective dimension, best first. Tracked, never gated.

dimension instance-attribute

dimension

ratings instance-attribute

ratings

n_total_comparisons instance-attribute

n_total_comparisons

treatment class-attribute instance-attribute

treatment = 'tracked'

EloRating dataclass

EloRating(trace_id, rating, n_comparisons)

One trace's sequential-Elo display score for one dimension.

Display only: sequential Elo depends on the order the pairs were processed in and has no interval. bradley_terry is the estimator of record.

trace_id instance-attribute

trace_id

rating instance-attribute

rating

n_comparisons instance-attribute

n_comparisons

EloRatings dataclass

EloRatings(
    dimension,
    ratings,
    n_total_comparisons,
    treatment="tracked",
    k_factor=32.0,
    initial_rating=1500.0,
)

All sequential-Elo display ratings for one subjective dimension.

k_factor and initial_rating are the update hyperparameters, recorded so the numbers can be reproduced given the same pairs in the same order.

dimension instance-attribute

dimension

ratings instance-attribute

ratings

n_total_comparisons instance-attribute

n_total_comparisons

treatment class-attribute instance-attribute

treatment = 'tracked'

k_factor class-attribute instance-attribute

k_factor = 32.0

initial_rating class-attribute instance-attribute

initial_rating = 1500.0

PreferencePair dataclass

PreferencePair(
    trace_id_a, trace_id_b, dimension, winner, judge_kind
)

One pairwise preference judgment.

trace_id_a instance-attribute

trace_id_a

trace_id_b instance-attribute

trace_id_b

dimension instance-attribute

dimension

winner instance-attribute

winner

judge_kind instance-attribute

judge_kind

SubjectiveDimension dataclass

SubjectiveDimension(name, description='')

One subjective dimension (tracked via preference/Elo only).

name instance-attribute

name

description class-attribute instance-attribute

description = ''

atom_set_from_dict

atom_set_from_dict(doc)

Build an AtomSet from a decoded atom-set file.

Raises:

Type Description
SubjectiveError

The document is not an object of the shape above, or the split is invalid.

Source code in src/calibrated_agent_evaluation/subjective/atomfile.py
def atom_set_from_dict(doc: object) -> AtomSet:
    """Build an :class:`AtomSet` from a decoded atom-set file.

    Raises:
        SubjectiveError: The document is not an object of the shape above, or the split is
            invalid.
    """
    parsed = _AtomFile.model_validate(doc)
    if not parsed.criterion.strip():
        raise SubjectiveError("an atom set needs a non-empty criterion")
    return define_atoms(
        parsed.criterion,
        [(e.name, e.description) for e in parsed.objective],
        [(e.name, e.description) for e in parsed.subjective],
    )

load_atom_set

load_atom_set(path)

Read and check the atom-set file at path.

Raises:

Type Description
SubjectiveError

The file cannot be read, is not JSON, or does not declare a valid split.

Source code in src/calibrated_agent_evaluation/subjective/atomfile.py
def load_atom_set(path: str | Path) -> AtomSet:
    """Read and check the atom-set file at ``path``.

    Raises:
        SubjectiveError: The file cannot be read, is not JSON, or does not declare a valid split.
    """
    file = Path(path)
    try:
        doc = json.loads(file.read_text(encoding="utf-8"))
    except OSError as exc:
        raise SubjectiveError(f"cannot read the atom set {file}: {exc.strerror or exc}") from exc
    except (UnicodeDecodeError, json.JSONDecodeError) as exc:
        raise SubjectiveError(f"the atom set {file} is not valid JSON: {exc}") from exc
    try:
        return atom_set_from_dict(doc)
    except SubjectiveError as exc:
        raise SubjectiveError(f"the atom set {file} is not valid: {exc}") from exc

define_atoms

define_atoms(criterion, objective, subjective)

Define an AtomSet from objective atoms and subjective dimensions.

Each tuple is (name, description). Raises SubjectiveError if names overlap.

Source code in src/calibrated_agent_evaluation/subjective/classify.py
def define_atoms(
    criterion: str,
    objective: list[tuple[str, str]],
    subjective: list[tuple[str, str]],
) -> AtomSet:
    """Define an AtomSet from objective atoms and subjective dimensions.

    Each tuple is (name, description). Raises SubjectiveError if names overlap.
    """
    if not objective and not subjective:
        raise SubjectiveError(
            "AtomSet must have at least 1 objective atom or 1 subjective dimension"
        )

    # Validate no empty/whitespace names
    for name, _ in [*objective, *subjective]:
        if not name or not name.strip():
            raise SubjectiveError("Dimension names must be non-empty strings")

    # Validate no duplicates within the same list
    obj_names_list = [name for name, _ in objective]
    subj_names_list = [name for name, _ in subjective]
    obj_names = set(obj_names_list)
    subj_names = set(subj_names_list)

    if len(obj_names) != len(obj_names_list):
        dupes = [n for n in obj_names_list if obj_names_list.count(n) > 1]
        raise SubjectiveError(f"Duplicate name {dupes[0]!r} in objective atoms")

    if len(subj_names) != len(subj_names_list):
        dupes = [n for n in subj_names_list if subj_names_list.count(n) > 1]
        raise SubjectiveError(f"Duplicate name {dupes[0]!r} in subjective dimensions")

    overlap = obj_names & subj_names
    if overlap:
        raise SubjectiveError(
            f"Dimension {sorted(overlap)[0]!r} appears in both objective and subjective "
            f"lanes; each must be in exactly one lane"
        )

    return AtomSet(
        criterion=criterion,
        objective_atoms=tuple(Atom(name=n, description=d) for n, d in objective),
        subjective_dimensions=tuple(
            SubjectiveDimension(name=n, description=d) for n, d in subjective
        ),
    )

is_objective

is_objective(dimension, atom_set)

True if dimension is classified as objective in the AtomSet.

Source code in src/calibrated_agent_evaluation/subjective/classify.py
def is_objective(dimension: str, atom_set: AtomSet) -> bool:
    """True if dimension is classified as objective in the AtomSet."""
    return any(a.name == dimension for a in atom_set.objective_atoms)

is_subjective

is_subjective(dimension, atom_set)

True if dimension is classified as subjective in the AtomSet.

Source code in src/calibrated_agent_evaluation/subjective/classify.py
def is_subjective(dimension: str, atom_set: AtomSet) -> bool:
    """True if dimension is classified as subjective in the AtomSet."""
    return any(d.name == dimension for d in atom_set.subjective_dimensions)

compute_ratings

compute_ratings(
    dimension,
    pairs,
    *,
    k_factor=32.0,
    initial_rating=1500.0,
)

Compute Elo ratings from preference pairs.

Returns EloRatings with treatment="tracked" (never "gated"), carrying k_factor and initial_rating. Returns empty ratings if pairs is empty. The result depends on the order of pairs; bradley_terry does not.

Source code in src/calibrated_agent_evaluation/subjective/elo.py
def compute_ratings(
    dimension: str,
    pairs: list[PreferencePair],
    *,
    k_factor: float = 32.0,
    initial_rating: float = 1500.0,
) -> EloRatings:
    """Compute Elo ratings from preference pairs.

    Returns EloRatings with treatment="tracked" (never "gated"), carrying ``k_factor`` and
    ``initial_rating``. Returns empty ratings if pairs is empty. The result depends on the order
    of ``pairs``; :func:`calibrated_agent_evaluation.subjective.bradley_terry` does not.
    """
    if k_factor <= 0:
        raise SubjectiveError(f"k_factor must be positive; got {k_factor}")

    if not pairs:
        return EloRatings(
            dimension=dimension,
            ratings=(),
            n_total_comparisons=0,
            treatment="tracked",
            k_factor=k_factor,
            initial_rating=initial_rating,
        )

    ratings: dict[str, float] = {}
    comparisons: dict[str, int] = {}

    for pair in pairs:
        if pair.dimension != dimension:
            raise SubjectiveError(
                f"compute_ratings({dimension!r}) received a pair for dimension "
                f"{pair.dimension!r}; all pairs must match the requested dimension"
            )

        if pair.winner not in ("a", "b"):
            raise SubjectiveError(f"winner must be 'a' or 'b'; got {pair.winner!r}")

        if pair.trace_id_a == pair.trace_id_b:
            raise SubjectiveError(
                f"Self-comparison in pair (trace_id={pair.trace_id_a!r}); "
                f"cannot compute Elo for a trace against itself"
            )

        for tid in (pair.trace_id_a, pair.trace_id_b):
            if tid not in ratings:
                ratings[tid] = initial_rating
                comparisons[tid] = 0

        ra = ratings[pair.trace_id_a]
        rb = ratings[pair.trace_id_b]

        ea = 1.0 / (1.0 + 10.0 ** ((rb - ra) / 400.0))
        eb = 1.0 - ea

        if pair.winner == "a":
            sa, sb = 1.0, 0.0
        else:
            sa, sb = 0.0, 1.0

        ratings[pair.trace_id_a] = ra + k_factor * (sa - ea)
        ratings[pair.trace_id_b] = rb + k_factor * (sb - eb)
        comparisons[pair.trace_id_a] += 1
        comparisons[pair.trace_id_b] += 1

    sorted_ratings = sorted(ratings.items(), key=lambda x: x[1], reverse=True)
    elo_ratings = tuple(
        EloRating(trace_id=tid, rating=score, n_comparisons=comparisons[tid])
        for tid, score in sorted_ratings
    )

    return EloRatings(
        dimension=dimension,
        ratings=elo_ratings,
        n_total_comparisons=len(pairs),
        treatment="tracked",
        k_factor=k_factor,
        initial_rating=initial_rating,
    )

validate_gateable

validate_gateable(dimension, atom_set)

Confirm a dimension is objective (gatable). Raises if subjective or unknown.

Passes silently if objective. Raises SubjectiveGateError if subjective. Raises SubjectiveError if unknown.

Source code in src/calibrated_agent_evaluation/subjective/gate.py
def validate_gateable(dimension: str, atom_set: AtomSet) -> None:
    """Confirm a dimension is objective (gatable). Raises if subjective or unknown.

    Passes silently if objective. Raises SubjectiveGateError if subjective.
    Raises SubjectiveError if unknown.
    """
    if is_subjective(dimension, atom_set):
        raise SubjectiveGateError(
            f"Dimension {dimension!r} is subjective "
            f"and is tracked via Elo ranking only, not gatable as a CI gate"
        )

    if not is_objective(dimension, atom_set):
        raise SubjectiveError(
            f"Dimension {dimension!r} not found in AtomSet "
            f"for criterion {atom_set.criterion!r}; declare it first"
        )

read_preference_file

read_preference_file(path, atom_set)

Every preference in the file at path with its 1-based line number, checked.

Raises:

Type Description
SubjectiveError

The file cannot be read, holds no preference, or any line is not valid JSON, lacks a field, carries an unknown one, or fails record_preference (an objective or undeclared dimension, a trace compared with itself).

Source code in src/calibrated_agent_evaluation/subjective/pairfile.py
def read_preference_file(path: str | Path, atom_set: AtomSet) -> list[tuple[int, PreferencePair]]:
    """Every preference in the file at ``path`` with its 1-based line number, checked.

    Raises:
        SubjectiveError: The file cannot be read, holds no preference, or any line is not valid
            JSON, lacks a field, carries an unknown one, or fails ``record_preference`` (an
            objective or undeclared dimension, a trace compared with itself).
    """
    file = Path(path)
    try:
        text = file.read_text(encoding="utf-8")
    except OSError as exc:
        raise SubjectiveError(f"cannot read the pairs file {file}: {exc.strerror or exc}") from exc
    except UnicodeDecodeError as exc:
        raise SubjectiveError(f"the pairs file {file} is not UTF-8 text: {exc}") from exc
    pairs: list[tuple[int, PreferencePair]] = []
    bad: list[str] = []
    for number, raw in enumerate(text.splitlines(), start=1):
        if not raw.strip():
            continue
        try:
            line = _Line.model_validate(json.loads(raw))
            pair = PreferencePair(
                trace_id_a=line.trace_id_a,
                trace_id_b=line.trace_id_b,
                dimension=line.dimension,
                winner=line.winner,
                judge_kind=line.judge_kind,
            )
            pairs.append((number, record_preference(pair, atom_set)))
        except json.JSONDecodeError as exc:
            bad.append(f"line {number}: not valid JSON ({exc.msg})")
        except SubjectiveError as exc:
            bad.append(f"line {number}: {exc}")
    if bad:
        more = f"; and {len(bad) - _SHOW} more" if len(bad) > _SHOW else ""
        raise SubjectiveError(
            f"{len(bad)} line(s) of the pairs file {file} were refused, so nothing was imported: "
            + "; ".join(bad[:_SHOW])
            + more
        )
    if not pairs:
        raise SubjectiveError(f"the pairs file {file} holds no preference")
    return pairs

record_preference

record_preference(pair, atom_set)

Validate a preference pair against the AtomSet and return it.

Raises SubjectiveError if: - winner is not 'a' or 'b' - trace_id_a or trace_id_b is empty - dimension is objective - dimension is unknown - trace_id_a == trace_id_b

Source code in src/calibrated_agent_evaluation/subjective/preference.py
def record_preference(pair: PreferencePair, atom_set: AtomSet) -> PreferencePair:
    """Validate a preference pair against the AtomSet and return it.

    Raises SubjectiveError if:
    - winner is not 'a' or 'b'
    - trace_id_a or trace_id_b is empty
    - dimension is objective
    - dimension is unknown
    - trace_id_a == trace_id_b
    """
    if pair.winner not in ("a", "b"):
        raise SubjectiveError(f"winner must be 'a' or 'b'; got {pair.winner!r}")

    if not pair.trace_id_a or not pair.trace_id_a.strip():
        raise SubjectiveError("trace_id_a must be a non-empty string")

    if not pair.trace_id_b or not pair.trace_id_b.strip():
        raise SubjectiveError("trace_id_b must be a non-empty string")

    if pair.trace_id_a == pair.trace_id_b:
        raise SubjectiveError(
            f"A trace cannot be preferred over itself (trace_id={pair.trace_id_a!r})"
        )

    if is_objective(pair.dimension, atom_set):
        raise SubjectiveError(
            f"Dimension {pair.dimension!r} is objective; "
            f"use binary PASS/FAIL labels, not preferences"
        )

    if not is_subjective(pair.dimension, atom_set):
        raise SubjectiveError(
            f"Dimension {pair.dimension!r} not found in AtomSet "
            f"for criterion {atom_set.criterion!r}; declare it first"
        )

    return pair

dump_preference_records

dump_preference_records(records, path, fmt='csv')

Write records to preference_record.{csv,json} under path.

Raises:

Type Description
SubjectiveStoreError

A bad fmt or any filesystem failure.

Source code in src/calibrated_agent_evaluation/subjective/store.py
def dump_preference_records(
    records: Sequence[PreferenceRecord], path: str | Path, fmt: str = "csv"
) -> None:
    """Write ``records`` to ``preference_record.{csv,json}`` under ``path``.

    Raises:
        SubjectiveStoreError: A bad ``fmt`` or any filesystem failure.
    """
    _STORE.dump(records, path, fmt)

load_preference_records

load_preference_records(path, fmt='csv')

Read preference records back from path; a missing file is an empty store.

Raises:

Type Description
SubjectiveStoreError

A schema-version mismatch, or any read, parse or validation failure.

Source code in src/calibrated_agent_evaluation/subjective/store.py
def load_preference_records(path: str | Path, fmt: str = "csv") -> list[PreferenceRecord]:
    """Read preference records back from ``path``; a missing file is an empty store.

    Raises:
        SubjectiveStoreError: A schema-version mismatch, or any read, parse or validation failure.
    """
    return _STORE.load(path, fmt)

preference_record

preference_record(
    pair,
    atom_set,
    *,
    preference_id,
    source,
    source_ref,
    rater_id=None,
)

Validate pair against atom_set and return it as a record to store.

Raises:

Type Description
SubjectiveError

The pair names an objective or undeclared dimension, compares a trace with itself, or has a winner other than "a" or "b".

SubjectiveStoreError

judge_kind is not human, llm-judge or code.

Source code in src/calibrated_agent_evaluation/subjective/store.py
def preference_record(
    pair: PreferencePair,
    atom_set: AtomSet,
    *,
    preference_id: str,
    source: PreferenceSource,
    source_ref: str,
    rater_id: str | None = None,
) -> PreferenceRecord:
    """Validate ``pair`` against ``atom_set`` and return it as a record to store.

    Raises:
        SubjectiveError: The pair names an objective or undeclared dimension, compares a trace
            with itself, or has a winner other than ``"a"`` or ``"b"``.
        SubjectiveStoreError: ``judge_kind`` is not ``human``, ``llm-judge`` or ``code``.
    """
    checked = record_preference(pair, atom_set)
    return PreferenceRecord(
        preference_id=preference_id,
        criterion=atom_set.criterion,
        dimension=checked.dimension,
        trace_id_a=checked.trace_id_a,
        trace_id_b=checked.trace_id_b,
        winner=checked.winner,
        judge_kind=checked.judge_kind,  # type: ignore[arg-type]  # validated by the model
        rater_id=rater_id,
        source=source,
        source_ref=source_ref,
        provenance=capture_provenance(),
    )