Skip to content

Inspect AI

You need Inspect AI .eval logs, and the inspect extra.

Ingest the logs

uv sync --inexact --extra inspect
source .venv/bin/activate    # puts `calibrated-agent-evaluation` on your PATH
calibrated-agent-evaluation ingest logs/

On a one-sample log of the repository's OrderBot:

$ calibrated-agent-evaluation ingest fixtures/raw/inspect/ORD-1001.eval
calibrated-agent-evaluation ingest: 1 ingested, 0 refused, 0 skipped
  [ok] fixtures/raw/inspect/ORD-1001.eval → ORD-1001 (inspect, 1 span(s), 3 tool call(s), owned)

A log holds one sample per dataset row, and ingest stores one trace per sample. From Python, use normalize_all(InspectEvalLog(path=...)) for every sample; normalize() on a multi-sample log raises an error unless sample_id= names one, so you always know which samples were read.

Or let Calibrated Agent Evaluation run the task

InspectExecutor runs an Inspect task with your agent as the solver and returns its traces. Replayed from the repository's cassette:

trace ORD-1001: ownership=owned, 1 span(s)
input:  {'message': "I want a refund for order A-7781, the blender just isn't for me."}
  orderbot called lookup_order({'order_id': 'A-7781'})
  orderbot called check_refund_policy({'order_id': 'A-7781', 'today': '2025-01-30'})
  orderbot called issue_refund({'amount_usd': 89.99, 'order_id': 'A-7781'})
output: {'answer': 'I processed a full refund of $89.99 for order A-7781 (Countertop Blender). The refund was within the return window and was successfully issued with Refund ID RF-30022.'}

See Route your agent through Calibrated Agent Evaluation for the code.

What you get

An owned trace per sample: the agent's step with every tool call, its arguments and results. The sample's metadata can carry the trace id and taxonomy cell.

Limits

  • An agent run through Inspect's agent bridge stores long strings as attachment:// references in the log. Check that the content you need is in the trace before relying on it.
  • The trace above is the OrderBot failure the bundled judge kit is built on: the order was outside the 30-day return window, the broken policy tool said it was within it, and the agent refunded $89.99 on its word. Ingest records what happened; judging it comes later.