Configuration

agenteval.yml

evaluator:
  model: claude-3
target:
  type: bedrock-agent
  bedrock_agent_id: string
  bedrock_agent_alias_id: string
tests:
  retrieve_missing_documents:
    steps:
    - Ask agent for a list of missing documents for claim-006.
    expected_results:
    - The agent returns a list of missing documents.
    initial_prompt: Give me a list of missing documents for claim-006.
    max_turns: 2
    hook: path.to.MyHook

evaluator (map)

Refer to Evaluators for the available configurations.

---

target (map)

Refer to Targets for the available configurations.

---

tests (map)

A map of test cases, where the test name serves as the key.

---

steps (list of strings)

The steps to perform for the test.

---

expected_results (list of strings)

The expected results for the test.

---

initial_prompt (string; optional)

The first message that is sent to the agent, which starts the conversation. If unspecified, the message will be generated based on the steps provided.

---

max_turns (integer; optional)

The maximum number of user-agent exchanges before the test fails. The default is 2.

---

hook (string; optional)

The module path to an evaluation hook. Refer to Hooks for more details.

---