Gen AI Evaluation Toolkit on AWS

v2.0.0

Copyright © 2026 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon’s trademarks and trade dress may not be used in connection with any product or service that is not Amazon’s, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon.

Interface: TestCase¶

A complete test case including input, expected output, and system metadata

Extended by¶

Properties¶

annotations?¶

optional annotations?: Annotation[]

Annotations associated with the dataset

context?¶

optional context?: DocumentType

Additional context for the test case. Can contain metadata such as flags indicating human review needed, artifacts providing evidence for agent-as-judge evaluation, or any other structured data needed for evaluation.

createdAt?¶

optional createdAt?: Date

Timestamp indicating when the test-case was created.

createdBy?¶

optional createdBy?: string

The creator of the test case—either a username (for human creators) or an application identifier (for synthetic data generators)

expected?¶

optional expected?: DocumentType

Expected output of the Gen AI Application for the given input. Can be any structured data including simple text responses, expected tool calls, expected topics, or any combination of expected behaviors.

input?¶

optional input?: DocumentType

The input to the Gen AI Application. Can be any structured data including prompts, context, and parameters.

metadata?¶

optional metadata?: DocumentType

Optional free-form metadata set by the caller. Carried through verbatim to results and ignored by the pipeline — use it to tag test cases (source, split, tenant, run label, etc.) for grouping and debugging.

testCaseId¶

testCaseId: string

Test case ID, also used as file name in S3

Table of contents