Gen AI Evaluation Toolkit on AWS
v2.0.0
Copyright © 2026 Amazon Web Services, Inc. and/or its affiliates. All rights reserved. Amazon’s trademarks and trade dress may not be used in connection with any product or service that is not Amazon’s, in any manner that is likely to cause confusion among customers, or in any manner that disparages or discredits Amazon. All other trademarks not owned by Amazon are the property of their respective owners, who may or may not be affiliated with, connected to, or sponsored by Amazon.
Table of contents
Gen AI Evaluation Toolkit on AWS¶
The Gen AI Evaluation Toolkit on AWS provides a framework for comprehensive evaluation of generative AI applications through test case generation, metrics-based assessment, and test data management.
Key capabilities include: - LLM-as-Judge: LLM-based scoring with custom criteria and prompts - Agent-as-Judge: Agentic evaluation with active artifact exploration and evidence collection - RAGAS: Purpose-built RAG evaluation metrics - AgentCore: OpenTelemetry-integrated evaluation for agentic systems - DeepEval: Comprehensive LLM evaluation metrics for single-turn and multi-turn conversations - PyRIT: Automated adversarial red-teaming for LLM security testing - fmeval: AWS Foundation Model Evaluation metrics
Core guides¶
- Solution Guide — Architecture, design, and deployment details (CDK and Terraform options)
- User Guide — Getting started, configuration, and usage instructions
SDK references¶
- Evaluator SDK — Build custom evaluators (scorers, app_invoke)
- Evaluator PyRIT — Adversarial red-teaming with PyRIT
- Python client reference — Python client API reference
- TypeScript client reference — TypeScript client API reference