UAEF

Universal Agent Evaluation Framework — a framework-agnostic evaluation system for AI agents. Plug it into any agentic platform to measure agent quality across 36 built-in metrics, persist results to AWS, and track performance over time.

UAEF works with LangGraph, LangChain, AWS Bedrock, Langfuse, Strands, AgentCore, or any custom framework via adapters. One evaluate() call gives you scores across tool calling, response quality, safety, performance, reasoning, and more.

from uaef.api import evaluate

result = evaluate(trace=trace, ground_truth=ground_truth)

print(f"Score: {result.overall_score:.2f}")
print(f"Passed: {result.passed}")

Install UAEF Run your first evaluation

Where to go

I want to…	Go to
Install and run a first evaluation	Installation · Quickstart
Understand traces, dimensions, and modes	Core Concepts
See every metric	Metrics Catalog
Connect my agent framework	Adapters
Batch, CI/CD gating, production monitoring	Use Cases
Track experiments and catch regressions	Experiments
Write my own metric	Custom Metrics
Look up a class or function	API Reference
Deploy the service to AWS	uaef-service/README.md

Architecture

flowchart TD
    U[User / Platform] -->|agent traces + ground truth| A

    subgraph UAEF
        A[Framework Adapters<br/>LangGraph · Bedrock · LangChain<br/>Langfuse · Strands · AgentCore · GenericJSON]
        A -->|canonical AgentTrace| E[Evaluation Engine<br/>SingleAgent · MultiTurn · MultiAgent · Offline]
        E --> M[Metrics — 36 built-in<br/>Tool Calling · Response Quality · Responsible AI<br/>Performance · Multi-Turn · Multi-Agent · Reasoning<br/>+ custom · + RAGAS/DeepEval]
        M --> R[EvaluationResult]
        R -->|persist=True| P[AWS Persistence<br/>DynamoDB: experiment, avg_scores, result_path<br/>S3: full result JSON]
        R --> AN[Analysis &amp; Reporting<br/>Root cause · Clustering · Trends<br/>Dashboards · Regression detection]
        P --> AN
    end

    R -->|returned| U

See Core Concepts for the canonical trace format, the seven evaluation dimensions, and the evaluation modes.

Metrics at a glance

36 built-in metrics across 7 dimensions, plus 2 opt-in use-case-specific metrics and 15 more via the RAGAS and DeepEval integrations.

Dimension	Metrics
Tool Calling	tool_selection_accuracy, tool_sequence_correctness, parameter_quality, mcp_compliance
Response Quality	answer_relevance, completeness, hallucination_score, accuracy
Responsible AI	safety_score, bias_score, prompt_injection_detection, toxicity_score
Performance	latency_score, token_efficiency, cost_efficiency, throughput
Multi-Turn	context_retention, coherence, conversation_completeness, turn_efficiency, role_adherence, holistic_llm_judge, user_satisfaction, sentiment, agent_tone, naturalness, instruction_compliance, optimum_turns
Multi-Agent	agent_utilization, delegation_quality, workflow_completion, coordination_efficiency
Reasoning	chain_of_thought_coherence, logical_consistency, reasoning_step_correctness, fallacy_detection

Run a subset by name:

result = evaluate(trace=trace, metrics=["answer_relevance", "safety_score", "latency_score"])

Full tables, including the integration metrics and Stickler: Metrics Catalog.

Tenant isolation

UAEF is a reference architecture designed to run embedded inside a platform that provides user/tenant isolation. Direct deployment in a multi-tenant environment without a host platform enforcing tenant isolation is not supported. Read SECURITY.md for the full model.

License

Apache License 2.0.