Skip to main content

Open source · Data + AI agents

Agentic data platforms on Amazon EKS.

Production-ready blueprints for Spark, Flink, Kafka and Trino, plus the agents, MCP servers and semantic layers that let AI work with your data. Build, deploy, and scale with confidence.

Ready-to-use Data Stacks
30+
GitHub Stars
800+
Official Project
AWS

Reference architecture

One platform, end to end.

From streaming ingest to GPU inference, every layer is an open-source blueprint on Amazon EKS, built on an open lakehouse, with AI agents working across it.

Agentic layer
Data plane
  1. 01

    Ingest

    Events, CDC and logs

    Real-time topics also feed Pinot, ClickHouse and StarRocks directly.

  2. 02

    Stream

    Parallel choices

    ↓ Writes to the lakehouse

  3. 03

    Batch

    Spark at scale

    ↕ Reads and writes the lakehouse

  4. 04

    Query & serve

    SQL, OLAP, search, BI and cache

    ↑ Reads from the lakehouse

  5. 05

    AI inference

    Batch, on GPUs

    Reads Iceberg tables, writes predictions back and embeddings to S3 Vectors.

    ↕ Reads tables, writes results and embeddings

The platform

What is Data on EKS?

Data on EKS is an open-source project for building agentic data platforms on Amazon EKS. It packages proven open-source data tools and AWS infrastructure as Terraform and ArgoCD blueprints, backed by performance benchmarks and best practices, so teams can run complex workloads across thousands of nodes with confidence. The agentic layer adds AI agents that investigate incidents and diagnose slow jobs, plus MCP servers and semantic layers that give AI structured access to your data.

Agentic DataNew

Production-ready reference architectures for AI agents built for data platforms. Deploy working agents that investigate metric anomalies, diagnose slow or failing Spark jobs, and trace pipeline breaks, built with open source frameworks like LangGraph and AWS services like Amazon Bedrock AgentCore.

Data Analytics

Transform your data with enterprise-grade analytics solutions. Deploy Apache Spark, Ray, and Jupyter environments with production-ready configurations. Scale from terabytes to petabytes with confidence using battle-tested architectures.

Streaming Data Platforms

Build real-time data pipelines that never sleep. Process millions of events per second with Apache Kafka, Flink, and Kinesis. From IoT sensors to financial transactions, handle any streaming workload at any scale.

Amazon EMR on EKS

Run enterprise-grade Spark workloads on Kubernetes with Amazon EMR on EKS. Get optimized Spark runtime, automatic scaling, simplified job management, and seamless integration with AWS services for faster, more cost-effective big data processing.

Workflow Orchestration

Orchestrate complex data workflows with precision. Deploy Apache Airflow and Argo Workflows to automate ETL pipelines, ML training, and data quality checks. Never miss a dependency again.

Watch

Featured videos

Kubeflow Graduation: Powering the Next Era of Cloud Native AI With an Open Community

Kubeflow Graduation: Powering the Next Era of Cloud Native AI With an Open Community

KubeCon + CloudNativeCon North America 2026 keynote on Kubeflow's CNCF graduation. Starts at 4:33.

AWS re:Invent 2023 - Data processing at massive scale on Amazon EKS

AWS re:Invent 2023 - Data processing at massive scale on Amazon EKS

Dec 4, 2023

Data processing at massive scale with Spark on Amazon EKS by Pinterest.

Ship your first data stack today.

Terraform and ArgoCD blueprints for Spark, Flink, Kafka, Trino, Airflow and more, plus AI agents for your data.