Skip to main content
warning

Deployment of ML models on EKS requires access to GPUs or Neuron instances. If your deployment isn't working, it’s often due to missing access to these resources. Also, some deployment patterns rely on Karpenter autoscaling and static node groups; if nodes aren't initializing, check the logs for Karpenter or Node groups to resolve the issue.

NVIDIA Dynamo on Amazon EKS

🚀 TL;DR – Serve LLMs with NVIDIA Dynamo on EKS

NVIDIA Dynamo is an open-source inference stack that sits above inference engines such as vLLM. It adds an OpenAI-compatible frontend, KV-aware routing, and disaggregated prefill/decode serving on top of the engine. This blueprint deploys the full Dynamo platform on Amazon EKS and ships ready-to-run model recipes.

What You Get​

  • ☸️ A complete platform – EKS cluster, Dynamo operator, Grove, KAI scheduler, NVIDIA GPU Operator, EFA device plugin, EFS CSI driver, and Prometheus, all delivered as ArgoCD applications
  • 📦 Model weight caching – Sync weights once from HuggingFace to Amazon S3 or to a shared EFS PVC, so replicas never download from HuggingFace at startup
  • ⚡ Fast scale-out with ModelExpress – New replicas copy weights and compile caches GPU-to-GPU over EFA from a running worker instead of reloading from S3
  • 🔀 Aggregated and disaggregated serving – Run prefill and decode in one worker, or split them across nodes with the KV cache moved over NIXL/EFA
  • 🧾 Model recipes – Tested DynamoGraphDeployment manifests per model and instance type

The workflow has three phases:

  1. Deploy infrastructure – provision the EKS cluster and platform add-ons with Terraform.
  2. Prepare the environment – configure kubectl, wait for the platform to converge, and create the ServiceAccounts, secrets, and storage the recipes depend on.
  3. Run a recipe – sync a model's weights and deploy it.

Key Implementation Details:

☸️
EKS Control Plane
v1.36
Default, overridable in tfvars
🌎
Default Region
us-west-2
Set via tfvars or AWS_REGION
🧩
Namespace
dynamo-system
Platform and recipes
🔁
Add-on Delivery
ArgoCD
GitOps-managed platform
🖥️
GPU Provisioning
Karpenter
Capacity Block aware
🔗
Interconnect
EFA + NIXL
KV cache and P2P weights
🪣
Weight Storage
S3 or EFS
S3 recommended
⏱️
First Deploy
20-30 min
Cluster and node groups
🖥️ Click to expand: GPU capacity requirements

We recommend using EC2 Capacity Blocks to guarantee GPU availability (for example p5.48xlarge or p6-b300.48xlarge) for your deployment window.

Tag your Capacity Block with ai-on-eks=true

When purchasing a Capacity Block, tag the reservation with ai-on-eks=true. Karpenter's GPU NodePool is configured to discover Capacity Blocks with this tag and launch instances into them, so your reserved capacity is used automatically when pods are scheduled.

🧠

What is NVIDIA Dynamo?​

NVIDIA Dynamo is an open-source, datacenter-scale inference serving framework for large language models. It doesn't replace the inference engine. It orchestrates engines such as vLLM, SGLang, and TensorRT-LLM, and adds the serving features that matter once a model spans multiple GPUs and nodes.

A Dynamo deployment is described by a single DynamoGraphDeployment custom resource. The Dynamo operator turns it into the pods, services, and gang-scheduled groups that make up the serving graph.

Core Building Blocks​

ComponentRole
FrontendOpenAI-compatible HTTP endpoint. Routes requests across workers with KV-aware routing, so requests land where their prefix is already cached.
WorkerThe inference engine (for example VllmWorker). In aggregated mode one worker does both prefill and decode.
PrefillWorker / DecodeWorkerIn disaggregated mode, prefill and decode run as separate workers, each sized for its phase.
NIXLNVIDIA's transfer library. Moves the KV cache from prefill to decode, and model weights between workers, over EFA using the LIBFABRIC backend.
ModelExpressOptional. Tracks which workers hold a loaded model so new workers can copy weights and compile caches peer-to-peer.

Aggregated vs Disaggregated Serving​

  • Aggregated – each worker runs the full request: prefill (processing the prompt) and decode (generating tokens). Simple to operate and a good default.
  • Disaggregated – prefill and decode run on separate workers. Prefill is compute-bound and decode is memory-bandwidth-bound, so splitting them lets you size and scale each phase independently. The KV cache produced by prefill moves to decode over NIXL/EFA, so both workers must land in the same zone.
🏗️

Architecture on Amazon EKS​

The blueprint provisions the cluster with Terraform and delivers every platform add-on as an ArgoCD application. Model weights live in Amazon S3 or on a shared EFS volume, and inference pods reach S3 through a ServiceAccount mapped to an IAM role.

🚀

Deploying the Solution​

Deployment steps are collapsed

The full step-by-step deployment guide is inside the expandable section below. Click it to open the prerequisites, the install steps, and the environment preparation.

▶️ Click to expand: step-by-step deployment instructions (provision an Amazon EKS cluster with the NVIDIA Dynamo platform and prepare it for model recipes)
1

Prerequisites

Install required tools

2

Deploy

Provision EKS and the Dynamo platform

3

Prepare

Configure access, secrets, and storage

Prerequisites​

Ensure that you have installed the following tools on your machine:

  • AWS CLI - AWS Command Line Interface
  • kubectl - Kubernetes command-line tool
  • terraform - Infrastructure as Code tool
  • helm - used to run the model sync jobs
  • jq and envsubst (from gettext) - used to read Terraform outputs and render manifests
  • A HuggingFace token - required for gated models and higher rate limits

Deploy​

1. Clone the repository​

Clone the repository
git clone https://github.com/awslabs/ai-on-eks.git
cd ai-on-eks/blueprints/inference/nvidia-dynamo
Working Directory

All commands below run from blueprints/inference/nvidia-dynamo. Paths to the Terraform stack (../../../infra/nvidia-dynamo) are resolved from there. Recipe commands are the exception: they run from each recipe's own folder.

Authentication Profile

If you are using a profile for authentication, set export AWS_PROFILE="<PROFILE_name>" to the desired profile name.

2. Review and customize the configuration​

Switch into the Terraform stack:

pushd ../../../infra/nvidia-dynamo

Edit terraform/blueprint.tfvars to match your environment. The most common changes are the AWS region and the EKS control plane version. The Dynamo-specific add-ons are already enabled:

blueprint.tfvars
name                             = "dynamo-on-eks"
enable_dynamo_platform = true
enable_kai_scheduler = true
enable_grove = true
enable_aws_efs_csi_driver = true
enable_aws_efa_k8s_device_plugin = true
enable_nvidia_gpu_operator = true
enable_s3_models_storage = true
# region = "us-west-2"
# eks_cluster_version = "1.36" # Uncomment to override default

You can control the deployment region in either of these ways:

  • Edit blueprint.tfvars and set the region variable.
  • Set the AWS_REGION=<AWS_REGION> environment variable.

Default AWS region is us-west-2.

1

Prerequisites

Install required tools

2

Deploy

Provision EKS and the Dynamo platform

3

Prepare

Configure access, secrets, and storage

3. Run the installer​

Deploy the NVIDIA Dynamo platform
./install.sh

The script copies the shared Terraform base into a local _LOCAL workspace, bootstraps the backend, and runs terraform apply. It creates:

  • The Amazon EKS cluster and the dynamo-system namespace
  • An S3 bucket for model weights
  • IAM roles for model sync (write) and inference (read)
  • EFS storage for shared caches
  • ArgoCD applications that install the platform and its dependencies

This step can take 20-30 minutes on a first run while the cluster and node groups come up. Once the apply completes, return to the blueprint directory:

popd
1

Prerequisites

Install required tools

2

Deploy

Provision EKS and the Dynamo platform

3

Prepare

Configure access, secrets, and storage

Prepare the Environment​

Step 1: Configure kubectl access

The Terraform stack exposes a ready-made aws eks update-kubeconfig command as the configure_kubectl output:

eval $(terraform -chdir="../../../infra/nvidia-dynamo/terraform/_LOCAL" output -raw configure_kubectl)

Step 2: Wait for the ArgoCD applications to be healthy

The platform add-ons (CSI drivers, GPU Operator, Dynamo operator, KAI scheduler, Grove, and so on) are ArgoCD applications. Wait for every application to report Healthy before continuing. Otherwise later steps can fail because a dependency, such as the EFS CSI driver or the Dynamo CRDs, isn't ready yet. This waits up to 10 minutes:

kubectl wait --for=jsonpath='{.status.health.status}'=Healthy \
applications.argoproj.io --all \
-n argocd \
--timeout=600s

Step 3: Set environment variables

Read the S3 bucket name and the two ServiceAccount names from the Terraform outputs. The recipes reference $S3_BUCKET, $MODEL_SYNC_SA, and $INFERENCE_SA directly, so export them in the shell you'll run the recipes from:

eval "$(
terraform -chdir="../../../infra/nvidia-dynamo/terraform/_LOCAL" output -json | jq -r '
"export S3_BUCKET=" + (.s3_models_buckets_name.value[0]),
"export MODEL_SYNC_SA=" + (.s3_models_sync_sa.value),
"export INFERENCE_SA=" + (.s3_models_inference_sa.value)
'
)"
VariablePurpose
S3_BUCKETThe bucket where model weights are stored.
MODEL_SYNC_SAServiceAccount used by the sync Job to write weights to S3.
INFERENCE_SAServiceAccount used by inference pods to read weights from S3.

Step 4: Create the ServiceAccounts

Terraform maps each name to an IAM role (via IRSA or EKS Pod Identity): the sync SA gets write access to the bucket and the inference SA gets read access. Keeping them separate holds a least-privilege boundary between syncing and serving.

kubectl -n dynamo-system create sa $MODEL_SYNC_SA
kubectl -n dynamo-system create sa $INFERENCE_SA

Step 5: Create the HuggingFace token secret

The download and sync Jobs read this secret to authenticate with HuggingFace:

export HF_TOKEN=<HF TOKEN>
kubectl -n dynamo-system create secret generic hf-token-secret \
--from-literal=HF_TOKEN=${HF_TOKEN}

Step 6: Create the EFS PersistentVolumeClaims

Create the shared EFS-backed caches. All three use the efs-sc-dynamic storage class with ReadWriteMany access so they can be shared across pods:

PVCPurpose
model-cacheModel weights for the PVC-based caching workflow.
compilation-cacheCompiled kernels and engines, so replicas don't recompile on every start.
perf-cachePerformance-tuning and benchmark artifacts.
View pvc-efs.yaml
pvc-efs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: model-cache
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: "efs-sc-dynamic"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: compilation-cache
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: "efs-sc-dynamic"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: perf-cache
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: "efs-sc-dynamic"
kubectl -n dynamo-system apply -f pvc-efs.yaml

Step 7: Verify

kubectl get applications.argoproj.io -n argocd
kubectl -n dynamo-system get sa,secret,pvc

Expected output: every ArgoCD application is Synced and Healthy, both ServiceAccounts and hf-token-secret exist, and the three PVCs are Bound.

⚡

NVIDIA ModelExpress (Optional)​

ModelExpress speeds up model distribution. It tracks model metadata and cache entries in-cluster and lets pods transfer weights peer-to-peer. Recipes with mx in the manifest name use it. Install its CRDs and RBAC once per cluster.

Creates the modelmetadatas.modelexpress.nvidia.com and modelcacheentries.modelexpress.nvidia.com CRDs:

kubectl apply -f https://raw.githubusercontent.com/ai-dynamo/modelexpress/refs/heads/main/examples/crds.yaml

How P2P Weight Transfer Works​

The first worker does a cold start: it streams weights from S3 into GPU memory, runs the full warm-up (torch compile, Triton kernels, FlashInfer autotuning), and only registers as a source once its health endpoint reports ready. When you scale out, new workers find that source through the ModelExpress server and copy weights GPU-to-GPU over EFA, plus the compile caches, so they skip most of the warm-up.

note

A target only pairs with a source that serves the same model with the same parallel layout, and each GPU rank copies from the matching rank. Workers use a required podAffinity on zone so the RDMA transfer stays on the EFA fabric.

📦

Model Weight Caching​

Recipes never download weights from HuggingFace at serving time. You sync them once to one of two targets, and the recipe manifest you choose decides which one it reads from.

Syncs the model repository to your S3 bucket with the AI on EKS inference-charts Helm chart. The chart's s3ModelCopy component runs a one-shot Job on an NVMe-equipped, high-bandwidth node and downloads files in parallel. Weights land under s3://$S3_BUCKET/$MODEL_NAME/.

Best for: durable storage shared across clusters, regions, or accounts, and the most cost-effective option.

helm repo add ai-on-eks https://awslabs.github.io/ai-on-eks-charts/
helm repo update
Sync a model to S3
export MODEL_NAME=<MODEL NAME>   # e.g. Qwen/Qwen3-32B
helm -n dynamo-system install hf-s3-sync-model ai-on-eks/inference-charts -f - <<EOF
s3ModelCopy:
namespace: dynamo-system
model: $MODEL_NAME
s3Bucket: $S3_BUCKET
hfTokenSecret:
name: "hf-token-secret"
envFrom: true
serviceAccountName: $MODEL_SYNC_SA
requireLocalNvme: true
storageSize: 100
requireNetworkBandwidth: 25000
parameters:
fileWorkers: 18
EOF
ValueMeaning
serviceAccountNameThe sync ServiceAccount, mapped to an IAM role with write access to the bucket.
requireLocalNvme / requireNetworkBandwidthPlace the Job on an NVMe-backed node with at least 25 Gbps of network bandwidth.
parameters.fileWorkersFiles downloaded in parallel.
storageSizeScratch volume in GiB. Size it as roughly the largest file in the repo times fileWorkers.

Each recipe lists tuned values for its model.

Monitor and clean up:

kubectl -n dynamo-system wait --for=condition=ready pod \
-l app.kubernetes.io/name=s3-model-copy,app.kubernetes.io/instance=hf-s3-sync-model --timeout=600s
kubectl -n dynamo-system logs -f \
-l app.kubernetes.io/name=s3-model-copy,app.kubernetes.io/instance=hf-s3-sync-model
kubectl -n dynamo-system get job -l app.kubernetes.io/instance=hf-s3-sync-model
aws s3 ls "s3://$S3_BUCKET/$MODEL_NAME/" --recursive --human-readable --summarize

helm -n dynamo-system uninstall hf-s3-sync-model
Pod stuck in Pending?

Karpenter may not find a node that satisfies the NVMe and bandwidth constraints, because of regional instance availability, EC2 vCPU quota, or NodePool requirements that exclude matching instances. Check the pod's FailedScheduling events and the Karpenter controller logs. If no matching instances are available to you, lower requireNetworkBandwidth first, then storageSize and fileWorkers.

🧾

Deploy a Model​

Each model lives under recipes/. A recipe README documents the model-specific sync values, the manifests it ships, and how to test it. Recipe commands use paths relative to the recipe folder, so switch into it first:

pushd recipes/<recipe>
# ...run the recipe commands...
popd

Available Recipes​

RecipeModelInstanceEngine
qwen3-32bQwen/Qwen3-32Bp5.48xlarge (H100)vLLM
glm-5.3zai-org/GLM-5.3p6-b300.48xlarge (B300)vLLM
glm-5.3-flashzai-org/GLM-5.3-Flashg7e.48xlargevLLM
nemotron-3-ultranvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16p6-b300.48xlarge (B300)vLLM
nemotron-3.5-bolt-e4bnvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16g6evLLM

Manifest names encode the instance, topology, and weight source. For example h100-agg-mx-s3.yaml is aggregated serving on H100 that loads weights from S3 through ModelExpress, and h100-disagg.yaml is disaggregated prefill/decode from the model-cache PVC.

Walkthrough: Qwen3-32B​

This example deploys Qwen3-32B with TP=4 on a p5.48xlarge, loading weights from S3 through ModelExpress. It assumes you synced the weights to S3 with MODEL_NAME="Qwen/Qwen3-32B" and installed ModelExpress.

Deploy Qwen3-32B (aggregated, S3 + ModelExpress)
cd recipes/qwen3-32b
envsubst '${S3_BUCKET} ${INFERENCE_SA}' < vllm/h100-agg-mx-s3.yaml | kubectl apply -n dynamo-system -f -

kubectl -n dynamo-system logs \
-l nvidia.com/dynamo-graph-deployment-name=qwen3-32b-h100-agg-mx-s3-vllm \
--all-containers=true --max-log-requests=20 --prefix=true --timestamps -f

The deployment starts three components: Frontend, ModelExpress (with Kubernetes as its metadata backend, so no Redis is needed), and VllmWorker.

For the PVC-based aggregated and disaggregated variants, see the Qwen3-32B recipe README.


Cleanup​

Delete each recipe you deployed using the cleanup commands in its README. For example:

pushd recipes/qwen3-32b
envsubst '${S3_BUCKET} ${INFERENCE_SA}' < vllm/h100-agg-mx-s3.yaml | kubectl delete -n dynamo-system -f -
popd