AWS Features for Kubeflow

Running Kubeflow on AWS gives you the following feature benefits and configuration options:

Note: Beginning with v1.3, development for Kubeflow on AWS can be found in the AWS Labs repository. Previous versions can be found in the Kubeflow manifests repository.

Manage AWS compute environments

• Provision and manage your Amazon Elastic Kubernetes Service (EKS) clusters with eksctl and easily configure multiple compute and GPU node configurations.
• Use AWS-optimized container images, based on AWS Deep Learning Containers, with Kubeflow Notebooks.

CloudWatch Logs and Metrics

• Integrate Kubeflow on AWS with Amazon CloudWatch for persistent logging and metrics on EKS clusters and Kubeflow pods.
• Use AWS ContainerInsights to collect, aggregate, and summarize metrics and logs from your containerized applications and microservices.

Load balancing, certificates, and identity management

• Manage external traffic with AWS Application Load Balancer.
• Get started with TLS authentication using AWS Certificate Manager and AWS Cognito.

AWS database and storage solutions

• Integrate Kubeflow with Amazon Relational Database Service (RDS) for a highly scalable pipelines and metadata store.
• Deploy Kubeflow with integrations for Amazon S3 for an easy-to-use pipeline artifacts store.
• Use Kubeflow with Amazon Elastic File System (EFS) for a simple, scalabale, and serverless storage solution.
• Leverage the Amazon FSx CSI driver to manage Lustre file systems which are optimized for compute-intensive workloads, such as high-performance computing and machine learning. Amazon FSx for Lustre can scale to hundreds of GBps of throughput and millions of IOPS.

Integrate with Amazon SageMaker

• Use Kubeflow on AWS with Amazon SageMaker to create hybrid machine learning workflows.
• Train, tune, and deploy machine learning models in Amazon SageMaker without logging into the SageMaker console using SageMaker Operators for Kubernetes (ACK).
• Create a Kubeflow Pipeline built entirely using SageMaker Components for Kubeflow Pipelines, or integrate individual components into your workflow as needed.