📄️ Introduction
Job schedulers are an essential component of many organizations' infrastructure, helping to automate and manage complex workflows. When deployed on Kubernetes, job schedulers can take advantage of the platform's features such as automatic scaling, rolling updates, and self-healing capabilities to ensure high availability and reliability. Tools like Apache Airflow, Argo Workflow, and Amazon MWAA provide a simple and efficient way to manage and schedule jobs on a Kubernetes cluster.
📄️ Amazon MWAA
Amazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale. Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows.” With Managed Workflows, you can use Airflow and Python to create workflows without having to manage the underlying infrastructure for scalability, availability, and security.
📄️ Airflow on EKS
Introduction
📄️ Argo Workflows on EKS
Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. It is implemented as a Kubernetes CRD (Custom Resource Definition). As a result, Argo workflows can be managed using kubectl and natively integrates with other Kubernetes services such as volumes, secrets, and RBAC.
📄️ AWS Batch on EKS
AWS Batch is a fully-managed AWS-native batch computing service that plans, schedules, and runs your containerized batch workloads (machine-learning, simulation, and analytics) on top of AWS managed container orchestrations services like Amazon Elastic Kubernetes Service (EKS).