Skip to main content

Job Schedulers on EKS

📄️ Introduction

Job schedulers are an essential component of many organizations' infrastructure, helping to automate and manage complex workflows. When deployed on Kubernetes, job schedulers can take advantage of the platform's features such as automatic scaling, rolling updates, and self-healing capabilities to ensure high availability and reliability. Tools like Apache Airflow, Argo Workflow, and Amazon MWAA provide a simple and efficient way to manage and schedule jobs on a Kubernetes cluster.

📄️ Amazon MWAA

Amazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale. Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows.” With Managed Workflows, you can use Airflow and Python to create workflows without having to manage the underlying infrastructure for scalability, availability, and security.