DEPRECATION NOTICE
This blueprint will be deprecated and eventually removed from this GitHub repository on October 27, 2024. No bugs will be fixed, and no new features will be added. The decision to deprecate is based on the lack of demand and interest in this blueprint, as well as the difficulty in allocating resources to maintain a blueprint that is not actively used by any users or customers.
If you are using this blueprint in production, please add yourself to the adopters.md page and raise an issue in the repository. This will help us reconsider and possibly retain and continue to maintain the blueprint. Otherwise, you can make a local copy or use existing tags to access it.
ACK Controller for EMR on EKS
Introduction
In this post, we will learn to build EMR on EKS Spark workloads by using AWS Controllers for Kubernetes (ACK). We will also build a end-to-end observability for Spark workloads by leveraging Amazon Managed Service for Prometheus to collect and store the metrics generated by Spark Applications and then use Amazon Managed Grafana to build dashboards for monitoring use cases.
Deploying the Solution
👈Setup Amazon Managed Grafana with SSO
Currently, this step is manual. Please follow the steps in this blog to create Amazon Managed Grafana with SSO enabled in your account. You can visualize the Spark jobs runs and metrics using Amazon Managed Prometheus and Amazon Managed Grafana.
Execute Sample Spark job - EMR Virtual Cluster
👈Cleanup
👈To avoid unwanted charges to your AWS account, delete all the AWS resources created during this deployment