Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nuke EKS clusters #1407

Merged
merged 5 commits into from
Dec 29, 2023
Merged

Nuke EKS clusters #1407

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
578193b
adding task to delete regression cluster
vatsrahul1001 Dec 29, 2023
8be5742
updating nuke dag to delete eks cluster from dag authoring account
vatsrahul1001 Dec 29, 2023
04e3113
updating task name
vatsrahul1001 Dec 29, 2023
a5bebb2
renaming eks deleteion file name
vatsrahul1001 Dec 29, 2023
ec75c80
update file name in aws nuke dag
vatsrahul1001 Dec 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions astronomer/providers/amazon/aws/example_dags/delete_eks_cluster_and_nodes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#!/bin/bash

# Check if the region parameter is provided
if [ -z "$1" ]; then
echo "Usage: $0 <AWS_REGION>"
exit 1
fi

AWS_REGION="$1"

# List all EKS clusters
clusters=$(aws eks list-clusters --region $AWS_REGION | jq -r '.clusters[]')

# Loop through each EKS cluster
for cluster in $clusters; do
echo "Processing EKS cluster: $cluster"

# List nodegroups for the cluster
nodegroups=$(aws eks list-nodegroups --cluster-name $cluster --region $AWS_REGION | jq -r '.nodegroups[]')

# Delete each nodegroup
for nodegroup in $nodegroups; do
echo "Deleting nodegroup '$nodegroup' for cluster '$cluster'"
aws eks delete-nodegroup --cluster-name $cluster --nodegroup-name $nodegroup --region $AWS_REGION
aws eks wait nodegroup-deleted --cluster-name $cluster --nodegroup-name $nodegroup --region $AWS_REGION
done


# Delete the EKS cluster
echo "Deleting EKS cluster: $cluster"
aws eks delete-cluster --name $cluster --region $AWS_REGION

# Wait for the EKS cluster to be deleted
echo "Waiting for EKS cluster '$cluster' to be deleted..."
aws eks wait cluster-deleted --name $cluster --region $AWS_REGION

done

echo "Script execution completed."
16 changes: 16 additions & 0 deletions astronomer/providers/amazon/aws/example_dags/example_aws_nuke.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@
SLACK_USERNAME = os.getenv("SLACK_USERNAME", "airflow_app")
SLACK_WEBHOOK_CONN = os.getenv("SLACK_WEBHOOK_CONN", "http_slack")

REGRESSION_CLUSTER_AWS_ACCESS_KEY = os.getenv("REGRESSION_CLUSTER_AWS_ACCESS_KEY", "**********")
REGRESSION_CLUSTER_AWS_SECRET_ACCESS_KEY = os.getenv(
"REGRESSION_CLUSTER_AWS_SECRET_ACCESS_KEY", "***********"
)
REGRESSION_CLUSTER_AWS_DEFAULT_REGION = os.getenv("REGRESSION_CLUSTER_AWS_DEFAULT_REGION", "us-east-1")


def generate_task_report(**context: Any) -> None:
"""Generate a report of the task statuses for the DAG run and send it to configured Slack channel for alerts."""
Expand Down Expand Up @@ -125,6 +131,15 @@ def check_dag_status(**kwargs: Any) -> None:
f"aws emr-containers list-virtual-clusters --state RUNNING --region {AWS_DEFAULT_REGION} | jq -r '.virtualClusters[].id' | xargs -I % aws emr-containers delete-virtual-cluster --id % --region {AWS_DEFAULT_REGION}; ",
)

terminate_regression_clusters = BashOperator(
task_id="terminate_regression_clusters",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's mention here that these are from Astro SDK
e.g. terminate_astro_sdk_regression_clusters or terminate_dag_authoring_regression_clusters

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

terminate_dag_authoring_regression_clusters would be better

bash_command=f"set -e; "
f"aws configure set aws_access_key_id {REGRESSION_CLUSTER_AWS_ACCESS_KEY}; "
f"aws configure set aws_secret_access_key {REGRESSION_CLUSTER_AWS_SECRET_ACCESS_KEY}; "
f"aws configure set default.region {REGRESSION_CLUSTER_AWS_DEFAULT_REGION}; "
f"sh $AIRFLOW_HOME/dags/delete_eks_cluster_and_nodes.sh {REGRESSION_CLUSTER_AWS_DEFAULT_REGION}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to pass REGRESSION_CLUSTER_AWS_DEFAULT_REGION as a parameter can't we access it in the bash file directly with something like ${REGRESSION_CLUSTER_AWS_DEFAULT_REGION}?

)

execute_aws_nuke = BashOperator(
task_id="execute_aws_nuke",
bash_command=f"aws configure set aws_access_key_id {AWS_ACCESS_KEY_ID}; "
Expand Down Expand Up @@ -163,6 +178,7 @@ def check_dag_status(**kwargs: Any) -> None:
start
>> [get_airflow_version, get_airflow_executor]
>> terminate_running_emr_virtual_clusters
>> terminate_regression_clusters
>> execute_aws_nuke
>> delete_stale_emr_vpcs
>> delete_stale_emr_iam_roles
Expand Down