This project provides a ready-to-use configuration for setting up an AWS EKS cluster with all necessary controllers, operators, and monitoring stack. By using this configuration, DevOps engineers can save 1-2 months of work.
- Adaptation for small clusters: Many modules, especially Grafana monitoring stack, are created with the intention of being used in large clusters. I understand the pain of small projects, so I tried to create a setup that is as simple and efficient as possible. For example, VictoriaMetrics stack selected as best prometheus-like monitoring engine and Uptrace with AWS S3 backend selected for long term metrics, logs and traces storage.
- Node Group Templates: Templates for creating Managed Node Groups and Fargate Profile linked to each availability zone individually.
- Default Settings and Integration: Reasonable default values and integration between modules for seamless setup.
Similar projects:
- Filipp Frizzy: 181h 30m
I'm Filipp - a Lead DevOps Engineer with 12+ years of experience, currently based in Poland (UTC+2). I am open to work and considering Senior, Lead, or Architect DevOps roles with a B2B contract from $7k/month and 100% remote. I have extensive experience as a primary or lead DevOps engineer in product teams and startups. If you are looking for a DevOps engineer for a project, contact me on LinkedIn.
From my side:
- Working as Ops and DevOps engineer since 2012, with over 7 years of experience with UK & US teams.
- Experience as Single, Main, or Lead DevOps for small teams of other Ops people.
- Migration of services into Docker environments, including Kubernetes, Docker Swarm, and AWS Elastic Containers.
- AWS is my primary cloud since 2015
- Proficient with GitLab, GitHub, Jenkins, ArgoCD, and FluxCD CI & CD.
- Writing Terraform, Terragrunt, Ansible, SaltStack, and other IaC setups.
- Solved several production disasters with various Kubernetes setups.
- Skilled in SQL and NoSQL HA setups, like Galera MySQL, MongoDB, Kafka, ZooKeeper, Clickhouse, Redis, etc.
- Developed many monitoring solutions with Prometheus, VictoriaMetrics, EFK, Zabbix, etc.
- Authored 2 open source projects with over 1k stars.
| Description | Purpose | Enabled | DNS |
|---|---|---|---|
| EKS cluster module based on terraform-aws-modules/eks/aws v19 | Base | True | |
| Templates for Managed Node Groups and Fargate Profile to link them to each availability zone instead of all zones at once | Base | True | |
| Integration of modules with each other and reasonable default values | Base | True | |
| CoreDNS EKS addon | Core | True | |
| Kube-Proxy EKS addon | Core | True | |
| VPC CNI EKS addon | Core | True | |
| AWS EBS CSI driver EKS addon | Core | True | |
| Snapshot Controller EKS addon | Core | True | |
| AWS EFS CSI driver | Core | True | |
| AWS Node Termination Handler | Core | True | |
| Cert Manager | Core | True | |
| Cluster Autoscaler | Core | True | |
| Metrics Server | Core | True | |
| Vertical Pod Autoscaler | Core | True | |
| Ingress Apisix | Ingress | True | |
| Ingress Nginx | Ingress | False | |
| Victoriametrics Operator | Operator | True | |
| Opentelemetry Operator | Operator | False | |
| Clickhouse Operator | Operator | False | |
| Grafana Operator | Operator | True | |
| Victoriametrics | Monitoring | True | vmauth.${var.ingress_domain} victoriametrics.${var.ingress_domain} vmalertmanager.${var.ingress_domain} vmagent.${var.ingress_domain} vmalert.${var.ingress_domain} |
| Grafana | Monitoring | True | grafana.${var.ingress_domain} |
| Uptrace | Monitoring | True | uptrace.${var.ingress_domain} |
| Vector | Monitoring | True | |
| Qryn | Monitoring | False | qryn.${var.ingress_domain} |
| Openobserve | Monitoring | False | openobserve.${var.ingress_domain} |
| Kubernetes Dashboard | Control | False | k8s-dashboard.${var.ingress_domain} |
- Email integration
- DNS integration
- Alert rules
- Resource limits
- CI & CD integration
- Network policies
- Host-based pod segregation
- terraform
- aws cli
- kubectl
- terraform-aws-eks
- aws-ia/eks-blueprints-addons/aws
This module contains a local-exec block with kubectl patch for applying tolerations and nodeSelector deployments in the kube-system namespace, which will only work in a Unix shell, and will fail on Windows. This patch is necessary as some EKS addons currently don't support tolerations and nodeSelector in their configurations, but it is only necessary if you use host nodes with taints to separate management processes from others. You can disable it by setting the apply_kubectl_patch variable to false.
cd example
terraform init
terraform apply -target=module.vpc
terraform apply
terraform output all
To destroy everything, run (you may need to run it twice):
terraform destroy -auto-approve
Force destroy in case of problems:
helm ls -a --all-namespaces | awk 'NR > 1 { print "-n "$2, $1}' | xargs -L1 helm delete
kubectl delete all --all --all-namespaces
terraform destroy -auto-approve
After terraform destroy, check EC2 volumes for unused disks as the aws-ebs-csi-driver doesn't delete them by default after deleting helm releases.
victoria-metrics-k8s-stack is deployed without internal password protection. Multiple charts such as apisix, qryn, and uptrace contain explicit passwords in the values and do not use Kubernetes secrets.
Helm upgrade reset_values flag is set to true for everything except databases like PostgreSQL and Clickhouse. See this explanation.
Check the ./example/outputs.example file to get an example of the output. For setting DNS, you can describe the ingress external address with kubectl:
kubectl get service/apisix-ingress-controller-apisix-gateway -n ingress-apisix
Additionally, a kubeconfig file ~/.kube/eks-${account_id}-${region}-${cluster_name} will be created by the aws eks utility.
You can support this or any other of my projects:
- donationalerts.com/r/filipp_frizzy
- ETH 0xCD9fC1719b9E174E911f343CA2B391060F931ff7
- BTC bc1q8fhsj24f5ncv3995zk9v3jhwwmscecc6w0tdw3
