ДЗ#11 Хранилище секретов для приложения. Vault.

README.md

@@ -9,6 +9,7 @@
 [ДЗ#8 Мониторинг приложения в кластере](kubernetes-monitoring/README.md)   
 [ДЗ#9 Сервисы централизованного логирования для Kubernetes](kubernetes-logging/README.md)   
 [ДЗ#10 GitOps и инструменты поставки](kubernetes-gitops/README.md)   
+[ДЗ#11 Хранилище секретов для приложения. Vault.](kubernetes-vault/README.md)   
 ## Tricks, Tools, Hints
 
 ### k9s

kubernetes-vault/README.md

@@ -0,0 +1,122 @@
+## ДЗ#11 Хранилище секретов для приложения. Vault.
+
+### Задания:  
+- Данное задание будет выполняться в managed k8s в Yandex cloud
+- Разверните managed Kubernetes cluster в Yandex cloud любым удобным вам способом. Создайте 3 ноды для кластера
+- В namespace consul установите consul из helm-чарта https://github.com/hashicorp/consul-k8s.git с параметрами 3 реплики для сервера. Приложите команду установки чарта и файл с переменными к результатам ДЗ.
+- В namespace vault установите hashicorp vault из helm-чарта https://github.com/hashicorp/vault-helm.git
+  - Сконфигурируйте установку для использования ранее установленного consul в HA режиме
+  - Приложите команду установки чарта и файл с переменными к результатам ДЗ.
+- Выполните инициализацию vault и распечатайте с помощью полученного unseal key все поды хранилища
+- Создайте хранилище секретов otus/ с Secret Engine KV, а в нем секрет otus/cred, содержащий username='otus' password='asajkjkahs’
+- В namespace vault создайте serviceAccount с именем vault-auth и ClusterRoleBinding для него с ролью system:auth-delegator. Приложите получившиеся манифесты к результатам ДЗ
+- В Vault включите авторизацию auth/kubernetes и сконфигурируйте ее используя токен и сертификат ранее созданного ServiceAccount
+- Создайте и примените политику otus-policy для секретов /otus/cred с capabilities = [“read”, “list”]. Файл .hcl с политикой приложите к результатам ДЗ
+- Создайте роль auth/kubernetes/role/otus в vault с использованием ServiceAccount vault-auth из namespace Vault и политикой otus-policy
+- Установите External Secrets Operator из helm-чарта в namespace vault. Команду установки чарта и файл с переменными, если вы их используете приложите к результатам ДЗ
+- Создайте и примените манифест crd объекта SecretStore в namespace vault, сконфигурированный для доступа к KV секретам Vault с использованием ранее созданной роли otus и сервис аккаунта vault-auth. Убедитесь, что созданный SecretStore успешно подключился к vault. Получившийся манифест приложите к результатам ДЗ.
+- Создайте и примените манифест crd объекта ExternalSecret с следующими параметрами:
+  - ns – vault
+  - SecretStore – созданный на прошлом шаге
+  - Target.name = otus-cred
+  - Получает значения KV секрета /otus/cred из vault и отображает их в два ключа – username и password соответственно
+- Убедитесь, что после применения ExternalSecret будет создан Secret в ns vault с именем otus-cred и хранящий в себе 2 ключа username и password, со значениями, которые были сохранены ранее в vault. Добавьте манифест объекта ExternalSecret к результатам ДЗ.
+
+### Запуск 
+- Создать в yc необходимые ресурсы согласно описанию (Я это сделал в прошлом ДЗ)
+- helm install --createnamespace --namespace consul consul consul
+- helm install --createnamespace --namespace vault vault vault
+- helm install --namespace vault external-secrets external-secrets
+
+### Описание решения
+Для разворачивания consul я использовал репозитории hashicrop, установил количество реплик values.yaml:
+```yaml
+# consul
+server:
+  replicas: 3
+```
+Для vault аналогично, вкелючил ha режим и установил storage consul:
+```yaml
+...
+  ha:
+    enabled: true
+    config: |
+      ui = true
+
+...
+      storage "consul" {
+        path = "vault"
+        address = "consul-consul-server.consul.svc.cluster.local:8500"
+      }
+```
+Далее многие команды будут выполняться внутри пода vault:
+```sh
+kubectl exec --namespace vault -ti vault-2 -- vault operator init
+```
+Эта команда отдаст ключи и root токен необходымые для распечатывания контейнеров:
+```sh
+Unseal Key 1: S2p18bRj8ZIBnuPvk+RCcWiunboTjkkbO****
+Unseal Key 2: 8FQA90iN9KVpQMKiyDNwhIhNUVwd/0y/6****
+Unseal Key 3: KS+ItQn+hJR2s/iH1b9yhqyX4nFFXyzvWF****
+Unseal Key 4: AigrSESb7Mvv58JqLhmOAhdumDLTacNJq****
+Unseal Key 5: amUvB9+nXop4Kv4ggSFFi4tZhKwMNpuxs****
+
+Initial Root Token: hvs.m793pcUPXb****
+```
+Далее с помощью команд и ключей unseal нужно распечатать все поды:
+нужно следить за Seal Progress ввыводе и вводить ключи столько раз сколько потребуется
+```sh
+kubectl exec --namespace vault -ti vault-* -- vault operator unseal
+# enter Unseal Key
+kubectl exec --namespace vault -ti vault-* -- vault operator unseal
+# enter Unseal Key
+kubectl exec --namespace vault -ti vault-* -- vault operator unseal
+# enter Unseal Key
+...
+```
+```
+Создайте хранилище секретов otus/ с Secret Engine KV, а в нем секрет otus/cred, содержащий username='otus' password='asajkjkahs’
+```
+Я сделал это через web-ui:  
+![alt text](../img/image17.png)
+![alt text](../img/image18.png)
+
+Далее в файле vault-auth-sa.yaml создал serviceAccount с именем vault-auth и ClusterRoleBinding для него с ролью system:auth-delegator 
+Далее внутри pod vault-0 включил и настроил kubernetes авторизацию:
+```sh
+# kubectl exec --namespace vault -ti vault-0 -- sh
+vault auth enable kubernetes
+vault write auth/kubernetes/config \
+token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
+kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
+kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
+```
+Далее прописал политику для otus/cred и настроил роль:
+```sh
+# kubectl exec --namespace vault -ti vault-0 -- sh
+vault policy write otus-policy - <<EOF
+path "otus/*" {
+  capabilities = ["read", "list"]
+}
+EOF
+vault write auth/kubernetes/role/otus \
+bound_service_account_names=vault-auth \
+bound_service_account_namespaces=vault \
+policies=otus-policy \
+ttl=72h
+```
+Установил helm chart external-secrets:
+```sh
+helm repo add external-secrets https://charts.external-secrets.io
+
+helm install --namespace vault external-secrets external-secrets
+```
+
+и создал SecretStore и ExternalSecret в external-secret.yaml
+```sh
+kubectl apply -f external-secret.yaml
+
+kubectl get --all-namespaces externalsecrets.external-secrets.io -o wide
+NAMESPACE   NAME   STORE   REFRESH INTERVAL   STATUS         READY
+vault       otus   otus    1h                 SecretSynced   True
+```

kubernetes-vault/consul/.helmignore

@@ -0,0 +1,5 @@
+.git/
+.terraform/
+bin/
+test/
+crds/kustomization.yaml

kubernetes-vault/consul/Chart.yaml

@@ -0,0 +1,33 @@
+annotations:
+  artifacthub.io/images: |
+    - name: consul
+      image: hashicorp/consul:1.19.2
+    - name: consul-k8s-control-plane
+      image: hashicorp/consul-k8s-control-plane:1.5.3
+    - name: consul-dataplane
+      image: hashicorp/consul-dataplane:1.5.3
+    - name: envoy
+      image: envoyproxy/envoy:v1.25.11
+  artifacthub.io/license: MPL-2.0
+  artifacthub.io/links: |
+    - name: Documentation
+      url: https://www.consul.io/docs/k8s
+    - name: hashicorp/consul
+      url: https://github.com/hashicorp/consul
+    - name: hashicorp/consul-k8s
+      url: https://github.com/hashicorp/consul-k8s
+  artifacthub.io/prerelease: "false"
+  artifacthub.io/signKey: |
+    fingerprint: C874011F0AB405110D02105534365D9472D7468F
+    url: https://keybase.io/hashicorp/pgp_keys.asc
+apiVersion: v2
+appVersion: 1.19.2
+description: Official HashiCorp Consul Chart
+home: https://www.consul.io
+icon: https://raw.githubusercontent.com/hashicorp/consul-k8s/main/assets/icon.png
+kubeVersion: '>=1.22.0-0'
+name: consul
+sources:
+- https://github.com/hashicorp/consul
+- https://github.com/hashicorp/consul-k8s
+version: 1.5.3

kubernetes-vault/consul/README.md

@@ -0,0 +1,109 @@
+# Consul on Kubernetes Helm Chart
+
+---
+
+ **We're looking for feedback on how folks are using Consul on Kubernetes. Please fill out our brief [survey](https://hashicorp.sjc1.qualtrics.com/jfe/form/SV_4MANbw1BUku7YhL)!** 
+
+## Overview
+
+This is the Official HashiCorp Helm chart for installing and configuring Consul on Kubernetes. This chart supports multiple use cases of Consul on Kubernetes, depending on the values provided.
+
+For full documentation on this Helm chart along with all the ways you can use Consul with Kubernetes, please see the Consul and Kubernetes documentation.
+
+> :warning: **Please note**: We take Consul's security and our users' trust very seriously. If
+you believe you have found a security issue in Consul K8s, _please responsibly disclose_
+by contacting us at [[email protected]](mailto:[email protected]).
+
+## Features
+
+  * [**Consul Service Mesh**](https://www.consul.io/docs/k8s/connect):
+    Run Consul Service Mesh on Kubernetes. This feature
+    injects Envoy sidecars and registers your Pods with Consul.
+
+  * [**Catalog Sync**](https://www.consul.io/docs/k8s/service-sync):
+    Sync Consul services into first-class Kubernetes services and vice versa.
+    This enables Kubernetes to easily access external services and for
+    non-Kubernetes nodes to easily discover and access Kubernetes services.
+
+## Installation
+
+`consul-k8s` is distributed in multiple forms:
+
+  * The recommended installation method is the official
+    [Consul Helm chart](https://github.com/hashicorp/consul-k8s/tree/main/charts/consul). This will
+    automatically configure the Consul and Kubernetes integration to run within
+    an existing Kubernetes cluster.
+
+  * A [Docker image `hashicorp/consul-k8s-control-plane`](https://hub.docker.com/r/hashicorp/consul-k8s-control-plane) is available. This can be used to manually run `consul-k8s-control-plane` within a scheduled environment.
+
+  * Consul K8s CLI, distributed as `consul-k8s`, can be used to install and uninstall Consul Kubernetes. See the [Consul K8s CLI Reference](https://www.consul.io/docs/k8s/k8s-cli) for more details on usage. 
+
+### Prerequisites
+
+The following pre-requisites must be met before installing Consul on Kubernetes. 
+
+  * **Kubernetes 1.27.x - 1.30.x** - This represents the earliest versions of Kubernetes tested.
+    It is possible that this chart works with earlier versions, but it is
+    untested.
+  * Helm install
+    * **Helm 3.6+** for Helm based installs. 
+  * Consul K8s CLI based install
+    * `kubectl` configured to authenticate to a Kubernetes cluster with a valid `kubeconfig` file.
+    * `brew`, `yum`, or `apt` package manager on your local machine 
+
+### CLI
+
+The Consul K8s CLI is the easiest way to get up and running with Consul on Kubernetes. See [Install Consul on K8s CLI](https://developer.hashicorp.com/consul/docs/k8s/installation/install-cli#install-the-cli) for more details on installation, and refer to 
+[Consul on Kubernetes CLI Reference](https://developer.hashicorp.com/consul/docs/k8s/k8s-cli) for more details on subcommands and a list of all available flags
+for each subcommand. 
+
+
+ 1. Install the HashiCorp tap, which is a repository of all Homebrew packages for HashiCorp:
+
+    ``` bash
+    brew tap hashicorp/tap
+    ```
+
+2. Install the Consul K8s CLI with hashicorp/tap/consul formula.
+
+    ``` bash
+    brew install hashicorp/tap/consul-k8s
+    ```
+
+3. Issue the install subcommand to install Consul on Kubernetes:
+
+    ``` bash 
+    consul-k8s install 
+    ```
+
+### Helm
+
+The Helm chart is ideal for those who prefer to use Helm for automation for either the installation or upgrade of Consul on Kubernetes. The chart supports multiple use cases of Consul on Kubernetes, depending on the values provided. Detailed installation instructions for Consul on Kubernetes are found [here](https://www.consul.io/docs/k8s/installation/overview). 
+
+1. Add the HashiCorp Helm repository:
+
+    ``` bash
+    helm repo add hashicorp https://helm.releases.hashicorp.com
+    ```
+
+2. Ensure you have access to the Consul Helm chart and you see the latest chart version listed. If you have previously added the 
+   HashiCorp Helm repository, run `helm repo update`. 
+
+    ``` bash
+    helm search repo hashicorp/consul
+    ```
+
+3. Now you're ready to install Consul! To install Consul with the default configuration using Helm 3.2 run the following command below.
+   This will create a `consul` Kubernetes namespace if not already present, and install Consul on the dedicated namespace. 
+ 
+   ``` bash
+   helm install consul hashicorp/consul --set global.name=consul --create-namespace -n consul
+
+Please see the many options supported in the `values.yaml`
+file. These are also fully documented directly on the
+[Consul website](https://www.consul.io/docs/platform/k8s/helm.html).
+
+## Tutorials
+
+You can find examples and complete tutorials on how to deploy Consul on 
+Kubernetes using Helm on the [HashiCorp Learn website](https://learn.hashicorp.com/collections/consul/kubernetes).

kubernetes-vault/consul/addons/gen.sh

@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+# Copyright (c) HashiCorp, Inc.
+# SPDX-License-Identifier: MPL-2.0
+
+
+WD=$(dirname "$0")
+WD=$(cd "$WD"; pwd)
+
+set -eux
+
+TEMPLATES="${WD}/../templates"
+DASHBOARDS="${WD}/dashboards"
+TMP=$(mktemp -d)
+
+# create Prometheus template
+helm template prometheus prometheus \
+  --repo https://prometheus-community.github.io/helm-charts \
+  --namespace "replace-me-namespace" \
+  --version 13.2.1 \
+  -f "${WD}/values/prometheus.yaml" \
+  > "${TEMPLATES}/prometheus.yaml"
+
+# Find and replace `replace-me-namespace` with `{{ .Release.Namespace }}` in Prometheus template.
+sed -i'.orig' 's/replace-me-namespace/{{ .Release.Namespace }}/g' "${TEMPLATES}/prometheus.yaml"
+# Add a comment to the top of the template file mentioning that the file is auto-generated.
+sed -i'.orig' '1i\
+# This file is auto-generated, see addons/gen.sh
+' "${TEMPLATES}/prometheus.yaml"
+# Add `{{- if .Values.prometheus.enabled }} to the top of the Prometheus template to ensure it is only templated when enabled.
+sed -i'.orig' '1i\
+{{- if .Values.prometheus.enabled }}
+' "${TEMPLATES}/prometheus.yaml"
+# Add `{{- end }} to the bottom of the Prometheus template to ensure it is only templated when enabled (closes the `if` statement).
+sed -i'.orig' -e '$a\
+{{- end }}' "${TEMPLATES}/prometheus.yaml"
+# Remove the `prometheus.yaml.orig` file that is created as a side-effect of the `sed` command on OS X.
+rm "${TEMPLATES}/prometheus.yaml.orig"

kubernetes-vault/consul/addons/values/prometheus.yaml

@@ -0,0 +1,21 @@
+# Copyright (c) HashiCorp, Inc.
+# SPDX-License-Identifier: MPL-2.0
+
+# Disable non-essential components
+alertmanager:
+  enabled: false
+pushgateway:
+  enabled: false
+kubeStateMetrics:
+  enabled: false
+nodeExporter:
+  enabled: false
+server:
+  podAnnotations:
+    "consul.hashicorp.com/connect-inject": "false"
+  persistentVolume:
+    enabled: false
+  readinessProbeInitialDelay: 0
+  # Speed up scraping a bit from the default
+  global:
+    scrape_interval: 15s

kubernetes-vault/consul/templates/NOTES.txt

@@ -0,0 +1,21 @@
+
+Thank you for installing HashiCorp Consul!
+
+Your release is named {{ .Release.Name }}.
+
+To learn more about the release, run:
+
+  $ helm status {{ .Release.Name }} {{- if .Release.Namespace }} --namespace {{ .Release.Namespace }}{{ end }}
+  $ helm get all {{ .Release.Name }} {{- if .Release.Namespace }} --namespace {{ .Release.Namespace }}{{ end }}
+
+Consul on Kubernetes Documentation:
+https://www.consul.io/docs/platform/k8s
+
+Consul on Kubernetes CLI Reference:
+https://www.consul.io/docs/k8s/k8s-cli
+
+{{- if (and .Values.global.acls.manageSystemACLs (gt (len .Values.server.extraConfig) 3)) }}
+Warning: Defining server extraConfig potentially disrupts the automatic ACL
+         bootstrapping required settings. This may cause future issues if
+         there are conflicts.
+{{- end }}