monitoring hashicorp vault telemetry with prometheus and grafana

Author: star3am

ansible-tower/README.md

@@ -930,8 +930,10 @@ null_resource.awx_cli: Creation complete after 36s [id=936123805330159798]
 Apply complete! Resources: 1 added, 0 changed, 1 destroyed.
 ```
 
-## The Code
+## Ansible AWX Tower Vagrant Provisioner
 
 [filename](ansible-tower.sh ':include :type=code')
 
+## Terraform calling Ansible AWX
+
 [filename](main.tf ':include :type=code hcl')

apache-airflow/README.md

@@ -56,7 +56,7 @@ The `test-ssh.py` just ssh into hashiqube to test the connection
 - https://airflow.readthedocs.io/_/downloads/en/1.10.2/pdf/
 - https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html
 
-## The Code
+## Apache Airflow Vagrant Provisioner
 
 [filename](apache-airflow.sh ':include :type=code')
 

database/README.md

@@ -210,7 +210,7 @@ mysql: [Warning] Using a password on the command line interface can be insecure.
 | root             | localhost |
 +------------------+-----------+
 ```
-## The Code
+## MySQL Vagrant Provisioner
 
 [filename](mysql.sh ':include :type=code')
 
@@ -360,7 +360,7 @@ NT AUTHORITY\SYSTEM
 (21 rows affected)
 ```
 
-## The Code
+## MSSQL Vagrant Provisioner
 
 [filename](mssql.sh ':include :type=code')
 
@@ -453,6 +453,6 @@ Bringing machine 'hashiqube0.service.consul' up with 'virtualbox' provider...
     hashiqube0.service.consul:  v-token-postgres-3AhBH3pbmVNnkbxXV8K3-1598841098 | Password valid until 2020-08-31 03:31:43+00                | {}
 ```
 
-## The Code
+## PostgreSQL Vagrant Provisioner
 
 [filename](postgresql.sh ':include :type=code')

dbt/README.md

@@ -123,7 +123,7 @@ In this way, if we have already run the model, the next RUN and TEST will execut
 
 This would open space as well to implement cool stuff, such as running the model over a pull request. 
 
-## The Code
+## DBT Vagrant Provisioner
 
 [filename](common.sh ':include :type=code')
 

docker/README.md

@@ -557,6 +557,6 @@ Bringing machine 'user.local.dev' up with 'virtualbox' provider...
     user.local.dev: vagrant ssh -c "docker exec -it apache2 /bin/bash -c "apache2 -t -v""
 ```
 
-## The Code
+## Docker Vagrant Provisioner
 
 [filename](docker.sh ':include :type=code')

docsify/README.md

@@ -75,6 +75,6 @@ Bringing machine 'user.local.dev' up with 'virtualbox' provider...
 After provision, you can access Docsify and HashiQube documentation at http://localhost:3333/
 ![Docsify](images/docsify.png?raw=true "Docsify")
 
-## The Code
+## Docsify Vagrant Provisioner
 
 [filename](docsify.sh ':include :type=code')

hashicorp/README.md

@@ -154,7 +154,7 @@ Packer uses the HashiCorp Configuration Language - HCL - designed to allow conci
 ### Windows 2019
 
 [filename](packer/windows/windowsserver/windows-2019.pkr.hcl ':include :type=code')
-## The Code
+## Packer Vagrant Provisioner
 
 [filename](packer.sh ':include :type=code')
 
@@ -261,7 +261,7 @@ can't guarantee that exactly these actions will be performed if
 "terraform apply" is subsequently run.
 ```
 
-## The Code
+## Terraform Vagrant Provisioner
 
 [filename](terraform.sh ':include :type=code')
 
@@ -344,7 +344,13 @@ Bringing machine 'user.local.dev' up with 'virtualbox' provider...
 ```
 ![Vault](images/vault.png?raw=true "Vault")
 
-## The Code
+## Monitoring Hashicorp Vault
+
+We use Prometheus and Grafana to Monitor Vault
+
+See: [__Monitoring Hashicorp Vault__](prometheus-grafana/README?id=monitoring-hashicorp-vault)
+
+## Vault Vagrant Provisioner
 
 [filename](vault.sh ':include :type=code')
 
@@ -486,7 +492,7 @@ When you register a service in Consul all you need to add is a tag that announce
 Fabio runs as a Nomad job, see `hashicorp/nomad/jobs/fabio.nomad`
 Some routes are added via Consul, see `hashicorp/consul.sh`
 
-## The Code
+## Nomad Vagrant Provisioner
 
 [filename](nomad.sh ':include :type=code')
 
@@ -663,7 +669,7 @@ Bringing machine 'user.local.dev' up with 'virtualbox' provider...
 ```    
 ![Consul](images/consul.png?raw=true "Consul")
 
-## The Code
+## Consul Vagrant Provisioner
 
 [filename](consul.sh ':include :type=code')
 
@@ -1097,7 +1103,7 @@ c97d646ce0ef: Already exists :
     hashiqube0.service.consul: bM152PWkXxfoy4vA51JFhR7LmV9FA9RLbSpHoKrysFnwnRCAGzV2RExsyAmBrHu784d1WZRW6Cx4MkhvWzkDHvEn49c4wkSZYScfJ
 ```
 
-## The Code
+## Waypoint Vagrant Provisioner
 
 [filename](waypoint.sh ':include :type=code')
 
@@ -1181,6 +1187,6 @@ Bringing machine 'hashiqube0.service.consul' up with 'virtualbox' provider...
     hashiqube0.service.consul: /tmp/vagrant-shell: line 5:  5093 Terminated              sh -c 'sudo tail -f /var/log/boundary.log | { sed "/worker successfully authed/ q" && kill $$ ;}'
 ```
 
-## The Code
+## Boundary Vagrant Provisioner
 
 [filename](boundary.sh ':include :type=code')

hashicorp/vault.sh

@@ -128,6 +128,12 @@ storage "raft" {
 #  address = "127.0.0.1:8500"
 #  path    = "vault"
 # }
+# https://developer.hashicorp.com/vault/docs/configuration/telemetry
+# https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus
+telemetry {
+  disable_hostname = true
+  prometheus_retention_time = "12h"
+}
 api_addr             = "http://10.9.99.10:8200"
 max_lease_ttl        = "10h"
 default_lease_ttl    = "10h"

hashiqube/README.md

@@ -8,7 +8,7 @@ This Provider (Basetools) installs some essential tools that Hashiqube provision
 swapspace rkhunter jq curl unzip software-properties-common bzip2 git make python3.9 python3-pip python3-dev python3-venv python3-virtualenv golang-go apt-utils ntp update-motd toilet figlet nano iputils-ping dnsutils iptables telnet
 ```
 
-## The Code
+## Hashiqune Vagrant Provisioner
 
 [filename](basetools.sh ':include :type=code')
 

localstack/README.md

@@ -130,7 +130,7 @@ Now check the assets with aws local inside vagrant
 2006-02-04 03:45:09 localstack-s3-bucket
 ```
 
-## The Code
+## Localstack Terraform Examples
 
 [filename](variables.tf ':include :type=code')
 

minikube/README.md

@@ -144,6 +144,6 @@ This guide explains how to use Traefik as an Ingress controller for a Kubernetes
 
 ![Traefik on Minikube](images/minikube-traefik-dashboard.png?raw=true "Traefik on Minikube")
 
-## The Code
+## Minikube Vagrant Provisioner
 
 [filename](minikube.sh ':include :type=code')

prometheus-grafana/README.md

@@ -16,7 +16,9 @@ Prometheus is an open source monitoring system for which Grafana provides out-of
 
 In order to provision Prometheus and Grafana, you need bastetools, docker, minikube as dependencies. 
 
-`vagrant up --provision-with basetools,docker,minikube,prometheus-grafana`
+:bulb: We enable Vault, because we monitor it with Prometheus and we enable Minikube because we host Grafana and Prometheus on Minikkube using Helm
+
+`vagrant up --provision-with basetools,docker,vault,minikube,prometheus-grafana`
 
 Prometheus http://localhost:9090 <br />
 Alertmanager http://localhost:9093 <br />
@@ -73,4 +75,55 @@ and you should be able to see some graphs.
 
 ![Grafana Dashboard Kubernetes Cluster (Prometheus)](images/grafana_dashboard_6417.png?raw=true "Grafana Dashboard Kubernetes Cluster (Prometheus)")
 
+## Monitoring Hashicorp Vault
+
+https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus <br />
+https://developer.hashicorp.com/vault/docs/configuration/telemetry
+
+In hashicorp/vault.sh we enabled Telemetry in the Vault config file see `hashicorp/vault.sh`
+
+```hcl
+# https://developer.hashicorp.com/vault/docs/configuration/telemetry
+# https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus
+telemetry {
+  disable_hostname = true
+  prometheus_retention_time = "12h"
+}
+```
+
+When we install Prometheus with Helm we set a values.yaml file that specify an `extraScrapeConfigs` You guessed it! Vault...
+
+`helm install prometheus prometheus-community/prometheus -f /vagrant/prometheus-grafana/values.yaml`
+
+[filename](values.yaml ':include :type=code')
+
+You should now see the Vault target in Prometheus web interface at http://localhost:9090/targets
+
+![Prometheus Vault Target](images/prometheus-targets-vault.png?raw=true "Prometheus Vault Target")
+
+We now need to Grafana Datasource of Type Prometheus based on this Target
+
+Please navigate to http://localhost:3000/connections/your-connections/datasources
+
+And add a Vault Datasource
+
+Name: Vault
+URL: http://10.9.99.10:9090
+
+![Grafana Datasource Prometheus Vault](images/grafana-datasource-prometheus-vault.png?raw=true "Grafana Datasource Prometheus Vault")
+
+Now, let's import the Vault Grafana Dashboard, to do that, click on the top right + and select `Import Dashboard` ref: https://grafana.com/grafana/dashboards/12904-hashicorp-vault/
+
+![Grafana Import Dashboard Vault 12904](images/grafana-import-dashboard-vault-12904.png?raw=true "Grafana Import Dashboard Vault 12904")
+
+Enter `12904` and click on Load
+
+![Grafana Import Dashboard Vault 12904 Load](images/grafana-import-dashboard-vault-12904-load.png?raw=true "Grafana Import Dashboard Vault 12904 Load")
+
+Navigating to Grafana -> Dashboards you should now be able to see the Hashicorp Vault Grafana Dashboard
+
+![Grafana Hashicorp Vault Dashboard](images/grafana-hashicorp-vault-dashboard.png?raw=true "Grafana Hashicorp Vault Dashboard")
+
+## Prometheus Grafana Vagrant Provisioner 
 
+[filename](prometheus-grafana.sh ':include :type=code')

prometheus-grafana/images/grafana-datasource-prometheus-vault.png

@@ -43,10 +43,17 @@ echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ helm search repo prometheus-community"
 echo -e '\e[38;5;198m'"++++ "
 sudo --preserve-env=PATH -u vagrant helm search repo prometheus-community
+
+# https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus
+echo -e '\e[38;5;198m'"++++ "
+echo -e '\e[38;5;198m'"++++ Set Vault token in values.yaml for prometheus for monitoring Vault"
+echo -e '\e[38;5;198m'"++++ "
+sed -i "s/VAULT_TOKEN/$VAULT_TOKEN/g" /vagrant/prometheus-grafana/values.yaml
+
 echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ helm install prometheus prometheus-community/prometheus"
 echo -e '\e[38;5;198m'"++++ "
-sudo --preserve-env=PATH -u vagrant helm install prometheus prometheus-community/prometheus
+sudo --preserve-env=PATH -u vagrant helm install prometheus prometheus-community/prometheus -f /vagrant/prometheus-grafana/values.yaml
 
 echo -e '\e[38;5;198m'"++++ "
 echo -e '\e[38;5;198m'"++++ Helm add Grafana repo"
@@ -129,6 +136,17 @@ done
 
 ps aux | grep kubectl | grep -ve sudo -ve grep -ve bin
 
+# https://developer.hashicorp.com/vault/tutorials/monitoring/monitor-telemetry-grafana-prometheus
+# https://developer.hashicorp.com/vault/docs/configuration/telemetry#prometheus
+echo -e '\e[38;5;198m'"++++ "
+echo -e '\e[38;5;198m'"++++ Vault policy write prometheus-metrics path /sys/metrics"
+echo -e '\e[38;5;198m'"++++ "
+vault policy write prometheus-metrics - << EOF
+path "/sys/metrics*" {
+  capabilities = ["read", "list"]
+}
+EOF
+
 # https://github.com/grafana/grafana/issues/29296
 echo -e '\e[38;5;198m'"++++ Prometheus http://localhost:9090"
 echo -e '\e[38;5;198m'"++++ Alertmanager http://localhost:9093"

prometheus-grafana/images/grafana-hashicorp-vault-dashboard.png

@@ -0,0 +1,24 @@
+extraScrapeConfigs: |
+  - job_name: vault
+    metrics_path: /v1/sys/metrics
+    params:
+      format: ['prometheus']
+    scheme: http
+    bearer_token: "hvs.m29F723W1wV8R8ths0ycs71x"
+    static_configs:
+    - targets: ['10.9.99.10:8200']
+  # - job_name: 'prometheus-blackbox-exporter'
+  #   metrics_path: /probe
+  #   params:
+  #     module: [http_2xx]
+  #   static_configs:
+  #     - targets:
+  #       - https://example.com
+  #   relabel_configs:
+  #     - source_labels: [__address__]
+  #       target_label: __param_target
+  #     - source_labels: [__param_target]
+  #       target_label: instance
+  #     - target_label: __address__
+  #       replacement: prometheus-blackbox-exporter:9115
+