Merge pull request Skatteetaten#1 from fredrikhgrelland/feature/initi…

…al-draft

Initial draft and example

CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+## [0.0.1]
+
+### Added
+
+- Initial draft #1

Makefile

@@ -33,8 +33,12 @@ else
 endif
 
 # clean commands
-destroy-box:
+# clean commands
+destroy:
 	vagrant destroy -f
+	rm terraform.tfstate || true
+	rm terraform.tfstate.backup || true
+	rm example/terraform.tfstate || true
 
 remove-tmp:
 	rm -rf ./tmp
@@ -45,3 +49,15 @@ clean: destroy-box remove-tmp
 update-box:
 	@SSL_CERT_FILE=${SSL_CERT_FILE} CURL_CA_BUNDLE=${CURL_CA_BUNDLE} vagrant box update || (echo '\n\nIf you get an SSL error you might be behind a transparent proxy. \nMore info https://github.com/fredrikhgrelland/vagrant-hashistack/blob/master/README.md#if-you-are-behind-a-transparent-proxy\n\n' && exit 2)
 
+# to-hivemetastore
+proxy-h:
+	consul connect proxy -service hivemetastore-local -upstream hive-metastore:9083 -log-level debug
+# to-minio
+proxy-m:
+	consul connect proxy -service minio-local -upstream minio:9000 -log-level debug
+# to-postgres
+proxy-p:
+	consul connect proxy -service postgres-local -upstream postgres:5432 -log-level debug
+# to-presto
+proxy-p:
+	consul connect proxy -service postgres-local -upstream presto:8080 -log-level debug

README.md

@@ -1,90 +1,2 @@
-# Starter template for `fredrikhgrelland/hashistack`
+# Terraform-nomad-presto
 
-This repository can be used as a base for developing services on the hashistack.
-On github, you may use the ["Use this template"](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to generate a new repository from this template.
-
-If you found this in `fredrikhgrelland/vagrant-hashistack`, you may be interested in this separate repository [vagrant-hashistack-template](https://github.com/fredrikhgrelland/vagrant-hashistack-template/generate) button to start a new repository from this repo
-.
-
-Documentation on [parent repository](https://github.com/fredrikhgrelland/vagrant-hashistack#usage).
-
-## Customizing and using the vagrant box
-
-### Building and testing docker image
-See docker [README.md](docker/README.md).
-
-### Starting a box
-The vagrant box ships with a default startup scheme. `make` from this directory will start the box, and it will run all books in [dev/ansible](dev/ansible) in lexical order (NB: `playbook.yml` is run first, but is only used to run all other playbooks) after the bootstrap-process for the hashistack is done. In the [example](test_example/dev/ansible/playbook.yml) we use it to start terraform which then starts a nomad-job.
-
-### Pre and post hashistack procedure
-You may change the hashistack configuration or add aditional pre and post steps to the startup procedure to match your needs.
-Detailed documentation in [dev/vagrant/conf/README.md](dev/vagrant/conf/README.md)
-
-### Pre packaged configuration switches
-
-The box comes standard with a set of environment switches to simplify testing of different scenarios and enable staged development efforts.
-
-NB: All lowercase variables will automatically get a corresponding TF_VAR_ prepended variant for use directly in terraform.
-To change from the default value, you may add the environment variable to [.env](dev/.env)
-
-#### Enterprise vs Open Source Software (OSS)
-As long as Enterprise is not set to `true` the box will utilise OSS version of the binaries.
-
-#### Nomad
-
-| default   | environment variable  |  value  |
-|:---------:|:----------------------|:-------:|
-|           | nomad_enterprise      |  true   |
-|     x     | nomad_enterprise      |  false  |
-|           | nomad_acl             |  true   |
-|     x     | nomad_acl             |  false  |
-
-When ACLs in Nomad are enabled the bootstrap token will be available in vault under `secret/nomad/management-token` with the two key-value pairs `accessor-id` and `secret-id`. `secret-id` is the token itself. These can be accessed in several ways:
-- From inside the vagrant box with `vault kv get secret/nomad-bootstrap-token`
-- From local machine with `vagrant ssh -c vault kv get secret/nomad-bootstrap-token"`
-- By going to vault's UI on `localhost:8200`, and signing in with the root token.
-
-#### Consul
-
-| default   | environment variable             |  value  |
-|:---------:|:---------------------------------|:-------:|
-|           | consul_enterprise                |  true   |
-|     x     | consul_enterprise                |  false  |
-|     x     | consul_acl                       |  true   |
-|           | consul_acl                       |  false  |
-|     x     | consul_acl_default_policy        |  allow  |
-|           | consul_acl_default_policy        |  deny   |
-
-#### Vault
-
-| default   | environment variable             |  value  |
-|:---------:|:---------------------------------|:-------:|
-|           | vault_enterprise                 |  true   |
-|     x     | vault_enterprise                 |  false  |
-
-##### Consul secrets engine
-
-If `consul_acl_default_policy` has value `deny`, it will also enable [consul secrets engine](https://www.vaultproject.io/docs/secrets/consul) in vault.  
-Ansible will provision additional custom roles (admin-team, dev-team), [policies](../ansible/templates/consul-policies) and tokens for test purpose with different access level.
-
-How to generate token:
-```text
-# generate token for dev team member
-vagrant ssh -c 'vault read consul/creds/dev-team'
-
-# generate token for admin team member
-vagrant ssh -c 'vault read consul/creds/admin-team'
-```
-
-*Tokens can be used to access UI (different access level depends on role)
-
-## Vagrant box life-cycle
-1. `/home/vagrant/.env_default` - _preloaded_ - default variables
-1. `vagrant/.env` - _user provided_ - variables override
-1. `vagrant/.env_override` - _system provided_ - variables are overridden for test purposes
-1. `vagrant/dev/vagrant/conf/pre_ansible.sh` - _user provided_ - script running before ansible bootstrap procedure
-1. `vagrant/dev/vagrant/conf/pre_bootstrap/*.yml` - _user provided_ - pre bootstrap tasks, running before hashistack software runs and ready
-1. `/etc/ansible/bootstrap.yml` - _preloaded_ - verify ansible variables and software configuration, run hashistack software & verify that it started correctly
-1. `vagrant/conf/post_bootstrap/*.yml` - _user provided_ - poststart scripts, running after hasistack software runs and ready
-1. `vagrant/dev/conf/pre_ansible.sh` - _user provided_ - script running after ansible bootstrap procedure
-1. `vagrant/ansible/*.yml` - _user provided_ - ansible tasks included in playbook

Vagrantfile

@@ -3,10 +3,11 @@ Vagrant.configure("2") do |config|
     config.vm.box_version = ">= 0.3, < 0.4"
     config.vm.provider "virtualbox" do |vb|
         vb.linked_clone = true
-        vb.memory = 2048
+        vb.cpus = 3
+        vb.memory = 8192
+    end
+    config.vm.provision "ansible_local" do |ansible|
+        ansible.provisioning_path = "/vagrant/dev/ansible"
+        ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml
     end
-        config.vm.provision "ansible_local" do |ansible|
-            ansible.provisioning_path = "/vagrant/dev/ansible"
-            ansible.playbook = "playbook.yml" # Note this playbook is, in this context, /ansible/playbook.yml
-        end
 end

conf/nomad/presto.hcl

@@ -0,0 +1,177 @@
+job "${nomad_job_name}" {
+  type = "service"
+  datacenters   = "${datacenters}"
+  namespace     = "${namespace}"
+
+  update {
+    max_parallel      = 1
+    health_check      = "checks"
+    min_healthy_time  = "10s"
+    healthy_deadline  = "12m"
+    progress_deadline = "15m"
+    auto_revert       = true
+    auto_promote      = true
+    canary            = 1
+    stagger           = "30s"
+  }
+  group "standalone" {
+    count = 1
+
+    network {
+      mode = "bridge"
+    }
+
+    service {
+      name = "${service_name}"
+      port = "${port}"
+      connect {
+        sidecar_service {
+          proxy {
+            upstreams {
+              destination_name = "${hivemetastore_service_name}"
+              local_bind_port  = "${hivemetastore_port}"
+            }
+            upstreams {
+              destination_name = "${minio_service_name}"
+              local_bind_port  = "${minio_port}"
+            }
+          }
+        }
+      }
+//      check {
+//        task     = "server"
+//        name     = "presto-hive-availability"
+//        type     = "script"
+//        command  = "presto"
+//        args     = ["--execute", "SHOW TABLES IN hive.default"]
+//        interval = "30s"
+//        timeout  = "15s"
+//      }
+      check {
+        expose   = true
+        name     = "presto-info"
+        type     = "http"
+        path     = "/v1/info"
+        interval = "10s"
+        timeout  = "2s"
+      }
+      check {
+        expose   = true
+        name     = "presto-node"
+        type     = "http"
+        path     = "/v1/node"
+        interval = "10s"
+        timeout  = "2s"
+      }
+    }
+
+    task "waitfor-hive-metastore" {
+      restart {
+        attempts = 100
+        delay    = "5s"
+      }
+      lifecycle {
+        hook = "prestart"
+      }
+      driver = "docker"
+      resources {
+        memory = 32
+      }
+      config {
+        image = "consul:latest"
+        entrypoint = ["/bin/sh"]
+        args = ["-c", "jq </local/service.json -e '.[].Status|select(. == \"passing\")'"]
+        volumes = ["tmp/service.json:/local/service.json" ]
+      }
+      template {
+        destination = "tmp/service.json"
+        data = <<EOH
+          {{- service "${hivemetastore_service_name}" | toJSON -}}
+        EOH
+      }
+    }
+
+    task "waitfor-minio" {
+      restart {
+        attempts = 100
+        delay    = "5s"
+      }
+      lifecycle {
+        hook = "prestart"
+      }
+      driver = "docker"
+      resources {
+        memory = 32
+      }
+      config {
+        image = "consul:latest"
+        entrypoint = ["/bin/sh"]
+        args = ["-c", "jq </local/service.json -e '.[].Status|select(. == \"passing\")'"]
+        volumes = ["tmp/service.json:/local/service.json" ]
+      }
+      template {
+        destination = "tmp/service.json"
+        data = <<EOH
+          {{- service "${minio_service_name}" | toJSON -}}
+        EOH
+      }
+    }
+
+    task "server" {
+      driver = "docker"
+
+      config {
+        image = "${image}"
+        volumes = [
+          "local/presto/config.properties:/lib/presto/default/etc/config.properties",
+          "local/presto/catalog/hive.properties:/lib/presto/default/etc/catalog/hive.properties",
+        ]
+      }
+      template {
+        data = <<EOH
+          MINIO_ACCESS_KEY = "${minio_access_key}"
+          MINIO_SECRET_KEY = "${minio_secret_key}"
+          EOH
+        destination = "secrets/.env"
+        env         = true
+      }
+      // NB! If credentials set as env variable, during spin up of this container it could be sort of race condition and query `SELECT * FROM hive.default.iris;`
+      //     could end up with exception: The AWS Access Key Id you provided does not exist in our records.
+      //     Looks like, slow render of env variables (when one template depends on other template). Maybe because, all runs on local machine
+      template {
+        destination = "/local/presto/catalog/hive.properties"
+        data = <<EOH
+connector.name=hive-hadoop2
+hive.metastore.uri=thrift://{{ env "NOMAD_UPSTREAM_ADDR_${hivemetastore_service_name}" }}
+hive.metastore-timeout=1m
+hive.s3.aws-access-key=${minio_access_key}
+hive.s3.aws-secret-key=${minio_secret_key}
+hive.s3.endpoint=http://{{ env "NOMAD_UPSTREAM_ADDR_${minio_service_name}" }}
+hive.s3.path-style-access=true
+hive.s3.ssl.enabled=false
+hive.s3.socket-timeout=15m
+EOH
+      }
+      template {
+        destination   = "local/presto/config.properties"
+        data = <<EOH
+node-scheduler.include-coordinator=true
+http-server.http.port=${port}
+discovery-server.enabled=true
+discovery.uri=http://127.0.0.1:${port}
+EOH
+      }
+      template {
+        destination = "local/data/.additional-envs"
+        change_mode = "noop"
+        env         = true
+        data        = <<EOF
+${envs}
+EOF
+      }
+      resources {
+        memory = 2048
+      }
+    }
+  }
+}

dev/ansible/02_run_terraform.yml

@@ -0,0 +1,10 @@
+- name: Terraform
+  terraform:
+    project_path: ../../example
+    force_init: true
+    state: present
+  register: terraform
+
+- name: Terraform stdout
+  debug:
+    msg: "{{terraform.stdout}}"