Deploy Nomad & Consult via ansible, on Centos nodes

What you will get?

• Setup a 3 nodes (1 server, 2 clients) hashicorp nomad container orchestrator
• Run all on your prefered cloud provider via ansible

Architecture of the stack

• Master/client nomad host: master will take care of electing cluster leader, planning and rescheduling nomad jobs. While client will report node status to the master and look for jobs to run, and then run container
• Consul: service discovery. Nomad will record nodes, services in this database. Consul is replicated and present on all nodes so nomad agents always have access to this data locally
• dnsmasq: a kind of proxy for dns resolution, route *.consul query to consul, and other query to localhost then internet
• Ansible: recipe will provide easy deployment of the solution

Prerequisit

• Ansible
• A cloud infra providing centos hosts

Deploy nomad cluster

Clone this repo

git clone https://github.com/gregbkr/nomad-consult-ansible-centos && cd nomad

Please deploy 3 standard Centos hosts (nomad-server1, nomad-client1, nomad-client2) on your prefered cloud provider.

Edit ansible inventory with your ips and ssh access key:

nano inventory

Deploy consul, docker, dnsmasq & nomad

Run few times until you got no more errors)

ansible-playbook playbook.yml

Checks

Consul: you can view consul UI on http://a_node_ip:8500/ Here are registered nomad clients and consul itself.

Dnsmask will help redirect dns queries to *.consul Ssh connect to a node and try some dsn query:

ping node1.node.consul.
ping node2.node.consul.
ping consul.service.consul.	

Run nomad jobs

We will deploy here a very simple application:

• One nginx which redirect 80 --> flask app on 5000
• Flask app, which records in a redis database a number of view
• A redis database, on port 6379

Each of these components exposes a service, registered and avalaible for query in consul.

Copy nomad jobs to a client node and connect to it:

scp -r -i ~/.ssh/id_rsa_sbexx jobs/* root@client_node_ip:/root/
ssh -i ~/.ssh/id_rsa_sbexx jobs root@client_node_ip

nomad run redis.nomad
nomad run flask.nomad
nomad run nginx.nomad

Test services

Redis

echo 'PING' | nc global-redis-check.service.consul 6379

flask

curl global-flask-check.service.consul:5000

Nginx

curl global-nginx-check.service.consul

Troubleshooting

check nomad status

nomad status
nomad server-members
nomad status redis
nomad alloc-status <alloc-id>

Delete nomad job

nomad stop redis