Swarm Cluster, shared volumes, HA

[hchas]

Hi Bret,

My name is Hernan and first of all, sorry for my english (is not my native language) and thank you in advance.

I've taken your Udemy course "Docker Swarm Mastery" and it was great.

I'm planning to build a Swarm cluster with three nodes, all managers/workers on premise and i have some of questions concerning shared storage, reverse proxy and backups that i'm sure you can help me to clarify.

A little description of the solution i'm planning:

• Three nodes Managers/Workers
• Portainer
• Private Docker Registry
• Reverse proxy/load balancer (NGINX) for load balance between the three nodes ips and use only one ip for the hole cluster, and map port 443 to the published service's ports
• Other things like Prometheus, Loki, Grafana, etc.

First, what is the best approach to obtain high availavility of the cluster regarding shared storage?
For example, is a good idea to mount on the three nodes an nfs volume on /var/lib/docker (or /var/lib/docker/volumes)?

Or is better to manage that on a case by case basis?
For example, one shared nfs volume for the Portainer data, one for images of the private registry, etc.?
In this case, what is the recomended way? Volumes mounted in the three nodes and reference it as local paths or manage the network volumes in the service's configs?

Second, regarding reverse proxy in HA.
My idea is install NGINX directly on the three hosts and use Keepalived to implement vrrp.

Is that a sensible approach?
What do you recomend in the case that it's not?

Again, thank you very much for the help.

Regards,
Hernan

[BretFisher]

Hey Hernan, Thanks for taking my course. That's an excellent plan. There are lots of pieces that will take you some time to work through. I can't cover all those topics in a reply, so here's a few things that can help you get going:

1. I'm helping moderate a Swarm Fans community in my Discord server at http://devops.fan in the #swarm channel. They can help with a lot of these discussions.
2. They also help keep a tools list updated here that has some cluster volume options on it https://github.com/BretFisher/awesome-swarm
3. "cluster storage" in Swarm is hard, mostly because if you want storage to be available on all nodes (so containers can start anywhere and see their storage), the options are very limited in 2023. I generally recommend people start with only stateless containers when first using Swarm, to keep it much easier to setup and maintain. If you were to do NFS from a single node to the others, then you've lost data fault tolerance, so that's not ideal. It would be easier if you had NFS running elsewhere on highly-available hardware or something like EFS in AWS. But even NFS can be slow for databases and high read/writes, so there's no single answer for everyone. As long as you're doing regular volume backups, you could just use local storage and placement constraints to ensure the container is always on that node. You could check out fancy data solutions like Ceph (open source) or Portworx (free and paid options) but realize those can be their own projects and for "real world production" are not plug-and-play.
4. NGINX is a fine proxy, but Traefik might be an easier time on Swarm, which it has built-in support for.

In the end, I recommend starting simple and join our Discord to ask questions. Oh, and we have a bi-monthly video hangout called Swarm Fans that is *this Friday if you can join. This link should help get you the calendar invite after you join our server: https://discord.gg/nyvGcWrb?event=1148311041554534570

[chrisbecke]

Answering from my on-permise production swarm experience:

First, what is the best approach to obtain high availavility of the cluster regarding shared storage?

Depending on your situation, HA storage options vary. If you have access to enterprise level SANs, then you have access to HA NFS. If you need to create your own storage, then a distributed file system like glusterfs, ceph, etc. are easy to set up.

We went from SAN provided nfs to a glusterfs cluster. Both worked well.

For example, is a good idea to mount on the three nodes an nfs volume on /var/lib/docker (or /var/lib/docker/volumes)?

Each docker node needs to store its own data, and /var/lib/docker should for performance reasons be as close to the node as possible. Also, at the application level you generally want to be quite explicit about wether your stack is storing local (per node) or shared (cluster wide) data, so mounting /var/lib/docker[...] is not a good idea.

What people generally seem to do is mount one (or more) shared file systems on the node: e.g. /mnt/nfs-docker-0/ and then explicitly reference this host relative location as either bind mounts or as volumes with a specific target.

In our case - after looking at the available docker volume plugins - I made a docker volume plugin to allow docker volume xxx commands to operate on the shared gluster volume as I really wanted our stacks to be able to declare their own volumes without cluster operator intervention. This worked well in production for years.

Or is better to manage that on a case by case basis?
For example, one shared nfs volume for the Portainer data, one for images of the private registry, etc.?

Definately case by case basis. Given your use cases, you might actually want to include a Minio cluster as s3 compatible object storage is a great place to store registry data and will be used a lot in the grafana / loki space too. Alternately if you have access to SANs or if whatever distributed filesystem might have added s3 api support...

Keepalived to implement vrrp.

Definately this. keepalived must monitor the health of the docker service at least.
Deploy Traefik - rather than nginx - to each node - listening on 80 / 443 and let it ssl offload, and use service labels to route (and load balance) traffic to your services.
In our case Traefik runs in host mode, and is deployed with a node.role==manager constraint so it directly receives traffic from 80 and 443 on those nodes. I don't have keepalived monitoring 80, 443 or traefik itself on those nodes as I actually use the same virtual ip to access the manager nodes over :2376 for swarm management so I need to know docker is up more than anything. As long as docker is up I can assume everything else is fine, but as this swarm was part of our ci/cd pipelines, being able to not interrupt deployments even when a manager is down (for maintenance of course!) was important. ymmv.

It might add a hop, but if reliability rather than performance is a concern, then don't put the reverse proxy in host mode. As long as you hit :80/:443 on any healthy docker node, dockers own overlay lb will route to a healthy traefik/caddy/nginx (if you must) instance.