How to run Infinispan server on Docker Swarm for Keycloak?

[jerrac]

I've been trying to get Keycloak HA working for the past while and I'm currently stuck. Specifically I'm running Keycloak on Docker Swarm. Which means that when I deploy to my test cluster, the embedded Infinispan cache hits moby/libnetwork#552 and fails to create a cluster between each replica.

So I thought I would try running Infinispan server separately. I thought that would let me bypass the multicast issue. But it seems like server also wants to use multicast to get the cluster working.

Which leaves me wondering how I could tell Infinispan to just look for "x, y, and z" hostnames to create the cluster without using multicast.

I have been reading, and searching, the docs, but I find it oddly hard to get anything out of them. Which means I haven't been able to find the answer, even if I might have read it already... :\

Is there a way to bypass multicast discovery? How would I do that?

Here's my (sanitized) service definition:

  inifinispan-server:
    image: infinispan/server:14.0
    container_name: "inifinispan-server-{{ .Task.Slot }}"
    hostname: "testcloakcache-{{ .Task.Slot }}.domain.tld"
    command:
      - "--bind-address=0.0.0.0"
      - "-Dinfinispan.cluster.stack=tcp"
      - "-Dinfinispan.cluster.name=testcloakcachecluster"
      - '-Djgroups.dns.query="inifinispan-server"'
    restart: always
    networks:
      traefiknet:
      idmnet:
    environment:
      IDENTITIES_BATCH: "/data/identities.batch"
      JAVA_OPTIONS: "-Xms1300m -Xmx1300m"
    volumes:
      - type: volume
        source: project_cache_data
        target: /data
      - type: volume
        source: project_data_share
        target: /opt/share
    labels:
      traefik.enable: "true"
      traefik.docker.network: traefiknet
      traefik.http.routers.project-cloakcache-http.entrypoints: "web"
      traefik.http.routers.project-cloakcache-http.rule: "Host(`testcloakcache`) || Host(`testcloakcache.domain.tld`)"
      traefik.http.routers.project-cloakcache-http.service: "project-cloakcache"
      traefik.http.services.project-cloakcache.loadbalancer.server.port: 11222
      traefik.http.services.project-cloakcache.loadbalancer.sticky.cookie: "true"
    deploy:
      replicas: 3
      resources:
        limits:
          cpus: "1.0"
          memory: "1500M"
      restart_policy:
        condition: any
      placement:
        constraints:
          - "node.role==worker"
        preferences:
          - spread: "node.labels.internalhostname"
      rollback_config:
        max_failure_ratio: 0.33
        failure_action: pause
        parallelism: 1
        delay: 45s
        monitor: 30s
      update_config:
        max_failure_ratio: 0.33
        failure_action: rollback
        parallelism: 1
        delay: 45s
        monitor: 30s
      labels:
        traefik.enable: "true"
        traefik.docker.network: "traefiknet"
        traefik.http.routers.project-cloakcache-http.entrypoints: "web"
        traefik.http.routers.project-cloakcache-http.rule: "Host(`testcloakcache`) || Host(`testcloakcache.domain.tld`)"
        traefik.http.routers.project-cloakcache-http.service: "project-cloakcache"
        traefik.http.services.project-cloakcache.loadbalancer.server.port: 11222
        traefik.http.services.project-cloakcache.loadbalancer.sticky.cookie: "true"

I also tried setting -Dinfinispan.cluster.stack=kubernetes instead of -Dinfinispan.cluster.stack=tcp, but it did not work.

Thanks in advance!

[pruivo]

JGroups has multiple discovery protocols available: http://jgroups.org/manual5/index.html#DiscoveryProtocols

You will need to update the Infinispan configuration XML file with a custom stack, as documented here: https://infinispan.org/docs/stable/titles/embedding/embedding.html#customizing-jgroups-stacks_cluster-transport

For example, TCPPINP is the simplest one and can be configured as follows

<jgroups>
  <!-- Creates a custom JGroups stack named "my-stack". -->
  <!-- Inherits properties from the default TCP stack. -->
  <stack name="my-stack" extends="tcp">
    <TCPPING initial_hosts="ip1[port],ip2[port]"
             stack.combine="REPLACE"
             stack.position="MPING" />
  </stack>
</jgroups>

The tcp stack uses MPING (multicast ping) for discovery and the example above replaces it with TCPPING. Then configure Infinispan to use that stack: -Dinfinispan.cluster.stack=my-stack

[pruivo]

@jerrac Any update? Did it work?

[jerrac]

Yes, tcpping did work. At least partially. I ended up moving Infinispan off of Swarm and set up a set of docker compose stand alone instances while trying to convince Keycloak to use it. Prior to that I think I had the cluster up on Swarm. Though I think there were some odd issues I hadn't worked through. If I go back to it, I'll share. For now I'll be figuring out the Keycloak specific issues...

Thanks for the help!