Discovery across nodes behind docker

[oxlade39]

Hi there. I have a bit of a custom setup that I'm trying to get working. I have several hosts across a network that have trino running.

1 coordinator, 6 workers.
Some of the workers will be on a separate physical host.

The workers that don't reside on the same physical host as the coordinator register with the coordinator via the discovery endpoint (they are visible in the nodes list), however, when work is sent to them they aren't able to respond and queries hang.

I think this is almost certainly because the workers are attempting to communicate over their internal docker ip.

Is there a way to configure which interface and port the workers publish?

Here's an example of what I see in the logs of one of the workers that doesn't appear to be able to accept work:

2023-08-08T12:23:34.888Z WARN http-client-node-manager-scheduler-1 io.trino.metadata.RemoteNodeState Error fetching node state from http://172.17.0.7:43000/v1/info/state: Failed communicating with server: http://172.17.0.7:43000/v1/info/state

172.17.0.7 is the ip of the docker container but not the host, so that address will not be publicly available.

[nineinchnick]

There are a few settings related to this, like node.bind-ip, check out the sources here: https://github.com/airlift/airlift/blob/master/node/src/main/java/io/airlift/node/NodeConfig.java
Looks like they're not documented in the Trino docs.

[nineinchnick]

Have you tried setting different node.bind-ip and node.internal-address (or node.external-address)? Or maybe I don't understand the question, are you also talking about how to setup Docker networking?

[oxlade39]

From what I can tell the node.external-address will mean the ip resolves correctly but I also need to be able to change the port to differentiate between the internal port and the host port.

To add some context:

Each of the workers is running in docker behind a load balancer, the workers may or may not be on the same host and so won't necessarily be running in the same docker stack. Each worker will have its external port dynamically allocated by the load balancer but the internal port will be fixed (43000 as set by http-server.http.port).
The load balancer can route traffic to each of the workers (I can use host header routing and give each of the workers a unique host address worker1.trino.mycorp.com, worker2.trino.mycorp.com, etc) but I would need to then change the port to 443 for the https load balancer.

[cahya-wirawan]

Hi, we are having the same issue. We use 6 workers in 3 different docker hosts and 1 coordinator in another docker host. Since they are in different network, they can't communicate using the internal IP addresses.
I am new with Trino, so I am wondering where I can try to configure these node.bind-ip , node.internal-address and node.external-address. Thanks

[dungruoc]

I am having the same issue, when separating coordinator docker container on 1 host, and the workers in docker containers in other hosts.
It seems that the discovery server of coordinator returns its container's IP to the workers and these use it for the status update via API v1/info/state. Very interesting that if the worker container has the same IP as the coordinator's one, it can work (it seems that the worker sends to its owner http server on v1/info/state then no error). Containers with a different IP from coordinator IP fails on sending v1/info/state

trino> select * from system.runtime.nodes;
node_id | http_uri | node_version | coordinator | state
------------------------+------------------------+--------------+-------------+--------
trino-worker1 | http://172.18.0.2:8080 | 470 | false | active
trino-coordinator | http://172.18.0.2:8080 | 470 | true | active
trino-worker2 | http://172.18.0.2:8080 | 470 | false | active
(3 rows)

Could trino has some fix so that the worker can use the discovery.uri to send status to the coordinator?

I tried also to put into node.properties of coordinator:
node.external-address={same as discovery.uri}
but it does not work.

[dungruoc]

Finally, I think that the current trino cannot be deployed with segmented networks, where the coordinator and workers are in different networks and connect via NAT. They need to have 2-way communication via HTTP(s), at least at endpoint v1/info/state to all nodes. And these are not configurable with hosting_IP:port but automatically at discovery registering.

Maybe docker swarm can be good to deploy, with an overlay network where all containers can communicate peer-2-peer.