Network and H2O cloud specification when creating tests

Update: I imply below that I was going to be able to create an h2o cloud across different private subnets, as long as there were public IPs that the nodes could communicate with. I now believe that's not possible. All h2o nodes have to be part of the same private subnet. See "Must be same private subnet" at bottom

Overview

There are special command line parameter, config json parameters and build_cloud()/build_cloud_with_hosts() params, that allow a test writer to test different network and cloud configurations and different ways to start the h2o jars on the target nodes.

Here's my interpretation of what h2o needs to be told, to work in various network topologies, and what needs to be tested, to allow the various relaxed specifications to work as per intent.

I start with what works in the most cases, and then introduce relaxations where you can not specify things, rather than starting relaxed and adding requirements when the relaxations don't work (and you don't know why)

I had the insight today that all the hadoop vendors deal with node identification one way (reverse dns has to work) but h2o doesn't say that.

So a network engineer might ask a h2o engineer "How do I make h2o work, if I know my network is perfect?" And "How does h2o define a perfect network?"

First is realizing that depending on your network topology, h2o nodes may not cloud up or some may cloud up and others interfere with that cloud silently. i.e. h2o doesn't help you to create a cloud that works. H2O has requirements and you have to meet these requirements.

Some are network requirements, some are requirements in files (flatfiles) and parameters (-ip, -port, -network, -name)

You shouldn't rely on h2o stdout logs for steering you to the right setup for h2o

Hadoop clusters

This information doesn't cover h2o on hadoop clusters. The parameters discussed here are on the java -jar h2o.jar command line, and the user doesn't have direct control over them when dispatching h2o on hadoop. The h2o on hadoop driver has it's own set of params. H2O still has to self-discover it's IP address when runnning on hadoop. It's expected that the self-discovery algortihm is more likely correct in the constrained environment of a single hadoop cluster.

Rules to understand

A) Every h2o node needs to know the IP address that all nodes in the cloud will use to communicate with it. This needs to be one address and h2o needs to self-identify with that IP address. If there are multiple IP addresses that can be used to point to a single node, and you put those in a flatfile, h2o will incorrectly cloud up with all those multiple IPs. So "one true IP" is an important thing. That driver also needs to communicate back to the machine that is running the h2odriver jar, which might cause issues (see -driverif param there for possible help)

1. The starting point is that if you use java -jar -ip <globally visible ip> start h2o, things will always work, if the ip meets the A) requirement and the network passes tcp and udp traffic between all node pairs.

2. H2O needs to be able to do mostly-reliable tcp and udp from/to all nodes you want to use, with that list of IPs from 1)

The way to prove this is by doing some iperf -s /iperf -c and iperf -s -u/iperf -u -c to show that sustained tcp and udp traffic is possible. Don't rely on h2o udp drop tests or network tests, because if you can't create a cloud correctly, you can't run those tests.

3. You tell H2O what nodes to communicate with, using -flatfile This file should have the IPs in 1), along with a :port suffix, where port is the lower of two sequential port numbers reserved for h2o. (and is the port used for REST/browser communication. Default 54321 (and implied 54322. Make sure the port you use, and the implied port, are open for all TCP and UDP traffic. Some software like Openstack may require UDP to be opened explicitly.

4. To ease use in laptop or dynamic environments, h2o has a number of mechanisms where you don't need to do 1,2,3. For instance it can self-discover the IP to use for a node in many cases, even with multiple NICs. Bridged environments with differing private and public IPs (say with subnets) can be problematic. It can also self-discover an available port that the OS says is available. However, if you have bridges or firewalls, unless you have all TCP and UDP ports opened between devices, it's recommended that you specify ports, so you can say what needs to be opened thru bridges and firewalls

5. If UDP multicast is supported in your environment, h2o will discover other nodes using it. So you don't need a flatfile, if the UDP multicast reaches to all nodes you care about. TTL is set to 2 for the multicast.

6. -name <cloudname> is used to restrict which h2o nodes cloud up with each other. Every node should be started with the same cloud name. Once you start having multiple clouds, relying on a default cloud name from h2o probably isn't desired

7. If you don't want to tell h2o what ip it is recognized by globally (a public ip), for instance if your public ip is not static, there is a -network <ipv4 network shorthand> param that can help in some cases. (I don't know which yet? if there's a route thru a bridge, does this get the right public ip if you identify the bridge with this?)

   -network=[, ...] The IP address discovery code will bind to the first interface that matches one of the networks in the comma-separated list. Use instead of -ip when a broad range of addresses is legal. (Example network specification: '10.1.2.0/24' allows 256 legal possibilities.)

Update: I tried -network=172.16.0.0/16 in a bridged environment (from 10.0.0.0/16 to 172.16.0.0/16) and it didn't seem to help, because the interface list h2o could see, didn't include the required 172.16.0.0/16 network..i.e. I got this h2o stdout. So I have to specify -ip

00:57:38.097 main      INFO WATER:     Considering fe80:0:0:0:f816:3eff:fe1c:b550%2 ...
00:57:38.097 main      INFO WATER:     Considering 10.0.0.53 ...
00:57:38.097 main      INFO WATER:     Considering 0:0:0:0:0:0:0:1%1 ...
00:57:38.097 main      INFO WATER:     Considering 127.0.0.1 ...
00:57:38.097 main      ERRR WATER: No interface matches the network list from the -network option.      Exiting.

8. If your cloud has nodes that cloud up with a mix of different subnet ip addresses (like 10.x.x.x with 172.x.x.x) it's unlikely it will cloud because those subnets typically won't route all traffic through bridges that probably exist. You have to use some set of the above 1-7 to get everyone to self-identify with a correct IP that's visible to all nodes

9. Again, don't rely on h2o stdout to tell you what's wrong. Use external tools like iperf, iperf -u and ping. Once you get a cloud built, you can use h2o's udp drop and network test, to test that your network doesn't drop packets or have low performance with h2o's heavy network demands.

10. Although we don't know the actual bounds, if you have UDP packet drop in the 5% or greater range in your system, due to loading or buffer sizing, it's likely you will have problems with h2o. H2O is tolerant of some amount of loss in TCP and UDP, but large loss will make it fail.

11. An additional problem is that users need to be able to http browse to the cloud that forms. Ideally this is one of the same ip addresses used above, although it can be different or proxied. The browser (and R) can interact with h2o with a different routed IP, and it doesn't affect cloud formation or other behaviors. (the REST api should work regardless of how it gets to the cloud) I've updated my wiki to say "H2O can't be made to work with nodes in different private subnets" If a customer asked me, that's what I would tell him.

H2O seems to require all nodes to be part of the same private subnet

Here's how it looks to me. I could be wrong, but I tried a lot of things (including changing some H2O.java)

re NAT:

well that's what I was doing, I think. The openstack stuff has virtual routers, which do the NAT

So I think h2o is just capable of clouding across private subnets. Here's why.

Normally, you think if you can ping and scp between hosts, you can then get h2o to cloud up. I can ping and scp between the hosts I want, but h2o won't cloud. And it's not a UDP issue: i can iperf -u and it looks great (UDP bw between nodes)

And I use a flatfile. So it's not multicast.

It's inherent in how h2o identifies nodes.

The local address is used to identify a node. It's also used to create sockets. So it has to be correct from the point of view of "this" node.

BUT: we send heartbeats around that have every node's self-identification. This is that same lcocal ip address.

KEY: we don't use hostnames

So when the information about a potential cloud gets circulated to everyone, it's got local ip addresses in it. And even though I could talk to you with the public ip address, you can't use the public ip address to self-identify, because you have to create sockets off of it. And I can't talk to you with the private ip address you send around to identify yourself.

I think it's inherent to us using ip addresses to identify nodes, rather than names. If names were used it could work. It could also work if the heartbeat/cloud forming packets had the concept of public and private ip's and only circulated public ips.