Test Isolation using linux namespaces

.github/workflows/bazelized_drake_ros.yml

@@ -34,6 +34,13 @@ jobs:
         run: du -hs $(readlink -f ~/.cache/bazel_ci) || true
 
       # Setup.
+      - name: Free up disk space
+        run: |
+          sudo rm -rf \
+          /usr/share/dotnet \
+          /opt/ghc \
+          /usr/local/share/boost \
+          "$AGENT_TOOLSDIRECTORY" || true
       - name: Simplify apt upgrades
         run: .github/simplify_apt_and_upgrades.sh
       - name: Configure drake_ros Bazel for CI

bazel_ros2_rules/WORKSPACE

@@ -15,3 +15,7 @@ http_archive(
 load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
 
 bazel_skylib_workspace()
+
+load("//deps:defs.bzl", "add_bazel_ros2_rules_dependencies")
+
+add_bazel_ros2_rules_dependencies()

bazel_ros2_rules/network_isolation/BUILD.bazel

@@ -0,0 +1,33 @@
+cc_library(
+    name = "network_isolation_cc",
+    srcs = ["network_isolation.cc"],
+    hdrs = ["network_isolation.h"],
+    visibility = ["//visibility:public"],
+)
+
+cc_binary(
+    name = "isolate",
+    srcs = ["isolate.cc"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":network_isolation_cc",
+    ],
+)
+
+# Create a CPython extension
+cc_binary(
+    name = "network_isolation_py.so",
+    srcs = ["network_isolation_py.cc"],
+    linkshared = True,
+    linkstatic = True,
+    deps = [
+        ":network_isolation_cc",
+        "@python_dev//:headers",
+    ],
+)
+
+py_library(
+    name = "network_isolation_py",
+    data = [":network_isolation_py.so"],
+    visibility = ["//visibility:public"],
+)

bazel_ros2_rules/network_isolation/README.md

@@ -0,0 +1,103 @@
+# Introduction
+This directory contains tools and targets to isolate ros2 tests in this repository using linux network namespaces.
+At its core, it uses the ``unshare()`` system call, and is meant to isolate ROS2 traffic. It creates a new user namespace,
+new network namespace to prevent cross talk via the network, and new IPC namespace to prevent cross talk using shared memory.
+
+The existing dload_shim is used to pass on the required argument and isolate the tests.
+
+## Why do we need this ?
+When running ROS2 tests in parallel, they might publish on the same topics and there might be cross talk between tests. RMW config or ROS domain id based isolation is possible,
+but it does not scale well, due to limited ports and domain ids available. Linux namespaces provide a generic and a scalable way to solve this problem.
+
+## Why not isolate individual targets instead of tests ?
+Isolation using the namespace approach requires 3 namespaces, or "credentials" for processes to talk to each other, or be in the same realm : IPC, user and network namespaces.
+Tests are more generic than individual targets, as the tests are free to fork() or run any number of processes they want, and they'll live in the same namespace.
+
+If we do isolate a process 'A', how do we make sure the next process 'B' will live in the same namespace as 'A' (if they're meant to talk to each other) ? There needs to be an API to provide the above mentioned "credentials" to the new process
+somehow, which out of scope of this feature for now.
+
+# Targets
+The logic lives in the following targets, which are meant to be used with the bazel rules ``ros_cc_test()`` and ``ros_py_test()`` :
+* ``network_isolation_cc`` (cc_library): This is where the core logic lives, and the ``unshare()`` call is run.
+* ``network_isolation_py.so``(cpython extension) : Python binding for the network isolation logic.
+* ``network_isolation_py`` (py_library) : Importable python module for the network isolation logic.
+
+Other than these, there is a standalone executable target called ``isolate`` which is meant to be used in a standalone way, and not with the
+``ros_*_test()`` rules. It isolates the process in the first argument provided to it. This uses the same ``unshare()`` logic as ``network_isolation_cc``.
+
+# How do we use this feature ?
+There are 3 ways to use this feature :
+
+## Using ``ros_cc_test()`` rule :
+There is now an extra argument (``network_isolation``) available to the rule, so for e.g. we can modify a test in ``ros2_example_bazel_installed/ros2_example_apps/BUILD.bazel`` as : 
+
+```
+ros_cc_test(
+    name = "talker_listener_cc_test",
+    size = "small",
+    srcs = ["test/talker_listener.cc"],
+    rmw_implementation = "rmw_cyclonedds_cpp",
+    network_isolation = True,
+    deps = [
+        ":listener_cc",
+        ":talker_cc",
+        "@ros2//:rclcpp_cc",
+        "@ros2//:std_msgs_cc",
+        "@ros2//resources/rmw_isolation:rmw_isolation_cc",
+    ],
+)
+```
+
+Whatever processes are spawned by this test will be contained in a namespace, and will be able to talk to each other, but not to any other ros nodes running outside of this test. 
+
+## Using the ``ros_py_test()`` rule :
+Similarly for the python test rule, we can add ``network_isolation`` to ``True``. Consider this section in ``drake_ros_examples/examples/iiwa_manipulator/BUILD.bazel`` :
+
+```
+ros_py_test(
+    name = "iiwa_manipulator_test",
+    network_isolation = True,
+    srcs = ["test/iiwa_manipulator_test.py"],
+    data = [
+        ":iiwa_manipulator",
+        ":iiwa_manipulator_py",
+    ],
+    main = "test/iiwa_manipulator_test.py",
+    deps = [
+        "@ros2//resources/bazel_ros_env:bazel_ros_env_py",
+    ],
+)
+```
+
+Similarly, the ros nodes spawned in this test can only talk to each other, and not other nodes running on the system at the same time, even if you have multiple instances of this test running in different terminals.
+
+
+## Using the ``isolate`` target:
+As mentioned before, the ``isolate`` target is meant to be used in a generic way and will isolate **any** process provided to it provided it is available in the bazel sandbox, for e.g. : 
+```
+cd bazel_ros2_rules
+bazel run //network_isolation:isolate -- /bin/bash -c "echo HELLO_WORLD"
+```
+
+Here, the bash command for printing "HELLO_WORLD" is isolated, and can be replaced with any ros2 command. For instance, if you have ros2 humble installed
+outside of drake-ros, using debs, you could run a publisher and subscriber that would be isolated from each other : 
+
+Lets start the talker from the demo nodes package:
+```
+bazel run //network_isolation:isolate -- /bin/bash -c "source /opt/ros/humble/setup.bash && export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp && ros2 run demo_nodes_cpp talker"
+```
+
+In another terminal, run :
+```
+bazel run //network_isolation:isolate -- /bin/bash -c "source /opt/ros/humble/setup.bash && export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp && ros2 run demo_nodes_cpp listener"
+```
+
+These 2 processes should not be able to talk to each other, when used with ``isolate``.
+
+# Testing this feature in CI
+Among the 3 ways to use this feature, as far as CI is concerned, there is a test for the ``isolate`` target mechanism.
+
+The test target is called ``network_isolation_test`` and uses ``ros2_example_bazel_installed/test/network_isolation_test.py`` to run 5 (by default) talker-listener pairs which have an id attached to them.
+They all publish and listen on the same topic at the same time, and expect no cross talk between the pairs. The number of pairs can be changed using the ``--id`` cmdline argument if needed.
+
+The other 2 ways to use this feature, using ``ros_*_test()`` rules, run on tests, and not on executable targets, so *writing a test for a test* seems like an anti-pattern. The ``isolate`` target uses the same logic, so should be sufficient for testing.

bazel_ros2_rules/network_isolation/isolate.cc

@@ -0,0 +1,35 @@
+#include <unistd.h>
+
+#include <iostream>
+#include <vector>
+
+#include "network_isolation.h"
+
+void die(const char * message) {
+    std::cerr << "isolate: " << message << ".\n";
+    exit(-1);
+}
+
+int main(int argc, char ** argv) {
+    if (argc < 2) {
+        die("shim must be given a command to execute");
+    }
+
+    if (!network_isolation::create_linux_namespaces()) {
+        die("Failed to fully create isolated environment");
+    }
+
+    // Copy to a new array that terminates with a null pointer at the end.
+    std::vector<char *> new_argv;
+    for (int i = 1; i < argc; ++i) {
+        new_argv.push_back(argv[i]);
+    }
+    new_argv.push_back(nullptr);
+
+    // Exec a new process - should never return!
+    execv(new_argv.at(0), &new_argv.at(0));
+
+    perror("execv");
+    die("Call to execv failed");
+    return -1;
+}

bazel_ros2_rules/network_isolation/network_isolation.cc

@@ -0,0 +1,109 @@
+#include "network_isolation.h"
+
+#include <iostream>
+
+#include <ifaddrs.h>
+#include <sched.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <arpa/inet.h>
+#include <net/if.h>
+#include <net/route.h>
+#include <sys/ioctl.h>
+#include <sys/types.h>
+
+namespace network_isolation {
+
+void error(const char * message)
+{
+    std::cerr << "create_linux_namesapces: "
+        << message << ":" << strerror(errno) << "\n";
+}
+
+bool create_linux_namespaces()
+{
+    int result = unshare(CLONE_NEWUSER | CLONE_NEWNET | CLONE_NEWIPC);
+
+    if (result != 0) {
+        error("failed to call unshare");
+        return false;
+    }
+
+    // Assert there is exactly one network interface
+    struct ifaddrs *ifaddr;
+
+    if (-1 == getifaddrs(&ifaddr)) {
+        error("could not get network interfaces");
+        return false;
+    }
+    if (nullptr == ifaddr) {
+        error("there are no network interfaces");
+        return false;
+    }
+    if (nullptr != ifaddr->ifa_next) {
+        error("there are multiple network interfaces");
+        return false;
+    }
+
+    // Need a socket to do ioctl stuff on
+    int fd = socket(AF_INET, SOCK_DGRAM, 0);
+    if( fd < 0 ){
+        error("could not open a socket");
+        freeifaddrs(ifaddr);
+        return false;
+    }
+
+    struct ifreq ioctl_request;
+
+    // Check what flags are set on the interface
+    strncpy(ioctl_request.ifr_name, ifaddr->ifa_name, IFNAMSIZ);
+    int err = ioctl(fd, SIOCGIFFLAGS, &ioctl_request);
+    if (0 != err) {
+        freeifaddrs(ifaddr);
+        error("failed to get interface flags");
+        return false;
+    }
+
+    // Expecting a loopback interface.
+    if (!(ioctl_request.ifr_flags & IFF_LOOPBACK)) {
+        error("the only interface is not a loopback interface");
+        freeifaddrs(ifaddr);
+        return false;
+    }
+
+    // Enable multicast
+    ioctl_request.ifr_flags |= IFF_MULTICAST;
+    // Bring up interface
+    ioctl_request.ifr_flags |= IFF_UP;
+
+    err = ioctl(fd, SIOCSIFFLAGS, &ioctl_request);
+    if (0 != err) {
+        error("failed to set interface flags");
+        freeifaddrs(ifaddr);
+        return false;
+    }
+
+    // For programs that use both LCM and ROS, we need an LCM route ala
+    //  sudo route add -net 224.0.0.0 netmask 240.0.0.0 dev lo
+    struct rtentry route = {};
+    auto* dest = reinterpret_cast<struct sockaddr_in*>(&route.rt_dst);
+    dest->sin_family = AF_INET;
+    dest->sin_addr.s_addr = inet_addr("224.0.0.0");
+    auto* mask = reinterpret_cast<struct sockaddr_in*>(&route.rt_genmask);
+    mask->sin_family = AF_INET;
+    mask->sin_addr.s_addr = inet_addr("240.0.0.0");
+    std::string device{"lo"};
+    route.rt_dev = device.data();
+    route.rt_flags = RTF_UP;
+    err = ioctl(fd, SIOCADDRT, &route);
+    if (0 != err) {
+        error("failed to set route");
+        freeifaddrs(ifaddr);
+        return false;
+    }
+
+    freeifaddrs(ifaddr);
+    return true;
+}
+}  // namespace network_isolation

bazel_ros2_rules/network_isolation/network_isolation.h

@@ -0,0 +1,22 @@
+#pragma once
+
+namespace network_isolation {
+/// Creates linux namespaces suitable for isolating ROS 2 traffic.
+///
+/// The new namespaces are:
+/// * A new user namespace to avoid needing CAP_SYS_ADMIN to create
+///   network and IPC namespaces
+/// * A new network namespace to prevent cross-talk via the network
+/// * A new IPC namespaces to prevent cross-talk via shared memory
+///
+/// It also configures network namespace to enable ROS 2 traffic.
+/// At the end of a successful call the current process will be in
+/// the created namespaces.
+/// Depending on what part of the process fails, an unsuccessful
+/// call may also leave the current process in new namespaces.
+/// There is no way to undo this.
+///
+/// \return true iff the namespaces were created successfully.
+bool create_linux_namespaces();
+
+}  // namespace network_isolation

bazel_ros2_rules/network_isolation/network_isolation_py.cc

@@ -0,0 +1,12 @@
+#include <pybind11/pybind11.h>
+
+#include "network_isolation/network_isolation.h"
+
+namespace py = pybind11;
+
+PYBIND11_MODULE(network_isolation_py, m)
+{
+  m.def("create_linux_namespaces", &network_isolation::create_linux_namespaces, R"pbdoc(
+        Creates linux namespaces suitable for isolating ROS 2 traffic.
+    )pbdoc");
+}

bazel_ros2_rules/ros2/ros_cc.bzl

@@ -166,6 +166,7 @@ def ros_cc_test(
         cc_binary_rule = native.cc_binary,
         cc_library_rule = native.cc_library,
         cc_test_rule = native.cc_test,
+        network_isolation = False,
         **kwargs):
     """
     Builds a C/C++ test and wraps it with a shim that will inject the minimal
@@ -211,6 +212,7 @@ def ros_cc_test(
         name = shim_name,
         target = ":" + noshim_name,
         env_changes = shim_env_changes,
+        network_isolation = network_isolation,
         **shim_kwargs
     )
 
@@ -220,4 +222,8 @@ def ros_cc_test(
         deps = ["@bazel_ros2_rules//ros2:dload_shim_cc"],
         tags = ["nolint"] + kwargs.get("tags", []),
     )
+    if network_isolation:
+        kwargs["deps"].append(
+            "@bazel_ros2_rules//network_isolation:network_isolation_cc",
+        )
     cc_test_rule(name = name, **kwargs)

bazel_ros2_rules/ros2/ros_py.bzl

@@ -135,6 +135,7 @@ def ros_py_test(
         rmw_implementation = None,
         py_binary_rule = native.py_binary,
         py_test_rule = native.py_test,
+        network_isolation = False,
         **kwargs):
     """
     Builds a Python test and wraps it with a shim that will inject the minimal
@@ -176,6 +177,7 @@ def ros_py_test(
         name = shim_name,
         target = ":" + noshim_name,
         env_changes = shim_env_changes,
+        network_isolation = network_isolation,
         **shim_kwargs
     )
 
@@ -186,4 +188,8 @@ def ros_py_test(
         deps = ["@bazel_ros2_rules//ros2:dload_shim_py"],
         tags = ["nolint"] + kwargs.get("tags", []),
     )
+    if network_isolation:
+        kwargs["deps"].append(
+            "@bazel_ros2_rules//network_isolation:network_isolation_py",
+        )
     py_test_rule(name = name, **kwargs)

bazel_ros2_rules/ros2/tools/dload.bzl

@@ -59,6 +59,7 @@ def get_dload_shim_attributes():
             cfg = "target",
         ),
         "env_changes": attr.string_list_dict(),
+        "network_isolation": attr.bool(default = False),
     }
 
 def _workaround_issue311(ament_prefixes, env_changes):