How should rattan in docker mode behave?

[un-lock-able]

How should the docker mode be?

I'm currently tring to reproduce the results from quic-interop, but using rattan instead of ns3 as the network emulator running inside a docker. However, since rattan will create multiple netns and will only listen on the veth itself creates, it's pretty difficult for rattan to listen for any packets that are sent into one of the docker's NICs, apply delay/loss to it and then send it out to another NIC on docker.

Currently, running rattan in isolated mode inside a docker would be like this:

+-------------------------+-----------------------------------+----------------------------+
|                         |            left-ns                |        rattan-ns           |
|                         |                                   |                            |
|+---------------+        |             +-------------+       |      +-------------+       |
||   left-net    |        |             |  vL1-L-XXX  |       |      |  vL1-R-XXX  |       |
||               |<- - - -|- - - - - -->|             |       |      |             |       |
||  193.167.0.2  |        |             |  10.1.1.1   |<------+----->|  10.1.1.2   |       |
||               |        |             |             |       |      |             |       |
|+---------------+        |             +-------------+       |      +-------------+       |
|                         |                                   |            ^               |
|                         |                                   |            |               |
|                         +-----------------------------------+            |rattan forwards|
|                         |            right-ns               |            |               |
|                         |                                   |            v               |
|+---------------+        |             +-------------+       |       +-------------+      |
||  right-net    |        |             |  vR1-R-XXX  |       |       |  vR1-L-XXX  |      |
||               |<- - - -|- - - - - -->|             |       |       |             |      |
|| 193.167.100.2 |        |             |  10.2.1.1   |<------+------>|  10.2.1.2   |      |
||               |        |             |             |       |       |             |      |
|+---------------+        |             +-------------+       |       +-------------+      |
|                         |                                   |                            |
|                         |                                   |                            |
+-------------------------+-----------------------------------+----------------------------+

To make the two dashed lines possible, it would require another two pairs of veth and many route table rules.

I'm wondering if it is possible to let rattan to simply capture on a 'physical' NIC, so that the network would be:

+-----------------------+
|                       |
|+---------------+      |
||   left-net    |      |
||               |      |
||  193.167.0.2  |      |
||               |      |
|+---------------+      |
|      ^                |
|      |                |
|      |rattan forwards |
|      |                |
|      v                |
|+---------------+      |
||  right-net    |      |
||               |      |
|| 193.167.100.2 |      |
||               |      |
|+---------------+      |
|                       |
+-----------------------+

I'm trying to read codes about veth creation and the 'external' cell, but haven't really understand whether this is possible, and more importantly, whether rattan should behave like this in 'docker' mode?

[BobAnkh]

cc @Centaurus99

[minhuw]

Maybe we could take the responsibility of creating veth pairs for containers with a custom network driver plugins.

[Centaurus99]

Capturing on a 'physical' NIC is feasible. This was also the approach taken in previous containerized implementations.

In Docker mode, Rattan could have a new EnvMode instead of continuing with the Isolated Mode. This mode is currently named Container Mode.

Previously, I attempted containerization but didn’t complete it. However, some broken code remains in the repository, like Dockerfile, docker-compose.yml, src/docker.rs, etc. You can refer to these, or start fresh.

What I have in mind is that in Docker mode, we could use Docker Compose to create a topology similar to the following, where Rattan runs in a container in Container Mode and directly captures the NIC created by Docker for it (perhaps refer to get_container_env function).

            ┌──────────┐        ┌───────────┐           
            │ Left Net │        │ Right Net │           
            └───┬──┬───┘        └───┬──┬────┘           
                │  │                │  │                
 ┌────────────┐ │  │ ┌────────────┐ │  │ ┌────────────┐ 
 │    App  ┌──┐ │  │ ┌──┐Rattan┌──┐ │  │ ┌──┐ App     │ 
 │         │  ┼─┘  └─┼  │      │  ┼─┘  └─┼  │         │ 
 │         └──┘      └──┘      └──┘      └──┘         │ 
 └────────────┘      └────────────┘      └────────────┘ 

This approach does require managing the routing tables and ARP tables within each container. For the Rattan container, Rattan itself can handle this. For App containers, I listen for the start event of each App container and run certain commands on start to set up network settings (perhaps refer to src/docker.rs for this).

If we use the custom network driver plugins mentioned by @minhuw, we might not need these setups. However, this would require users to install the Docker plugin we provide.