Status Update: 2023-01-04

[haixuanTao]

Happy new year! 🎉

[haixuanTao]

dora

• Detected Pyo3 memory leak issue in dora operator API. This was due to pyo3 holding in memory variables that were no longer needed due to its memory model. Counterintuitively, pyo3 variables are not bounded by their Rust lifetime and live for the whole lifetime of the runtime.
  • There is a couple of temporary fixes that we can use. I am going to investigate the most reliable one.
  • For more info. See: Force removal of Pyo3 Object to avoid memory leak #168, Memory leak on pyo3 object created within Rust PyO3/pyo3#2853 and Unexpected memory management behavior PyO3/pyo3#1056

[haixuanTao]

dora-drives

• Added Vulkan drivers in the docker container that enables speeding up CARLA simulation rendering using GPU.
• Fixes many small bugs ( initialization, goal waypoints updating, improving the driving performance ... ) to make the user experience smoother.
• Posted a docker container self-sufficient for the competition on dockerhub and sent it to Lei, the online competition software engineer. See: https://hub.docker.com/repository/docker/haixuantao/dora-drives
• Added lanes detection with lane 3D position estimation using LIDAR.
  • The 3D position of lanes is inferred from the nearest neighbours of the point cloud generated by LIDAR.

Latest run

Latest run sped up by 20.

Remark

This run exhausted all my RAM due to the pyo3 memory leak issue.
This run was executed within the docker container.

[phil-opp]

dora-daemon

We made good progress on dora-daemon:

• dora-daemon is now working with Rust and Python nodes
  • CLI and examples are working as well
  • Tested on Linux, Windows, and macOS
• we implemented periodic watchdog messages to detect disconnects between daemon and coordinator
• we created a basic benchmark example for evaluating throughput and latency

New Node API

The new daemon design required a few changes to the node API. One notable change is that we're now passing an event stream instead of an input stream. This allows us to also signal other types of events to nodes, for example a closed input or a manual stop commands. This enables nodes to handle these events in a special way, for example by exiting early once a specific input is closed.

The new APIs look like this:

Rust

let (mut node, mut events) = DoraNode::init_from_env()?;

while let Some(event) = events.recv() {
    match event {
        Event::Input {
            id,
            metadata: _,
            data,
        } => match id.as_str() {
            "message" => {
                let data = data.wrap_err("no data")?;

                println!("received {} bytes", data.len());

                node.send_output("foo".to_owned().into(), Default::default(), 8, |data| {
                    data.copy_from_slice(&data.len().to_le_bytes())
                })?;
            }
            other => eprintln!("Ignoring unexpected input `{other}`"),
        },
        Event::Stop => {
            println!("Received manual stop");
        }
        Event::InputClosed { id } => {
            println!("Input `{id}` was closed");
        }
        other => eprintln!("Received unexpected input: {other:?}"),
    }
}

Python

node = Node()

for event in node:
    match event["type"]:
        case "INPUT":
            match event["id"]:
                case "image":
                    print("received image input")
                    frame = np.frombuffer(event["data"], dtype="uint8")
                    frame = cv2.imdecode(frame, -1)
                    frame = frame[:, :, ::-1]  # OpenCV image (BGR to RGB)
                    results = model(frame)  # includes NMS
                    arrays = np.array(results.xyxy[0].cpu()).tobytes()

                    node.send_output("bbox", arrays, event["metadata"])
                case other:
                    print("ignoring unexpected input:", other)
        # handling of these events is optional
        case "STOP":
            print("received stop")
        case other:
            print("received unexpected event:", other)

The above example uses pattern matching, which is new in Python 3.10. Older Python versions need to use if blocks instead, e.g. if event["type"] == "INPUT".

Benchmark Results

As mentioned above, we created a basic benchmark example that sends messages of different sizes and records the average latency and throughput. Note that this benchmark is highly unscientific, so the results should only be considered as a rough estimate.

For each message size, the example sends 100 messages and then averages the recorded time.

Throughput in time-per-message, i.e. the delay between two received messages:

message size (in bytes)	zenoh	iceoryx	dora-daemon
0x8	29.851µs	33.241µs	70.876µs
0x40	1.893µs	2.956µs	86.175µs
0x0	1.875µs	3.258µs	75.688µs
0x200	1.772µs	3.353µs	72.918µs
0x800	8.646µs	2.999µs	76.322µs
0x1000	16.242µs	9.698µs	74.671µs
0x4000	55.334µs	51.398µs	102.847µs
0xa000	156.539µs	145.971µs	171.931µs
0x64000	1.310325ms	1.215636ms	1.075954ms
0x3e8000	13.585679ms	12.109094ms	10.563545ms
0x2710000	143.784761ms	138.420436ms	103.560525ms

Average latency, i.e. the delay between sending a message and receiving it:

message size (in bytes)	zenoh	iceoryx	dora-daemon
0x8	1.368918ms	1.338828ms	71.394µs
0x40	1.423934ms	1.426594ms	91.465µs
0x0	1.373609ms	1.48646ms	83.914µs
0x200	1.03314ms	1.371293ms	80.042µs
0x800	250.318µs	823.608µs	80.119µs
0x1000	46.813µs	114.046µs	75.796µs
0x4000	37.409µs	37.95µs	78.851µs
0xa000	59.367µs	53.923µs	93.373µs
0x64000	331.771µs	259.225µs	211.636µs
0x3e8000	4.860249ms	3.503654ms	1.870011ms
0x2710000	99.443405ms	90.696233ms	17.616989ms

Planned performance improvements:

• Reuse shared memory regions instead of reallocating them on each message
• Avoid TCP overhead by communicating directly through shared memory
  • Right now the shared memory is only used for passing data, the control requests currently use a TCP socket
  • Create a shared memory channel that passes messages through shared memory
  • Use condition variables placed in shared memory for signaling (to avoid polling)

[phil-opp]

We found some issues in the zenoh/iceoryx benchmark examples. See #170 (comment) for updated results.