Add RDMA collector

[yeahdongcn]

This PR collects RDMA stats using rdmamap.

Testing Done

1. docker run --pull=always --cap-add=SYS_TIME --net=host -it node-exporter-linux-amd64:master --collector.rdma --web.listen-address=:8864 --collector.disable-defaults
2. curl localhost:8864/metrics

The results can be viewed below:

metrics.log

# HELP node_rdma_mlx5_local_ack_timeout_err_total Total number of times the QP's ACK timer expired for RC, XRC, or DCT QPs at the sender side (retry limit not exceeded)
# TYPE node_rdma_mlx5_local_ack_timeout_err_total gauge
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_0",interfaces="ens2np0",port="1"} 960
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_1",interfaces="ens3np0",port="1"} 576
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_10",interfaces="ens3v0",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_11",interfaces="ens3v1",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_12",interfaces="ens3v2",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_13",interfaces="ens3v3",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_2",interfaces="ens4f0np0",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_3",interfaces="ens4f1np1",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_4",interfaces="ens5f0np0",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_5",interfaces="ens5f1np1",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_6",interfaces="ens2v0",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_7",interfaces="ens2v1",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_8",interfaces="ens2v2",port="1"} 0
node_rdma_mlx5_local_ack_timeout_err_total{device="mlx5_9",interfaces="ens2v3",port="1"} 0

Additionally, the above tests were performed on a node without RDMA devices. The node-exporter runs without issues, as shown in the following logs:

time=2024-11-08T11:18:18.441Z level=ERROR source=collector.go:168 msg="collector failed" name=rdma duration_seconds=4.3574e-05 err="no rdma devices found"

[fishingfly]

This looks super useful for GPU clusters.

[SuperQ]

I took a look at the new data reading library and I'm not very happy with it.

The main issue is this isn't setup to respect the sysPath and rootfsPath variables in paths.go. This means this collector will fail when running in a container when the host sysfs is remapped to a different path.

[yeahdongcn]

I took a look at the new data reading library and I'm not very happy with it.

The main issue is this isn't setup to respect the sysPath and rootfsPath variables in paths.go. This means this collector will fail when running in a container when the host sysfs is remapped to a different path.

@SuperQ Thank you for the review! I've updated the code as per your comments.

I'll also open an issue in rdmamap to see if the Dir paths can be moved to package variables for easier configuration (currently replaced with my fork).

[yeahdongcn]

My PR has been merged in rdmamap. The latest commit updates the rdmamap go module.

[discordianfish]

Still doesn't seem to support using different dirs for the files its reading

[yeahdongcn]

Still doesn't seem to support using different dirs for the files its reading

Please see lines 122-139 at collector/rdma_linux.go.

[discordianfish]

Still doesn't seem to support using different dirs for the files its reading

Please see lines 122-139 at collector/rdma_linux.go.

Ah I missed this, ok this looks better. Not a fan of that libs using global vars for this but good enough IMO. But the names need change, see above

[yeahdongcn]

Still doesn't seem to support using different dirs for the files its reading

Please see lines 122-139 at collector/rdma_linux.go.

Ah I missed this, ok this looks better. Not a fan of that libs using global vars for this but good enough IMO. But the names need change, see above

No problem! I’ll update the code accordingly and request your review once it's ready. Thanks!