Af xdp multi threading #5

zzxgzgz · 2022-12-06T22:35:53Z

This PR does the following:

Added AF_XDP support in Arion Agent.
Refactored db_client.h, in order to use it among different files.
Temporarily using std::unordered_map for endpoint querying.
Ported a couple of utility functions from the arion-dp repository.

Performance:

As of 3a5d5b8, Arion Agent is able to achieve the following performance:

10 traffic flows (10 iperfs), all traffic handled by AF_XDP , running for 200 seconds, 10G NIC card:

Same test, but all traffic handled by in-kernel XDP:

Comparison:

What's next:

Investigate on if is it feasible to add multi-threading packet processing in AF_XDP. If so, implement and test it.
Investigate which DB is the most suitable one for Arion Agent. The std::unordered_map shall be replaced by some DB.

…socket, and able to parse packet until the ARP level

…d tx packets; however, more investigation is needed in order to use the db functions

…for grpc_client

…c client

…successful

…emented endpoint cache layer for better lookup performance; refactored functions in af_xdp_user to utils

…eighborInMemory from pointer to value, in order to make the packet correct after modification

…formance

…s; need to investigate how to add use round robin in kernel, how to make iperf work and how to user more threads

…en after trading from bpool. Made a hack to make iperf work.

… it still stops receiving traffic from time to time

…Completion Queue to have the same size

…DP sockets.

…bpf map fd for security group ebpf table; added sample code to insert data into security group ebpf maps

…need to be tested

…e ebpf map

…security_group_port_bindnig info from arion master, and program into the sg_cidr_map ebpf map

…elarted port bindings and programs the rules

lfu-ps · 2023-01-28T01:23:20Z

include/db_client.h

+
+struct SecurityGroupPortBindingProgrammingState {
+    int version;
+}; // local db table 2 - security rule ebpf programmed version


lfu-ps · 2023-01-28T01:25:48Z

src/CMakeLists.txt

@@ -4,14 +4,16 @@ set(SOURCES
    ./util/dispatch_queue.cpp
    ./util/segment_lock.cpp
    ./comm/grpc_client.cpp
-    )
+        #    db/db_client.cpp
+        comm/af_xdp_user_multi_thread.cpp )


indentation alignment

lfu-ps · 2023-01-28T01:26:28Z

src/main.cpp

@@ -25,6 +25,8 @@
 #include "marl/event.h"
 #include "marl/scheduler.h"
 #include "marl/waitgroup.h"
+#include "af_xdp_user_multi_thread.h"
+#include "grpc_client.h"


there's a //#include "grpc_client.h" above, please keep only 1

lfu-ps · 2023-01-28T01:26:56Z

src/main.cpp

@@ -138,10 +144,16 @@ int main(int argc, char *argv[]) {
    marl::schedule([=] {
        g_grpc_client->RunClient(g_arion_master_address,
                                 g_arion_master_port,
-                                 g_arion_group,
-                                 g_arion_neighbor_table);
+                                 g_arion_group,\


do we need '' ?

lfu-ps · 2023-01-28T01:27:48Z

src/comm/af_xdp_user_multi_thread.cpp

+//
+
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright(c) 2020 - 2022 Intel Corporation. */


Can we add co-author of Futurewei, just like grpc server/client files?

lfu-ps · 2023-01-28T01:31:17Z

src/comm/grpc_client.cpp

+                                // lock transaction section
+                                //     segment lock allows some level of concurrent manipulations of concurrent version map
+                                //     as long as the multi-threading version updates' keys are not hashed to the same slot in segment array
+                                segment_lock.lock(neighbor_key);


you didn't include util\segment_lock.h or .cpp

lfu-ps · 2023-01-28T01:33:35Z

src/comm/grpc_client.cpp

+                                            sg_key.local_ip = local_ip_sock.sin_addr.s_addr;
+                                            sg_key.direction = direction == "egress" ? 0 : 1; // going out is 0 and coming in is 1
+
+                                            if (protocol == "TCP") {


should it be lower case?

lfu-ps · 2023-01-28T01:33:45Z

src/comm/grpc_client.cpp

+
+                                            if (protocol == "TCP") {
+                                                sg_key.protocol = IPPROTO_TCP;
+                                            } else if (protocol == "UDP") {


same as above

lfu-ps · 2023-01-28T01:34:09Z

src/comm/grpc_client.cpp

+                                            }
+
+                                            printf("GPPC: Inserted this sg rule into map: vip: %s, vni: %d\n", vpc_ip.c_str(), vni);
+


lfu-ps · 2023-01-28T01:34:37Z

src/comm/grpc_client.cpp

+                                                                add_programmed_security_group_port_binding_version_db_stmt);
+                                                    });
+                                                }
+


lfu-ps · 2023-01-28T01:34:55Z

src/comm/grpc_client.cpp

+                                                }
+
+                                            }
+                                            if (ebpf_rc < 0) {


line break here

lfu-ps

After we test with DP E2E, and clean up dead code. Let's review again. Before that, you don't need to address.

zzxgzgz added 15 commits October 4, 2022 17:12

Initial commit that includes changes in the libbpf installation

3a5d5b8

Initial commit for af_xdp module

a304675

Updated build script to include af_xdp sample code

2761449

Pulled updates from upstream main

658a4ac

Included glog install in the machine-init.sh

ec99f8f

Added code to parse packet headers all the way to inner IP

4be77b7

Check in code, right now the af_xdp_user is able to get packets from …

cc1babb

…socket, and able to parse packet until the ARP level

With this commit, the ArionAgent is able to receive, parse, modify an…

344ee7a

…d tx packets; however, more investigation is needed in order to use the db functions

Rewrote db_client.h so that it doesn't crash; next is to do the same …

b8943ab

…for grpc_client

Check in updates, right now program crashes near the db insert in grp…

9104656

…c client

This commit enables the happy path for pinging; ARP request/reply is …

acde1bc

…successful

AF_XDP working for ARP and IP

e336e90

Refactored db_client to make local_db accessible by other files; impl…

3e0fc7d

…emented endpoint cache layer for better lookup performance; refactored functions in af_xdp_user to utils

added counter to count processed packets; changed return type of GetN…

c7813b6

…eighborInMemory from pointer to value, in order to make the packet correct after modification

Changed folly::ConcurrentHashMap to std::unordered_map for better per…

491bc41

…formance

zzxgzgz requested review from w-yue, lfu-ps and DavidLiu506 December 6, 2022 22:35

zzxgzgz self-assigned this Dec 6, 2022

zzxgzgz added 11 commits December 21, 2022 15:26

With this commit, AF_XDP multi threading works for ping with 8 thread…

c296811

…s; need to investigate how to add use round robin in kernel, how to make iperf work and how to user more threads

bcache_cons_check has a bug where the first n_buffer_cons is zero, ev…

1f3b4e9

…en after trading from bpool. Made a hack to make iperf work.

With this commit, AF_XDP with 8 sockets works for multipe iperfs, but…

d15fa06

… it still stops receiving traffic from time to time

Fixed the cached drained problem, by setting the UMEM Fill Queue and …

13b3250

…Completion Queue to have the same size

Added logic to save 8 cores for the machine and use the rest for AF_X…

357aabb

…DP sockets.

Added data models for security group and connection tracking; added e…

b757ea1

…bpf map fd for security group ebpf table; added sample code to insert data into security group ebpf maps

Modified the sg_cidr_key_t, added remote_ip field

1653e6d

Added security group rule to the local DB and to the gRPC client.

26fc211

Added support for adding security group port binding to grpc client; …

8d31868

…need to be tested

Moved sync_schema() to the right place; fixed creation of the LPM Tri…

075f5e1

…e ebpf map

With this commit, arion agent is able to get security_group_rule and …

af57a00

…security_group_port_bindnig info from arion master, and program into the sg_cidr_map ebpf map

zzxgzgz added 7 commits January 27, 2023 13:05

Added TODOs and made changes based on feedbacks.

7dda69e

Added logic where when an sg rule is added, arion agent queires the r…

0b10145

…elarted port bindings and programs the rules

Tried to merge updated upstream main code

783a084

Adapted the segment_lock lock for the Neighbor Rules

4f13432

Fixed bug for filling in remote ip into sg cidr key

8991fc4

Changed protocol to lowercase

b39c10f

Adapted seglock logic for security group rule

df115a1

lfu-ps reviewed Jan 28, 2023

View reviewed changes

src/comm/grpc_client.cpp

if (protocol == "TCP") {

sg_key.protocol = IPPROTO_TCP;

} else if (protocol == "UDP") {

Copy link

Collaborator

lfu-ps Jan 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

lfu-ps reviewed Jan 28, 2023

View reviewed changes

src/comm/grpc_client.cpp

}

printf("GPPC: Inserted this sg rule into map: vip: %s, vni: %d\n", vpc_ip.c_str(), vni);

Copy link

Collaborator

lfu-ps Jan 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty line

lfu-ps reviewed Jan 28, 2023

View reviewed changes

src/comm/grpc_client.cpp

add_programmed_security_group_port_binding_version_db_stmt);

});

}

Copy link

Collaborator

lfu-ps Jan 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

empty line

lfu-ps reviewed Jan 28, 2023

View reviewed changes

src/comm/grpc_client.cpp

}

}

if (ebpf_rc < 0) {

Copy link

Collaborator

lfu-ps Jan 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

line break here

Adapted the segment lock logic to the security group port binding

f755f8f

lfu-ps suggested changes Jan 28, 2023

View reviewed changes

Updated sg rule schema

3737b91

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Af xdp multi threading #5

Af xdp multi threading #5

zzxgzgz commented Dec 6, 2022

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps Jan 28, 2023

lfu-ps left a comment

		}

		printf("GPPC: Inserted this sg rule into map: vip: %s, vni: %d\n", vpc_ip.c_str(), vni);

Af xdp multi threading #5

Are you sure you want to change the base?

Af xdp multi threading #5

Conversation

zzxgzgz commented Dec 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lfu-ps left a comment

Choose a reason for hiding this comment