Skip to content

Commit 3f9fa92

Browse files
AnonymousAnonymous
Anonymous
authored and
Anonymous
committedOct 14, 2023
ONCache added
0 parents  commit 3f9fa92

36 files changed

+41904
-0
lines changed
 

‎.gitignore

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
dev/
2+
*
3+
!*/
4+
!*.c
5+
!*.cpp
6+
!*.h
7+
!*.py
8+
!*.md
9+
!Makefile
10+
!common.mk
11+
!.git*
12+
!*.sh
13+
!*.deb

‎.gitmodules

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[submodule "libbpf"]
2+
path = libbpf
3+
url = https://github.com/libbpf/libbpf/
4+
[submodule "yaml-cpp"]
5+
path = yaml-cpp
6+
url = https://github.com/jbeder/yaml-cpp.git

‎Makefile

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
PARTS = user_prog tc_prog
2+
PARTS_CLEAN = $(addsuffix _clean,$(PARTS))
3+
4+
COMMON_DIR ?= ./common/
5+
LIBBPF_DIR ?= ./libbpf/src/
6+
YAML_DIR ?= ./yaml-cpp/
7+
include $(COMMON_DIR)/common.mk
8+
9+
$(OBJECT_LIBBPF):
10+
cd $(LIBBPF_DIR) && $(MAKE) all OBJDIR=.; \
11+
mkdir -p build; $(MAKE) install_headers DESTDIR=build OBJDIR=.; \
12+
13+
$(OBJECT_YAML):
14+
cd $(YAML_DIR) && mkdir -p build && cd build; \
15+
cmake .. && $(MAKE); \
16+
17+
$(PARTS):
18+
$(MAKE) -C $@ all
19+
20+
$(PARTS_CLEAN):
21+
$(MAKE) -C $(subst _clean,,$@) clean
22+
23+
.PHONY: all clean $(PARTS) $(PARTS_CLEAN)
24+
all: $(OBJECT_LIBBPF) $(OBJECT_YAML) $(PARTS)
25+
core: $(PARTS)
26+
clean: $(PARTS_CLEAN)

‎README.md

+184
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,184 @@
1+
# ONCache
2+
## In this repo
3+
This repo includes all the code and scripts that are required to run ONCache.
4+
5+
`common` folder includes the frequently used definitions in the C++ code and Makefiles.
6+
7+
`headers` folder includes the header files.
8+
9+
`rpeer_kernel_patch` folder is the kernel source code that are modified to support `bpf_redirect_rpeer`. The modification is based on Linux kernel 5.14. The main modification is in `filter.c`. You can search `rpeer` to see the detail.
10+
11+
`scripts` includes the scripts to provision a simple kubernetes cluster and to run a iperf3/netperf test.
12+
13+
`tc_prog` and `user_prog` include the source code of the eBPF programs and the user space programs of ONCache.
14+
15+
`libbpf` and `yaml-cpp` are the submodules included in the repo, and are used in compiling.
16+
## Tutorial to try ONCache
17+
Before the start of the tutorial, you should prepare two hosts (e.g. VM or cloud server), with Linux kernel verion>=v5.13 . Our tutorial has been tested on Ubuntu 20.04, with Linux kernel v5.14.
18+
19+
### Step0: Clone this repo on all the hosts
20+
The repo includes libbpf and yaml-cpp as submodules, and should be cloned at the same time.
21+
```
22+
git clone --recurse-submodules https://github.com/nothepeople/ONCache.git ~/ONCache
23+
```
24+
25+
### Step1: Provision a container cluster
26+
The Kubernetes is the most common container orchestrater. We take Kubernetes as an example in this tutorial. We have prepared a script that helps to provision a simple Kubernetes cluster with two nodes. You should first install docker, kubeadm, kubelet, and kubectl on all of your hosts. You can reference to these pages:
27+
28+
> [Install docker using the script](https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script) \
29+
[Installing kubeadm, kubelet and kubectl](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl)
30+
31+
Then, run the script **only on the master node** to provision a two-node kubernetes cluster with Antrea as the CNI. Note that some environment varibles in the script should be modified to match your testbed.
32+
33+
```
34+
cd ~/ONCache/scripts; bash ./provision.sh
35+
```
36+
37+
### Step2: Install Compilation Requirements
38+
```
39+
sudo apt install -y -qq clang llvm libelf-dev libpcap-dev gcc-multilib build-essential cmake python3 >/dev/null
40+
```
41+
42+
### Step3: Compile ONCache
43+
ONCache should be compiled on all the hosts.
44+
```
45+
make all -C ~/ONCache
46+
```
47+
48+
### Step4: Start the ONCache daemon.
49+
To use ONCache, you can simply run the daemon.py on all the hosts to start the ONCache daemon. And the daemon will attach the eBPF program on all of the containers. Remember to change the NODE_IFNAME and POD_IFNAME to that of your testbed in daemon.py. **Skip this step to use the standard overlay network (Antrea in this tutorial).**
50+
```
51+
cd ~/ONCache/user_prog/; sudo python3 daemon.py
52+
```
53+
54+
### Step5: Run iperf3/netperf tests
55+
To do the performance test, we prepare a script to provision a test server and a client on the two hosts. Run the following commands **only on the master node**:
56+
```
57+
cd ~/ONCache/scripts; bash ./netperf_test.sh
58+
```
59+
On the return of the script, it prints the privision result of the two container.
60+
```
61+
ubuntu@node-0:~$ cd ~/ONCache/scripts; bash ./netperf_test.sh
62+
some outputs...
63+
+ kubectl get pods -owide
64+
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
65+
test-client 1/1 Running 0 5s 10.10.1.2 node2 <none> <none>
66+
test-server 1/1 Running 0 6s 10.10.0.4 node1 <none> <none>
67+
```
68+
69+
Then you can run iperf3 test on the containers:
70+
```
71+
kubectl exec -it test-server -- iperf3 -s
72+
# On another terminal
73+
kubectl exec -it test-client -- iperf3 -l1M -c <test-server-ip>
74+
```
75+
Or the netperf test:
76+
```
77+
kubectl exec -it test-server -- netserver -D
78+
# On another terminal
79+
kubectl exec -it test-client -- netperf -t TCP_RR -H <test-server-ip>
80+
```
81+
82+
## Results
83+
Here are our experiment results in the tutorial.
84+
85+
### ONCache
86+
```
87+
ubuntu@node-0:~# kubectl exec -it test-client -- iperf3 -l1M -c 10.10.0.30
88+
Connecting to host 10.10.0.30, port 5201
89+
[ 5] local 10.10.1.29 port 37960 connected to 10.10.0.30 port 5201
90+
[ ID] Interval Transfer Bitrate Retr Cwnd
91+
[ 5] 0.00-1.00 sec 4.25 GBytes 36.5 Gbits/sec 118 3.11 MBytes
92+
[ 5] 1.00-2.00 sec 4.55 GBytes 39.0 Gbits/sec 58 2.66 MBytes
93+
[ 5] 2.00-3.00 sec 4.13 GBytes 35.5 Gbits/sec 151 4.11 MBytes
94+
[ 5] 3.00-4.00 sec 4.30 GBytes 36.9 Gbits/sec 265 1.93 MBytes
95+
[ 5] 4.00-5.00 sec 4.55 GBytes 39.1 Gbits/sec 116 999 KBytes
96+
[ 5] 5.00-6.00 sec 4.23 GBytes 36.3 Gbits/sec 206 2.56 MBytes
97+
[ 5] 6.00-7.00 sec 4.56 GBytes 39.1 Gbits/sec 100 2.93 MBytes
98+
[ 5] 7.00-8.00 sec 4.43 GBytes 38.0 Gbits/sec 14 3.00 MBytes
99+
[ 5] 8.00-9.00 sec 4.31 GBytes 37.0 Gbits/sec 0 3.02 MBytes
100+
[ 5] 9.00-10.00 sec 4.00 GBytes 34.4 Gbits/sec 200 2.31 MBytes
101+
- - - - - - - - - - - - - - - - - - - - - - - - -
102+
[ ID] Interval Transfer Bitrate Retr
103+
[ 5] 0.00-10.00 sec 43.3 GBytes 37.2 Gbits/sec 1228 sender
104+
[ 5] 0.00-10.00 sec 43.3 GBytes 37.2 Gbits/sec receiver
105+
106+
iperf Done.
107+
108+
ubuntu@node-0:~# kubectl exec -it test-client -- netperf -H 10.10.0.30 -t TCP_RR
109+
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.30 (10.10.0) port 0 AF_INET : first burst 0
110+
Local /Remote
111+
Socket Size Request Resp. Elapsed Trans.
112+
Send Recv Size Size Time Rate
113+
bytes Bytes bytes bytes secs. per sec
114+
115+
16384 131072 1 1 10.00 36977.42
116+
16384 131072
117+
```
118+
119+
### Antrea
120+
```
121+
ubuntu@node-0:~# kubectl exec -it test-client -- iperf3 -l1M -c 10.10.0.29
122+
Connecting to host 10.10.0.29, port 5201
123+
[ 5] local 10.10.1.28 port 54742 connected to 10.10.0.29 port 5201
124+
[ ID] Interval Transfer Bitrate Retr Cwnd
125+
[ 5] 0.00-1.00 sec 4.01 GBytes 34.4 Gbits/sec 83 3.34 MBytes
126+
[ 5] 1.00-2.00 sec 4.24 GBytes 36.4 Gbits/sec 0 3.34 MBytes
127+
[ 5] 2.00-3.00 sec 3.96 GBytes 34.0 Gbits/sec 89 4.57 MBytes
128+
[ 5] 3.00-4.00 sec 4.00 GBytes 34.4 Gbits/sec 82 965 KBytes
129+
[ 5] 4.00-5.00 sec 4.29 GBytes 36.8 Gbits/sec 0 3.15 MBytes
130+
[ 5] 5.00-6.00 sec 4.35 GBytes 37.4 Gbits/sec 0 3.15 MBytes
131+
[ 5] 6.00-7.00 sec 4.33 GBytes 37.1 Gbits/sec 0 3.15 MBytes
132+
[ 5] 7.00-8.00 sec 4.20 GBytes 36.1 Gbits/sec 53 2.89 MBytes
133+
[ 5] 8.00-9.00 sec 4.19 GBytes 36.0 Gbits/sec 45 3.22 MBytes
134+
[ 5] 9.00-10.00 sec 4.34 GBytes 37.3 Gbits/sec 0 3.22 MBytes
135+
- - - - - - - - - - - - - - - - - - - - - - - - -
136+
[ ID] Interval Transfer Bitrate Retr
137+
[ 5] 0.00-10.00 sec 41.9 GBytes 36.0 Gbits/sec 352 sender
138+
[ 5] 0.00-10.00 sec 41.9 GBytes 36.0 Gbits/sec receiver
139+
140+
iperf Done.
141+
142+
ubuntu@node-0:~# kubectl exec -it test-client -- netperf -H 10.10.0.29 -t TCP_RR
143+
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.29 (10.10.0) port 0 AF_INET : first burst 0
144+
Local /Remote
145+
Socket Size Request Resp. Elapsed Trans.
146+
Send Recv Size Size Time Rate
147+
bytes Bytes bytes bytes secs. per sec
148+
149+
16384 131072 1 1 10.00 28246.08
150+
16384 131072
151+
```
152+
### Bare Metal
153+
```
154+
ubuntu@node-0:~# iperf3 -c 192.168.0.101 -l1M
155+
Connecting to host 192.168.0.101, port 5201
156+
[ 5] local 192.168.0.100 port 51930 connected to 192.168.0.101 port 5201
157+
[ ID] Interval Transfer Bitrate Retr Cwnd
158+
[ 5] 0.00-1.00 sec 4.31 GBytes 37.0 Gbits/sec 115 743 KBytes
159+
[ 5] 1.00-2.00 sec 3.95 GBytes 33.9 Gbits/sec 139 2.14 MBytes
160+
[ 5] 2.00-3.00 sec 4.33 GBytes 37.2 Gbits/sec 176 2.53 MBytes
161+
[ 5] 3.00-4.00 sec 4.98 GBytes 42.8 Gbits/sec 50 1.44 MBytes
162+
[ 5] 4.00-5.00 sec 4.33 GBytes 37.2 Gbits/sec 218 2.63 MBytes
163+
[ 5] 5.00-6.00 sec 5.07 GBytes 43.5 Gbits/sec 40 2.22 MBytes
164+
[ 5] 6.00-7.00 sec 4.47 GBytes 38.4 Gbits/sec 435 944 KBytes
165+
[ 5] 7.00-8.00 sec 4.52 GBytes 38.9 Gbits/sec 284 804 KBytes
166+
[ 5] 8.00-9.00 sec 5.12 GBytes 43.9 Gbits/sec 98 2.71 MBytes
167+
[ 5] 9.00-10.00 sec 4.90 GBytes 42.1 Gbits/sec 55 891 KBytes
168+
- - - - - - - - - - - - - - - - - - - - - - - - -
169+
[ ID] Interval Transfer Bitrate Retr
170+
[ 5] 0.00-10.00 sec 46.0 GBytes 39.5 Gbits/sec 1610 sender
171+
[ 5] 0.00-10.00 sec 46.0 GBytes 39.5 Gbits/sec receiver
172+
173+
iperf Done.
174+
175+
ubuntu@node-0:~# netperf -H 192.168.0.101 -t TCP_RR
176+
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.101 () port 0 AF_INET : demo : first burst 0
177+
Local /Remote
178+
Socket Size Request Resp. Elapsed Trans.
179+
Send Recv Size Size Time Rate
180+
bytes Bytes bytes bytes secs. per sec
181+
182+
16384 131072 1 1 10.00 39271.76
183+
16384 131072
184+
```

‎common/common.mk

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
LLC := llc
2+
CLANG := clang
3+
CC := gcc
4+
CPP := g++
5+
6+
COMMON_DIR ?= ../common/
7+
LIBBPF_DIR ?= ../libbpf/src/
8+
YAML_DIR ?= ../yaml-cpp/build/
9+
OBJECT_LIBBPF = $(LIBBPF_DIR)/libbpf.a
10+
OBJECT_YAML = $(YAML_DIR)/libyaml-cpp.a
11+
12+
EXTRA_DEPS += $(COMMON_DIR)/common_defines.h
13+
14+
CFLAGS ?= -g -I../headers/ -I../common/ -I../yaml-cpp/include/
15+
LDFLAGS ?= -L$(LIBBPF_DIR) -L$(YAML_DIR)
16+
17+
LIBS = -l:libbpf.a -l:libyaml-cpp.a -lz -lelf
18+
19+
# For build dependency on this file, if it gets updated
20+
COMMON_MK = $(COMMON_DIR)/common.mk

‎common/common_defines.h

+113
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
#ifndef __COMMON_DEFINES_H
2+
#define __COMMON_DEFINES_H
3+
4+
#include <stdint.h>
5+
#include <stdbool.h>
6+
#include <stdlib.h>
7+
#include <stdio.h>
8+
#include <locale.h>
9+
#include <unistd.h>
10+
#include <time.h>
11+
#include <sys/sysinfo.h>
12+
#include <sys/statfs.h> /* statfs */
13+
#include <sys/stat.h> /* stat(2) + S_IRWXU */
14+
#include <sys/mount.h> /* mount(2) */
15+
16+
#include <net/if.h>
17+
#include <linux/if_ether.h>
18+
#include <linux/types.h>
19+
#include <linux/pkt_cls.h>
20+
#include <linux/pkt_sched.h> /* TC_H_MAJ + TC_H_MIN */
21+
#include <linux/if_packet.h>
22+
#include <linux/ip.h>
23+
#include <linux/udp.h>
24+
#include <linux/tcp.h>
25+
#include <arpa/inet.h>
26+
#include <linux/types.h>
27+
#include <getopt.h>
28+
#include "linux/bpf.h"
29+
30+
#define MACLEN 14
31+
#define IPLEN 20
32+
#define UDPLEN 8
33+
#define TCPLEN 20
34+
#define VXLANLEN 8
35+
36+
#define bpf_printkm(fmt, ...) \
37+
({ \
38+
char ____fmt[] = fmt; \
39+
bpf_trace_printk(____fmt, sizeof(____fmt), \
40+
##__VA_ARGS__); \
41+
})
42+
43+
#define MAX_IFINDEX 4096
44+
45+
struct bpf_elf_map {
46+
__u32 type;
47+
__u32 size_key;
48+
__u32 size_value;
49+
__u32 max_elem;
50+
__u32 flags;
51+
__u32 id;
52+
__u32 pinning;
53+
__u32 inner_id;
54+
__u32 inner_idx;
55+
};
56+
57+
// Must pad the struct to avoid eBPf verifier
58+
// think the stack boundary is iligal.
59+
// #pragma pack(1)
60+
struct fivetuple {
61+
__be32 laddr;
62+
__be32 raddr;
63+
__be16 lport;
64+
__be16 rport;
65+
__u32 protocol;
66+
};
67+
68+
struct egressinfo {
69+
unsigned char outer_header[64];
70+
__u32 ifidx;
71+
};
72+
73+
struct action {
74+
__u16 ingress;
75+
__u16 egress;
76+
};
77+
78+
struct devinfo {
79+
__be32 ip;
80+
unsigned char mac[ETH_ALEN];
81+
};
82+
83+
struct rule {
84+
struct fivetuple fivetuple_;
85+
int isIngress;
86+
};
87+
88+
// Should write dmac to a map forahead becuase pod mac is not carried in VXLAN
89+
struct ingressinfo {
90+
__u32 ifidx;
91+
unsigned char dmac[ETH_ALEN];
92+
unsigned char smac[ETH_ALEN];
93+
};
94+
95+
#define PORT_AVAILABLE 1024
96+
97+
int verbose;
98+
99+
#define EXIT_OK 0 /* == EXIT_SUCCESS (stdlib.h) man exit(3) */
100+
#define EXIT_FAIL 1 /* == EXIT_FAILURE (stdlib.h) man exit(3) */
101+
#define EXIT_FAIL_OPTION 2
102+
#define EXIT_FAIL_XDP 3
103+
#define EXIT_FAIL_MAP 20
104+
#define EXIT_FAIL_MAP_KEY 21
105+
#define EXIT_FAIL_MAP_FILE 22
106+
#define EXIT_FAIL_MAP_FS 23
107+
#define EXIT_FAIL_IP 30
108+
#define EXIT_FAIL_CPU 31
109+
#define EXIT_FAIL_BPF 40
110+
#define EXIT_FAIL_BPF_ELF 41
111+
#define EXIT_FAIL_BPF_RELOCATE 42
112+
113+
#endif

0 commit comments

Comments
 (0)
Please sign in to comment.