Skip to content

Commit a9643ea

Browse files
author
logwang
committed
init
1 parent 08d6a85 commit a9643ea

File tree

7,717 files changed

+4051144
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

7,717 files changed

+4051144
-1
lines changed

‎.gitignore

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
*~
2+
*.a
3+
*.core
4+
*.lo
5+
*.la
6+
*.o
7+
*.so

‎F-Stack.png

48.1 KB
Loading

‎LICENSE

+216
Large diffs are not rendered by default.

‎README.md

+105-1
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,106 @@
11
# F-Stack
2-
Since the F-Stack project involves a large number of third party code, there are many open source licenses, copyrights need to be reviewed. So, the project is closed temporarily, and when this work is completed, we will reopen the project before April 24th.
2+
![](F-Stack.png)
3+
4+
## Introduction
5+
With the rapid development of NIC, the poor performance of data packets processing with Linux kernel has become the bottleneck. However, the rapid development of the Internet needs high performance of network processing, kernel bypass has caught more and more attention. There are various similar technologies appear, such as DPDK, NETMAP and PF_RING. The main idea of kernel bypass is that Linux is only used to deal with control flow, all data streams are processed in user space. Therefore, kernel bypass can avoid performance bottlenecks caused by kernel packet copy, thread scheduling, system calls and interrupt. Furthermore, kernel bypass can achieve higher performance with multi optimizing methods. Within various techniques, DPDK has been widely used because of its more thorough isolation from kernel scheduling and active community support.
6+
7+
[F-Stack](http://www.f-stack.org/?from=github) is an open source network framework with high performance based on DPDK. With follow characteristics
8+
9+
1. Ultra high network performance which can achieve network card under full load, 10 million concurrent connection, 5 million RPS, 1 million CPS.
10+
2. Transplant FreeBSD 11.01 user space stack, provides a complete stack function, cut a great amount of irrelevant features. Therefore greatly enhance the performance.
11+
3. Support Nginx, Redis and other mature applications, service can easily use F-Stack
12+
4. With Multi-process architecture, easy to extend
13+
5. Provide micro thread interface. Various applications with stateful app can easily use F-Stack to get high performance without processing complex asynchronous logic.
14+
6. Provide Epoll/Kqueue interface that allow many kinds of applications easily use F-Stack
15+
16+
## History
17+
18+
In order to deal with the increasingly severe DDoS attacks, authorized DNS server of Tencent Cloud DNSPod switched from Gigabit Ethernet to 10-Gigabit at the end of 2012. We faced several options, one is to continue to use the original model another is to use kernel bypass technology. After several rounds of investigation, we finally chose to develop our next generation of DNS server based on DPDK. The reason is DPDK provides ultra-high performance and can be seamlessly extended to 40G, or even 100G NIC in the future.
19+
20+
After several months of development and testing, DKDNS, high-performance DNS server based on DPDK officially released in October 2013. It's capable of achieving up to 11 million QPS with a single 10GE port and 18.2 million QPS with two 10GE ports. And then we developed a user-space TCP/IP stack called F-Stack that can process 0.6 million RPS with a single 10GE port.
21+
22+
With the fast growth of Tencent Cloud, more and more services need higher network access performance. Meanwhile, F-Stack was continuous improving driven by the business growth, and ultimately developed into a general network access framework. But this TCP/IP stack couldn't meet the needs of these services while continue to develop and maintain a complete network stack will cost high, we've tried several plans and finally determined to port FreeBSD(11.0 stable) TCP/IP stack into F-Stack. Thus, we can reduce the cost of maintenance and follow up the improvement from community quickly.Thanks to [libplebnet](https://gitorious.org/freebsd/kmm-sandbox/commit/fa8a11970bc0ed092692736f175925766bebf6af?p=freebsd:kmm-sandbox.git;a=tree;f=lib/libplebnet;h=ae446dba0b4f8593b69b339ea667e12d5b709cfb;hb=refs/heads/work/svn_trunk_libplebnet) and [libuinet](https://github.com/pkelsey/libuinet), this work becomes a lot easier.
23+
24+
With the rapid development of all kinds of application, in order to help different APPs quick and easily use F-Stack, F-Stack has integrated Nginx, Redis and other commonly used APPs, and a micro thread framework, and provides a standard Epoll/Kqueue interface.
25+
26+
Currently, besides authorized DNS server of DNSPod, there are various products in Tencent Cloud has used the F-Stack, such as HttpDNS (D+), COS access module, CDN access module, etc..
27+
28+
## Quick Start
29+
30+
#clone F-Stack
31+
mkdir /data/f-stack
32+
git clone https://github.com/F-Stack/f-stack.git /data/f-stack
33+
34+
cd f-stack
35+
# compile DPDK
36+
cd dpdk/tools
37+
./dpdk-setup.sh # compile with x86_64-native-linuxapp-gcc
38+
39+
# Set hugepage
40+
# single-node system
41+
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
42+
43+
# or NUMA
44+
echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
45+
echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
46+
47+
# Using Hugepage with the DPDK
48+
mkdir /mnt/huge
49+
mount -t hugetlbfs nodev /mnt/huge
50+
51+
# offload NIC
52+
modprobe uio
53+
insmod /data/f-stack/dpdk/x86_64-native-linuxapp-gcc/build/kmod/igb_uio.ko
54+
insmod /data/f-stack/dpdk/x86_64-native-linuxapp-gcc/build/kmod/rte_kni.ko
55+
python dpdk-devbind.py --status
56+
ifconfig eth0 down
57+
python dpdk-devbind.py --bind=igb_uio eth0 # assuming that use 10GE NIC and eth0
58+
59+
# Compile F-Stack
60+
cd ../../lib/
61+
make
62+
export FF_PATH=/data/f-stack
63+
export FF_DPDK=/data/f-stack/dpdk/x86_64-native-linuxapp-gcc/lib
64+
65+
#### Nginx
66+
67+
cd app/nginx-1.11.10
68+
./configure --prefix=/usr/local/nginx_fstack --with-ff_module
69+
make
70+
make install
71+
cd ../..
72+
./start.sh -b /usr/local/nginx_fstack/sbin/nginx -c config.ini
73+
74+
#### Redis
75+
76+
cd app/redis-3.2.8/
77+
make
78+
make install
79+
80+
81+
## Nginx Testing Result
82+
83+
Test environment
84+
85+
NIC:Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+
86+
CPU:Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
87+
Memory：128G
88+
OS:CentOS Linux release 7.2 (Final)
89+
Kernel：3.10.104-1-tlinux2-0041.tl2
90+
91+
Nginx uses linux kernel's default config, all soft interrupts are working in the first CPU core.
92+
93+
Nginx si means modify the smp_affinity of every IRQ, so that the decision to service an interrupt with a particular CPU is made at the hardware level, with no intervention from the kernel.
94+
95+
96+
CPS (Connection:close, Small data packet) test result
97+
![](http://i.imgur.com/PvCRmXR.png)
98+
99+
RPS (Connection:Keep-Alive, Small data packet) test data
100+
![](http://i.imgur.com/CTDPx3a.png)
101+
102+
Bandwidth (Connection:Keep-Alive, 3.7k bytes data packet) test data
103+
![](http://i.imgur.com/1ZM6yT9.png)
104+
105+
## Licenses
106+
See [LICENSE](LICENSE)

‎app/micro_thread/Makefile

+95
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
2+
#
3+
# Tencent is pleased to support the open source community by making MSEC available.
4+
#
5+
# Copyright (C) 2016 THL A29 Limited, a Tencent company. All rights reserved.
6+
#
7+
# Licensed under the GNU General Public License, Version 2.0 (the "License");
8+
# you may not use this file except in compliance with the License. You may
9+
# obtain a copy of the License at
10+
#
11+
# https://opensource.org/licenses/GPL-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing, software distributed under the
14+
# License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
15+
# either express or implied. See the License for the specific language governing permissions
16+
# and limitations under the License.
17+
#
18+
19+
20+
########MAKEFILE##########
21+
22+
ifeq ($(FF_PATH),)
23+
$(error variable FF_PATH is not set)
24+
endif
25+
26+
DEBUG= -g
27+
BINARY = libmt.a
28+
FF_LIB=$(FF_PATH)/libfstack.a
29+
DPDK_LIBS = $(shell pkg-config --define-variable=TOPDIR=${FF_PATH} --libs f-stack.pc)
30+
31+
# Comment the following line if you are not using the gnu c compiler
32+
#C_ARGS = -Wall -g -fPIC -D_DEBUG
33+
C_ARGS = -Wall -g -fPIC -O0 -DMT_REAL_TIME
34+
#.SUFFIXES: .o .cpp
35+
36+
ifeq ($(ARCH),32)
37+
C_ARGS += -march=pentium4 -m32 -DSUS_LINUX -pthread
38+
else
39+
C_ARGS += -m64 -DSUS_LINUX -pthread
40+
endif
41+
# You might have to change this if your c compiler is not cc
42+
CC = g++
43+
44+
# You shouldn't need to make any more changes below this line.
45+
INCCOMM=-I./valgrind -I$(FF_PATH)/lib
46+
47+
#.c.o:
48+
# $(CC) -c $*.c
49+
50+
all: $(BINARY)
51+
52+
### shell color config
53+
RED = \\e[1m\\e[31m
54+
DARKRED = \\e[31m
55+
GREEN = \\e[1m\\e[32m
56+
DARKGREEN = \\e[32m
57+
BLUE = \\e[1m\\e[34m
58+
DARKBLUE = \\e[34m
59+
YELLOW = \\e[1m\\e[33m
60+
DARKYELLOW = \\e[33m
61+
MAGENTA = \\e[1m\\e[35m
62+
DARKMAGENTA = \\e[35m
63+
CYAN = \\e[1m\\e[36m
64+
DARKCYAN = \\e[36m
65+
RESET = \\e[m
66+
CRESET = ;echo -ne \\e[m; test -s $@
67+
68+
%.o: %.cpp
69+
@echo -e Compiling $(GREEN)$<$(RESET) ...$(RED)
70+
@$(CC) $(C_ARGS) -c -o $@ $< $(INCCOMM) $(CRESET)
71+
72+
%.o: %.c
73+
@echo -e Compiling $(GREEN)$<$(RESET) ...$(RED)
74+
@$(CC) $(C_ARGS) -c -o $@ $< $(INCCOMM) $(CRESET)
75+
76+
%.o: %.S
77+
@echo -e Compiling $(GREEN)$<$(RESET) ...$(RED)
78+
@$(CC) $(C_ARGS) -c -o $@ $< $(INCCOMM) $(CRESET)
79+
80+
clean:
81+
@rm -f $(BINARY) *.a *.o echo
82+
83+
84+
LIB_OBJ = micro_thread.o kqueue_proxy.o arch_ctx.o mt_session.o mt_notify.o mt_action.o mt_mbuf_pool.o mt_api.o\
85+
mt_connection.o mt_concurrent.o mt_sys_hook.o ff_hook.o heap_timer.o mt_cache.o mt_net.o
86+
87+
libmt.a: $(LIB_OBJ)
88+
@echo -e Linking $(CYAN)$@$(RESET) ...$(RED)
89+
@-rm -f $@
90+
@ar crs $@ $^ $(FST_LIB) $(CRESET)
91+
@chmod +x $@
92+
93+
echo: echo.o libmt.a
94+
@echo -e Compile $(CYAN)$@$(RESET) ...$(RED)
95+
@$(CC) -O -gdwarf-2 -o $@ $^ -lstdc++ -ldl -lm $(DPDK_LIBS) $(CRESET)

‎app/micro_thread/arch_ctx.S

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
2+
/**
3+
* Tencent is pleased to support the open source community by making MSEC available.
4+
*
5+
* Copyright (C) 2016 THL A29 Limited, a Tencent company. All rights reserved.
6+
*
7+
* Licensed under the GNU General Public License, Version 2.0 (the "License");
8+
* you may not use this file except in compliance with the License. You may
9+
* obtain a copy of the License at
10+
*
11+
* https://opensource.org/licenses/GPL-2.0
12+
*
13+
* Unless required by applicable law or agreed to in writing, software distributed under the
14+
* License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
15+
* either express or implied. See the License for the specific language governing permissions
16+
* and limitations under the License.
17+
*/
18+
19+
20+
#
21+
# context x86 or x86_64 save and restore
22+
#
23+
# x86_64 x86
24+
# 0 %rbx %ebx
25+
# 1 %rsp %esp
26+
# 2 %rbp %ebp
27+
# 3 %r12 %esi
28+
# 4 %r13 %edi
29+
# 5 %r14 %eip
30+
# 6 %r15
31+
# 7 %rip
32+
33+
34+
35+
#if defined(__amd64__) || defined(__x86_64__)
36+
37+
##
38+
# @brief save_context
39+
##
40+
.text
41+
.align 4
42+
.globl save_context
43+
.type save_context, @function
44+
save_context:
45+
pop %rsi
46+
xorl %eax,%eax
47+
movq %rbx,(%rdi)
48+
movq %rsp,8(%rdi)
49+
push %rsi
50+
movq %rbp,16(%rdi)
51+
movq %r12,24(%rdi)
52+
movq %r13,32(%rdi)
53+
movq %r14,40(%rdi)
54+
movq %r15,48(%rdi)
55+
movq %rsi,56(%rdi)
56+
ret
57+
58+
.size save_context,.-save_context
59+
60+
##
61+
# @brief restore_context
62+
##
63+
.text
64+
.align 4
65+
.globl restore_context
66+
.type restore_context, @function
67+
restore_context:
68+
movl %esi,%eax
69+
movq (%rdi),%rbx
70+
movq 8(%rdi),%rsp
71+
movq 16(%rdi),%rbp
72+
movq 24(%rdi),%r12
73+
movq 32(%rdi),%r13
74+
movq 40(%rdi),%r14
75+
movq 48(%rdi),%r15
76+
jmp *56(%rdi)
77+
78+
.size restore_context,.-restore_context
79+
80+
##
81+
# @brief replace_esp
82+
##
83+
.text
84+
.align 4
85+
.globl replace_esp
86+
.type replace_esp, @function
87+
replace_esp:
88+
movq %rsi,8(%rdi)
89+
ret
90+
91+
.size replace_esp,.-replace_esp
92+
93+
94+
#elif defined(__i386__)
95+
96+
##
97+
# @brief save_context
98+
##
99+
.text
100+
.align 4
101+
.globl save_context
102+
.type save_context, @function
103+
save_context:
104+
movl 4(%esp),%edx
105+
popl %ecx
106+
xorl %eax,%eax
107+
movl %ebx,(%edx)
108+
movl %esp,4(%edx)
109+
pushl %ecx
110+
movl %ebp,8(%edx)
111+
movl %esi,12(%edx)
112+
movl %edi,16(%edx)
113+
movl %ecx,20(%edx)
114+
ret
115+
116+
.size save_context,.-save_context
117+
118+
119+
##
120+
# @brief restore_context
121+
##
122+
.text
123+
.align 4
124+
.globl restore_context
125+
.type restore_context, @function
126+
restore_context:
127+
movl 4(%esp),%edx
128+
movl 8(%esp),%eax
129+
movl (%edx),%ebx
130+
movl 4(%edx),%esp
131+
movl 8(%edx),%ebp
132+
movl 12(%edx),%esi
133+
movl 16(%edx),%edi
134+
jmp *20(%edx)
135+
136+
.size restore_context,.-restore_context
137+
138+
##
139+
# @brief replace_esp
140+
##
141+
.text
142+
.align 4
143+
.globl replace_esp
144+
.type replace_esp, @function
145+
replace_esp:
146+
movl 4(%esp),%edx
147+
movl 8(%esp),%eax
148+
movl %eax,4(%edx)
149+
ret
150+
151+
.size replace_esp,.-replace_esp
152+
153+
154+
#else
155+
#error "Linux cpu arch not supported"
156+
#endif
157+

0 commit comments

Comments
 (0)