network policy workload

Network policy testing touches 3 key areas 1. Template to include all network policy configuration options 2. Latency measurement through connection testing 3. Flow tracking through Convergence tracker This PR is adding templates which proposed unified approach to add namespace and pod selectors, CIDRs, single ports and port ranges. Legacy network policy templates will be retired and removed once this new approach gets stabilized. This PR depends on following kube-burner and images PRs for connection testing cloud-bulldozer/images#13 cloud-bulldozer/images#12 kube-burner/kube-burner#679 A follow up PR will be submitted for adding Converagnce tracker tool support. More details are added in README.md Signed-off-by: venkataanil <[email protected]>
kube-burner · Oct 7, 2024 · 89f1807 · 89f1807
1 parent 8bc08a1
commit 89f1807
Show file tree

Hide file tree

Showing 10 changed files with 637 additions and 22 deletions.
diff --git a/README.md b/README.md
@@ -178,30 +178,95 @@ For User-Defined Network (UDN) L3 segmentation testing. It creates two deploymen
 
 ## Network Policy workloads
 
-With the help of [networkpolicy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) object we can control traffic flow at the IP address or port level in Kubernetes. A networkpolicy can come in various shapes and sizes. Allow traffic from a specific namespace, Deny traffic from a specific pod IP, Deny all traffic, etc. Hence we have come up with a few test cases which try to cover most of them. They are as follows.
+Network policy scale testing tooling involved  3 components:
+1. Template to include all network policy configuration options
+2. Latency measurement through connection testing
+3. Flow tracking through Convergence tracker
 
-### networkpolicy-multitenant
+A network policy defines the rules for ingress and egress traffic between pods in  local and remote namespaces. These remote namespace addresses can be configured using a combination of namespace and pod selectors, CIDRs, ports, and port ranges. Given that network policies offer a wide variety of configuration options, we developed a unified template that incorporates all these configuration parameters. Users can specify the desired count for each option.
 
-- 500 namespaces
-- 20 pods in each namespace. Each pod acts as a server and a client
-- Default deny networkpolicy is applied first that blocks traffic to any test namespace
-- 3 network policies in each namespace that allows traffic from the same namespace and two other namespaces using namespace selectors
+```console
+spec:
+  podSelector:
+    matchExpressions:
+    - key: num
+      operator: In
+      values:
+      - "1"
+      - "2"
+  ingress:
+  - from:
+    - namespaceSelector:
+        matchExpressions:
+        - key: kubernetes.io/metadata.name
+          operator: In
+          values:
+          - network-policy-perf-13
+          - network-policy-perf-14
+      podSelector:
+       matchExpressions:
+       - key: num
+         operator: In
+         values:
+         - "1"
+         - "2"
+    ports:
+    - port: 8080
+      protocol: TCP
+
+```
 
-### networkpolicy-matchlabels
+### Scale Testing and Unique ACL Flows
+In our scale tests, we aim to create between 10 to 100 network policies within a single namespace. The primary focus is on preventing duplicate configuration options, which ensures that each network policy generates unique Access Control List (ACL) flows. To achieve this, we carefully designed our templating approach based on the following considerations:
+
+**Round-Robin Assignment:** We use a round-robin strategy to distribute
+1. remote namespaces among ingress and egress rules across kube burner job iterations
+2. remote namespaces among ingress and egress rules in the same kube burner job iteration
+
+This ensures that we don’t overuse the same remote namespaces in a single iteration or among multiple interations. For instance, if namespace-1 uses namespace-2 and namespace-3 as its remote namespaces, then namespace-2 will start using namespace-4 and namespace-5 as remote namespaces in the next iteration.
+
+**Unique Namespace and Pod Combinations:** To avoid redundant flows, the templating system generates unique combinations of remote namespaces and pods for each network policy. Initially, we iterate through the list of remote namespaces, and once all remote namespaces are exhausted, we move on to iterate through the remote pods. This method ensures that every network policy within a namespace is assigned a distinct combination of remote namespaces and remote pods, avoiding duplicate pairs.
+
+**Templating Logic**
+Our templating logic is implemented as follows:
+``` console
+// Iterate over the list of namespaces to configure network policies.
+for namespace := namespaces {
+
+  // Each network policy uses a combination of a remote namespace and a remote pod to allow traffic.
+  for networkPolicy := networkPolicies {
+
+    /*
+    Iterate through the list of remote pods. Once all remote namespaces are exhausted,
+    continue iterating through the remote pods to ensure unique namespace/pod combinations.
+    */
+    for i, remotePod := range remotePods {
+        // Stop when we reach the maximum number of remote pods allowed.
+        if i == num_remote_pods {
+            break
+        }
+
+        // Iterate through the list of remote namespaces to pair with the remote pod.
+        for idx, remoteNamespace := range remoteNamespaces {
+            // Combine the remote namespace and pod into a unique pair for ACL configuration.
+            combine := fmt.Sprintf("%s:%s", remoteNamespace, remotePod)
+
+            // Stop iterating once we’ve exhausted the allowed number of remote namespaces.
+            if idx == num_remote_namespace {
+                break
+            }
+        }
+    }
+  }
+}
 
-- 5 namespaces
-- 100 pods in each namespace. Each pod acts as a server and a client
-- Each pod with 2 labels and each label shared is by 5 pods
-- Default deny networkpolicy is applied first
-- Then for each unique label in a namespace we have a networkpolicy with that label as a podSelector which allows traffic from pods with some other randomly selected label. This translates to 40 networkpolicies/namespace
+```
 
-### networkpolicy-matchexpressions
+**CIDRs and Port Ranges**
+We apply the same round-robin and unique combination logic to CIDRs and port ranges, ensuring that these options are not reused in network policies within the same namespace.
 
-- 5 namespaces
-- 25 pods in each namespace. Each pod acts as a server and a client
-- Each pod with 2 labels and each label shared is by 5 pods
-- Default deny networkpolicy is applied first
-- Then for each unique label in a namespace we have a networkpolicy with that label as a podSelector which allows traffic from pods which *don't* have some other randomly-selected label. This translates to 10 networkpolicies/namespace
+**Connection Testing Support**
+kube-burner measures network policy latency through connection testing. Currently, all pods are configured to listen on port 8080. As a result, client pods will send requests to port 8080 during testing.
 
 ## EgressIP workloads
 

diff --git a/cmd/config/network-policy/egress-np.yml b/cmd/config/network-policy/egress-np.yml
@@ -0,0 +1,95 @@
+kind: NetworkPolicy
+apiVersion: networking.k8s.io/v1
+metadata:
+  name: egress-{{.Iteration}}-{{.Replica}}
+spec:
+  {{- $lpList := list }}
+  {{- range $lp, $e := until $.local_pods }}
+    {{- $nextPod := add 1 $lp }}
+    {{- $lps := (toString $nextPod)}}
+    {{- $lpList = append $lpList $lps }}  
+  {{- end }}
+  {{- $lpNames := toJson $lpList }}
+  podSelector:   
+    matchExpressions:
+      - key: num
+        operator: In
+        values: {{$lpNames}}
+  egress:  
+  {{- $peerPodIdx := mul $.Replica .pod_selectors .peer_namespaces}}
+  {{- $peerPodIdx = div (sub $peerPodIdx 1) $.namespaces}}
+  {{- $peerPodIdx = mul $peerPodIdx .peer_pods}}
+  {{- $peerPodList := list }}
+  {{- range $lp, $e := until $.peer_pods }}
+    {{- $nextPod := add 1 $lp $peerPodIdx }}
+    {{- if gt $nextPod $.pods_per_namespace }}
+      {{- $nextPod = add (mod $nextPod $.pods_per_namespace) 1 }}
+    {{- end }}
+    {{- $lps := (toString $nextPod)}}
+    {{- $peerPodList = append $peerPodList $lps }}  
+  {{- end }}
+  {{- $peerPodNames := toJson $peerPodList }}
+  {{- $peerNsIdx := add (mul $.Iteration .pod_selectors .netpols_per_namespace .peer_namespaces) (mul (sub $.Replica 1) .pod_selectors .peer_namespaces) 1}}
+  {{- range $ps, $e := until $.pod_selectors }}
+    {{- $nsStart := add $peerNsIdx (mul $ps $.peer_namespaces) }}
+    {{- $nsList := list }}
+    {{- range $i, $v := until $.peer_namespaces }}
+      {{- $nextNs := add $nsStart $i }}
+      {{- if ge $nextNs $.namespaces }}
+        {{- $nextNs = mod $nextNs $.namespaces }}
+      {{- end }}
+      {{- $next_namespace := print "network-policy-perf-" $nextNs }}
+      {{- $nsList = append $nsList $next_namespace }}
+    {{- end }}
+    {{- $nsNames := toJson $nsList }}
+  - to:
+    - namespaceSelector:
+        matchExpressions:
+          - key: kubernetes.io/metadata.name
+            operator: In
+            values: {{$nsNames}}
+      podSelector:
+        matchExpressions:
+          - key: num
+            operator: In
+            values: {{$peerPodNames}}
+    ports:
+    {{- $single_port := 8079 }}
+    {{- range $i, $e := until $.single_ports }}
+      {{- $single_port = add $single_port 1 }}
+      - protocol: TCP
+        port: {{$single_port}}
+    {{- end }}
+    {{- $rangeStart := 5000 }}
+    {{- range $i, $e := until $.port_ranges }}
+      {{- $rangeEnd := add $rangeStart 5 }}
+      - protocol: TCP
+        port: {{$rangeStart}}
+        endPort: {{$rangeEnd}}
+      {{ $rangeStart = add $rangeStart 10}}
+    {{- end }}        
+  {{- end }}
+  {{- $subnetStartIdx := add (mul $.Replica $.cidr_rules) 1 }}
+  {{- range $i, $e := until .cidr_rules }}
+      {{- $subnetIdx := add $subnetStartIdx $i }}
+  - to:
+    - ipBlock:
+        cidr: {{GetSubnet24 (int $subnetIdx) }}
+    ports:
+      {{- $single_port := 1000 }}
+      {{- range $i, $e := until $.single_ports }}
+        {{- $single_port = add $single_port 1 }}
+      - protocol: TCP
+        port: {{$single_port}}
+      {{- end }}
+      {{- $rangeStart := 5000 }}
+      {{- range $i, $e := until $.port_ranges }}
+        {{- $rangeEnd := add $rangeStart 5 }}
+      - protocol: TCP
+        port: {{$rangeStart}}
+        endPort: {{$rangeEnd}}
+        {{ $rangeStart = add $rangeStart 10}}
+      {{- end }}        
+  {{- end }}
+  policyTypes:
+    - Egress
diff --git a/cmd/config/network-policy/ingress-np.yml b/cmd/config/network-policy/ingress-np.yml
@@ -0,0 +1,95 @@
+kind: NetworkPolicy
+apiVersion: networking.k8s.io/v1
+metadata:
+  name: ingress-{{.Iteration}}-{{.Replica}}
+spec:
+  {{- $lpList := list }}
+  {{- range $lp, $e := until $.local_pods }}
+    {{- $nextPod := add 1 $lp }}
+    {{- $lps := (toString $nextPod)}}
+    {{- $lpList = append $lpList $lps }}  
+  {{- end }}
+  {{- $lpNames := toJson $lpList }}
+  podSelector:   
+    matchExpressions:
+      - key: num
+        operator: In
+        values: {{$lpNames}}
+  ingress:  
+  {{- $peerPodIdx := mul $.Replica .pod_selectors .peer_namespaces}}
+  {{- $peerPodIdx = div (sub $peerPodIdx 1) $.namespaces}}
+  {{- $peerPodIdx = mul $peerPodIdx .peer_pods}}
+  {{- $peerPodList := list }}
+  {{- range $lp, $e := until $.peer_pods }}
+    {{- $nextPod := add 1 $lp $peerPodIdx }}
+    {{- if gt $nextPod $.pods_per_namespace }}
+      {{- $nextPod = add (mod $nextPod $.pods_per_namespace) 1 }}
+    {{- end }}
+    {{- $lps := (toString $nextPod)}}
+    {{- $peerPodList = append $peerPodList $lps }}  
+  {{- end }}
+  {{- $peerPodNames := toJson $peerPodList }}
+  {{- $peerNsIdx := add (mul $.Iteration .pod_selectors .netpols_per_namespace .peer_namespaces) (mul (sub $.Replica 1) .pod_selectors .peer_namespaces) 1}}
+  {{- range $ps, $e := until $.pod_selectors }}
+    {{- $nsStart := add $peerNsIdx (mul $ps $.peer_namespaces) }}
+    {{- $nsList := list }}
+    {{- range $i, $v := until $.peer_namespaces }}
+      {{- $nextNs := add $nsStart $i }}
+      {{- if ge $nextNs $.namespaces }}
+        {{- $nextNs = mod $nextNs $.namespaces }}
+      {{- end }}
+      {{- $next_namespace := print "network-policy-perf-" $nextNs }}
+      {{- $nsList = append $nsList $next_namespace }}
+    {{- end }}
+    {{- $nsNames := toJson $nsList }}
+  - from:
+    - namespaceSelector:
+        matchExpressions:
+          - key: kubernetes.io/metadata.name
+            operator: In
+            values: {{$nsNames}}
+      podSelector:
+        matchExpressions:
+          - key: num
+            operator: In
+            values: {{$peerPodNames}}
+    ports:
+    {{- $single_port := 8079 }}
+    {{- range $i, $e := until $.single_ports }}
+      {{- $single_port = add $single_port 1 }}
+      - protocol: TCP
+        port: {{$single_port}}
+    {{- end }}
+    {{- $rangeStart := 5000 }}
+    {{- range $i, $e := until $.port_ranges }}
+      {{- $rangeEnd := add $rangeStart 5 }}
+      - protocol: TCP
+        port: {{$rangeStart}}
+        endPort: {{$rangeEnd}}
+      {{ $rangeStart = add $rangeStart 10}}
+    {{- end }}        
+  {{- end }}
+  {{- $subnetStartIdx := add (mul $.Replica $.cidr_rules) 1 }}
+  {{- range $i, $e := until .cidr_rules }}
+      {{- $subnetIdx := add $subnetStartIdx $i }}
+  - from:
+    - ipBlock:
+        cidr: {{GetSubnet24 (int $subnetIdx) }}
+    ports:
+      {{- $single_port := 1000 }}
+      {{- range $i, $e := until $.single_ports }}
+        {{- $single_port = add $single_port 1 }}
+      - protocol: TCP
+        port: {{$single_port}}
+      {{- end }}
+      {{- $rangeStart := 5000 }}
+      {{- range $i, $e := until $.port_ranges }}
+        {{- $rangeEnd := add $rangeStart 5 }}
+      - protocol: TCP
+        port: {{$rangeStart}}
+        endPort: {{$rangeEnd}}
+        {{ $rangeStart = add $rangeStart 10}}
+      {{- end }}        
+  {{- end }}
+  policyTypes:
+    - Ingress