Automated cherry pick of #5532: Improvements to FlowExporter benchmarks #5592: Remove redundant log in fillPodInfo/fillServiceInfo #4252: e2e framework for Antrea's native secondary network #5731: Enhance Records Filtering: Utilizing Labels In The E2E Test #5770: Improve flow-visibility e2e test #5882
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Also add BenchmarkConnStore, meant to measure memory usage of the connection store (and all the connections). Signed-off-by: Antonin Bas <[email protected]>
The net/netip package was introduced in Go 1.18. Compared to the existing net.IP type, the netip.Addr type takes less memory (especially for IPv6 addresses), is immutable, and is comparable so it supports == and can be used as a map key. In our case, this means that the flowexporter.Connection type takes a little less memory, and that the flowexporter.Tuple can be used directly as the flowexporter.ConnectionKey (no need to generate a string for use as the map key). Overall, this leads to higher performance (decreased CPU and memory usage) for the FlowExporter. There is potential for further improvement, as several key functions take as parameters both the flowexporter.Connection and flowexporter.ConnectionKey objects, which is no longer really required. See antrea-io#5271 Signed-off-by: Antonin Bas <[email protected]>
This release uses netip.Addr instead of net.IP to represent connections. Signed-off-by: Antonin Bas <[email protected]>
Signed-off-by: Antonin Bas <[email protected]>
Signed-off-by: Antonin Bas <[email protected]>
Signed-off-by: Yun-Tang Hsu <[email protected]>
Signed-off-by: Yun-Tang Hsu <[email protected]>
Signed-off-by: Yun-Tang Hsu <[email protected]>
Signed-off-by: Yun-Tang Hsu <[email protected]>
Signed-off-by: Yun-Tang Hsu <[email protected]>
Signed-off-by: Yun-Tang Hsu <[email protected]>
Signed-off-by: Arunkumar Velayutham <[email protected]>
In this commit, we: 1. Set a label to Perftest Pods before initiating traffic in each subtest to filter records in both the IPFIX collector Pod and the ClickHouse. 2. Remove the --since-time flag used during log retrieval from the IPFIX collector Pod in the e2e test. 3. Cease reliance on timestamps for record filtering due to potential time discrepancies between the testbed and Kubernetes nodes, which might hinder the retrieval of desired logs. Signed-off-by: Yun-Tang Hsu <[email protected]>
In this commit, we do:
1. Changed the order where we append expired records before exporting them from our exporter.
For inter-node traffic with egress/ingress np with action drop, we will receive records from PacketIn and the conntrack table. For the ingress case, there is no issue as we will receive the records from both nodes and these two records are both correlationRequired. For the egress case, the record from the conntrack table is correlationRequired and the record from PacketIn is not correlationRequired. If the record from the conntrack table arrive at the FA first, then the record will need to do correlation at FA. The ReadyToSend will be true forever as it keeps waiting to do correlation.
2. Add check to verify if Flow Exporters can successfully resolve the Flow Aggregator Service address before sending traffic.
3. Add check to verify if Flow Aggregator can successfully connect to the ClickHouse before sending traffic.
4. Add labels to External subtest to filter useless logs from the IPFIX collector Pod.
5. Confirm the correct addition of a label to a specific Pod after updating the Pod.
6. Remove the octetDeltaCount check and, instead, filter out all records with octetDeltaCount=0 when retrieving records from the IPFIX collector Pod and ClickHouse.
7. Use new image from go-ipfix. We improve the IPFIX collector by:
a. Disable printing records whenever we receive it. Instead, we store records in a string array.
b. Add http listener and handler to receive request to return or reset records.
In this way, we can reduce the retrieving log time from ~4s to ~80ms when we have ~1900 records
inside it.
Signed-off-by: Yun-Tang Hsu <[email protected]>
yuntanghsu
added
the
kind/cherry-pick
Signed-off-by: Yun-Tang Hsu <[email protected]>
yuntanghsu
force-pushed
the
automated-cherry-pick-of-#5532-#5592-#4252-#5731-#5770-upstream-release-1.14
branch
from
58c6172
to
88e165f
Compare
Signed-off-by: Yun-Tang Hsu <[email protected]>
|
Do we really need #5532? |
|
Yes. #5592 is using |
it's very localized, so we could just manually edit |
Yes, you can edit a PR manually when you cherry-pick. If there is a conflict, the script will ask you to resolve it manually in another terminal windows. |
|
replaced by #5886 |
yuntanghsu
deleted the
automated-cherry-pick-of-#5532-#5592-#4252-#5731-#5770-upstream-release-1.14
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry pick of #5532 #5592 #4252 #5731 #5770 on release-1.14.
#5532: Improvements to FlowExporter benchmarks
#5592: Remove redundant log in fillPodInfo/fillServiceInfo
#4252: e2e framework for Antrea's native secondary network
#5731: Enhance Records Filtering: Utilizing Labels In The E2E Test
#5770: Improve flow-visibility e2e test
For details on the cherry pick process, see the cherry pick requests page.