Issue 3602 pod network bandwidth usage action #3621

okolesn · 2024-07-30T07:00:22Z

…tion # Conflicts: # api/src/main/java/com/epam/pipeline/manager/cluster/performancemonitoring/ResourceMonitoringManager.java

SilinPavel · 2024-07-31T08:31:32Z

api/src/main/java/com/epam/pipeline/controller/pipeline/PipelineRunController.java

@@ -595,4 +595,18 @@ public Result<List<RunInfo>> loadRunsByParentId(@PathVariable(RUN_ID) final Long
    public Result<RunChartInfo> loadActiveRunsCharts(@RequestBody final RunChartFilterVO filter) {
        return Result.success(runApiService.loadActiveRunsCharts(filter));
    }
+
+    @PostMapping("/run/{runId}/network")


lets rename it to /run/{runId}/network/limit to get instant sense what this method does

SilinPavel · 2024-07-31T08:32:44Z

api/src/main/java/com/epam/pipeline/controller/pipeline/PipelineRunController.java

+            produces = MediaType.APPLICATION_JSON_VALUE)
+    @ApiResponses(value = {@ApiResponse(code = HTTP_STATUS_OK, message = API_STATUS_DESCRIPTION)})
+    public Result<Boolean> setLimitBoundary(@PathVariable(value = RUN_ID) final Long runId,
+                                         @RequestParam(defaultValue = "true") final Boolean limit,


'limit' -> 'enable' since we will have limit in the name of the method. Don't forget to change @ApiOperation notes

SilinPavel · 2024-07-31T08:35:53Z

.../java/com/epam/pipeline/manager/cluster/performancemonitoring/ResourceMonitoringManager.java

@@ -489,7 +491,9 @@ private void processHighNetworkConsumingRuns(Map<String, PipelineRun> running,
                    if (bandwidth >= bandwidthLimit) {
                        processHighNetworkConsumingRun(run, actionTimeout, action, runsToNotify,
                                runsToUpdateNotificationTime, bandwidth, runsToUpdateTags);
-                    } else if (run.getLastNetworkConsumptionNotificationTime() != null) {
+                    } else if (run.getLastNetworkConsumptionNotificationTime() != null &&
+                            !run.getTags().containsKey(getNetworkLimitDateTag()) &&


do we even need to change this code? it seems that logic for limiting and defining if run under network pressure should be separate

SilinPavel · 2024-07-31T08:42:45Z

api/src/main/java/com/epam/pipeline/manager/pipeline/BandwidthMonitoringServiceCore.java

+            if (shouldLimit(tags)) {
+                boundary = Integer.parseInt(tags.get(NETWORK_LIMIT));
+                dockerContainerOperationManager.limitNetworkBandwidth(run, boundary, true);
+                run.addTag(networkLimitDateTag, DateUtils.nowUTCStr());
+            } else if (shouldCleanLimit(tags)) {
+                dockerContainerOperationManager.limitNetworkBandwidth(run, 0, false);
+                run.removeTag(NETWORK_CONSUMING_LEVEL_HIGH);
+                run.removeTag(NETWORK_LIMIT);
+                run.removeTag(networkLimitDateTag);
+            } else {
+                return;
+            }


it seems that there is a problem. If you define shouldCleanLimit as !shouldLimit it means that else is unreachable. But we actually want to:

set limit if there is NETWORK_LIMIT tag but not networkLimitDateTag

unset limit if there is networkLimitDateTag but not NETWORK_LIMIT

do not do anything as default action

SilinPavel · 2024-07-31T08:45:36Z

scripts/autoscaling/init_multicloud_v1.15.4.sh

+cd /bin
+git clone https://github.com/magnific0/wondershaper.git
+cd wondershaper
+sudo make install


rewrite to

cd /bin && \ git clone https://github.com/magnific0/wondershaper.git && \ cd wondershaper && \ sudo make install

at the end also check if installation was successful and print log about the result

SilinPavel · 2024-07-31T08:47:10Z

scripts/commit-run-scripts/limit_bandwidth.sh

+    if [ "$i" != "lo" ]; then
+      link_num=$(sudo docker exec -it $container_id cat /sys/class/net/$i/iflink | tr -d '\r')
+      interface_id=$(sudo ip ad | grep "^$link_num:" | awk -F ': ' '{print $2}' | awk -F '@' '{print $1}')
+      if [ ! -z $interface_id ]; then


would be good to warn if interface_id can't be found for i

also please wrap $interface_id with ""

SilinPavel · 2024-08-01T06:51:58Z

scripts/commit-run-scripts/limit_bandwidth.sh

+
+set -x
+
+interfaces=$(sudo docker exec -it $container_id ls /sys/class/net | tr -d '\r')


let's check result of the command and if it's != 0 warn to the run and exit

SilinPavel · 2024-08-01T06:54:20Z

scripts/commit-run-scripts/limit_bandwidth.sh

+#          download_rate and upload_rate in kilobits per second
+          result=$($command -a $interface_id -d $download_rate -u $upload_rate 2>&1)
+          if [ ! -z "$result" ]; then
+            pipe_log_warn "[WARN] Failed to apply bandwidth limiting"


to write to the run log you need to have its RUN_ID and TASK_NAME in the env, but you doesn't. Please see how it is done in commit run. Basically we pass RUN_ID wher launch a command

SilinPavel · 2024-08-01T06:55:25Z

api/src/main/java/com/epam/pipeline/manager/docker/DockerContainerOperationManager.java

+            Assert.notNull(containerId,
+                    messageHelper.getMessage(MessageConstants.ERROR_CONTAINER_ID_FOR_RUN_NOT_FOUND, run.getId()));
+            final int boundaryKBitsPerSec = limit ? boundary * 8 / BYTES_IN_KB : 0;
+            final String getSizeCommand = LimitBandwidthCommand.builder()


We also should pass runId to the script, because script will need it

…le to write logs into run

…etwork.bandwidth.command.timeout

…script to be more resilient and atomic

…work_bandwidth_usage_action # Conflicts: # api/src/main/java/com/epam/pipeline/acl/run/RunApiService.java # api/src/main/java/com/epam/pipeline/controller/pipeline/PipelineRunController.java

(cherry picked from commit 9d77b53)

okolesn added 2 commits July 19, 2024 14:28

Issue 3591 Pod nework bandwidth usage fixes

232e556

Issue 3591 Pod network bandwidth usage limitation

6ec0a9c

okolesn requested a review from SilinPavel July 30, 2024 07:00

okolesn added 2 commits July 30, 2024 10:04

Merge branch 'develop' into Issue_3602_Pod_network_bandwidth_usage_ac…

74ab199

…tion # Conflicts: # api/src/main/java/com/epam/pipeline/manager/cluster/performancemonitoring/ResourceMonitoringManager.java

Issue 3591 Pod network bandwidth usage limitation

c88ce38

SilinPavel requested changes Jul 31, 2024

View reviewed changes

SilinPavel reviewed Aug 1, 2024

View reviewed changes

okolesn and others added 7 commits August 1, 2024 12:43

Issue 3591 Pod network bandwidth usage limitation - fix remarks

0545fba

issue 3602 added common_utils.sh usage in limit_bandwidth.sh to be ab…

37f40e1

…le to write logs into run

issue 3602 rename sys pref limit.network.bandwidth.timeout -> limit.n…

b8741e0

…etwork.bandwidth.command.timeout

issue 3602 rollback for util script, incorporate all function in the …

c7eaf5b

…script to be more resilient and atomic

Merge remote-tracking branch 'origin/develop' into Issue_3602_Pod_net…

05000c3

…work_bandwidth_usage_action # Conflicts: # api/src/main/java/com/epam/pipeline/acl/run/RunApiService.java # api/src/main/java/com/epam/pipeline/controller/pipeline/PipelineRunController.java

fixes

7c13cdf

fixes

699f564

SilinPavel force-pushed the Issue_3602_Pod_network_bandwidth_usage_action branch from 23f3f9d to 699f564 Compare August 21, 2024 12:46

SilinPavel merged commit 9d77b53 into develop Aug 21, 2024
4 checks passed

SilinPavel pushed a commit that referenced this pull request Aug 28, 2024

Issue 3602 pod network bandwidth usage action (#3621)

d3b9e6f

(cherry picked from commit 9d77b53)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue 3602 pod network bandwidth usage action #3621

Issue 3602 pod network bandwidth usage action #3621

okolesn commented Jul 30, 2024

SilinPavel Jul 31, 2024

SilinPavel Jul 31, 2024

SilinPavel Jul 31, 2024

SilinPavel Jul 31, 2024

SilinPavel Jul 31, 2024

SilinPavel Jul 31, 2024

SilinPavel Aug 1, 2024

SilinPavel Aug 1, 2024

SilinPavel Aug 1, 2024

SilinPavel Aug 1, 2024


		set -x

		interfaces=$(sudo docker exec -it $container_id ls /sys/class/net \| tr -d '\r')

Issue 3602 pod network bandwidth usage action #3621

Issue 3602 pod network bandwidth usage action #3621

Conversation

okolesn commented Jul 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment