Add gitlab-runner-scaler #6499

fira42073 · 2025-01-25T21:23:14Z

Provide a description of what has been changed

Checklist

When introducing a new scaler, I agree with the scaling governance policy
I have verified that my change is according to the deprecations & breaking changes policy
Tests have been added
Changelog has been updated and is aligned with our changelog requirements
A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
A PR is opened to update the documentation on (repo) (if applicable)
Commits are signed with Developer Certificate of Origin (DCO - learn more)

Signed-off-by: Fira Curie <[email protected]>

JorTurFer · 2025-02-03T21:20:13Z

pkg/scalers/gitlab_runner_scaler_test.go

+	meta := &gitlabRunnerMetadata{
+		ProjectID:                 "12345",
+		TargetPipelineQueueLength: 5,
+		TriggerIndex:              0,


It's something quite small but, could we use TriggerIndex: 1 ? 0 is the default value so just using 1 we can also check that the TriggerIndex is correctly reflected in the metric name

JorTurFer · 2025-02-03T21:28:45Z

tests/scalers/gitlab_runner/gitlab_runner_test.go

@@ -0,0 +1,503 @@
+//go:build e2e


Reading the e2e test, it waits a PAT so I guess that we need to create an e2e account to run it. Does it have any special requirement? Does it make sense to use gitlab.com instead of spin up a dummy gitlab instance in the cluster during the e2e test? https://docs.gitlab.com/charts/installation/deployment.html

I actually already tried doing that, and it's a bit too complicated
#5616 (comment)

I think it's best to create a gitlab.com account, since it would only require phone verification to run pipelines, and that's it. Repos will be cleaned up automatically as well as part of e2e tests.

JorTurFer · 2025-02-03T21:31:33Z

tests/scalers/gitlab_runner/gitlab_runner_test.go

+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{.DeploymentName}}
+  namespace: {{.TestNamespace}}
+  labels:
+    app: gitlab-runner
+spec:
+  replicas: 0
+  selector:
+    matchLabels:
+      app: gitlab-runner
+  template:
+    metadata:
+      labels:
+        app: gitlab-runner
+    spec:
+      terminationGracePeriodSeconds: 90
+      containers:
+      - name: gitlab-runner
+        image: busybox
+        command: ["/bin/sh", "-c", "trap 'echo SIGTERM received; exit 0' SIGTERM; while true; do sleep 30; done"]


Does it make sense creating a real runner using their runner image to check that the job is dequeued? https://hub.docker.com/r/gitlab/gitlab-runner/tags

I sat down and tried doing that this evening, but then realized that we actually already kind of had this discussion here.

Gitlab already does have a mechanism of autoscaling (creating runner per job)

don't worry about the token in the image, it's been rotated already.

So I think doesn't make sense to manage that using keda. Again, please correct me if I'm wrong, maybe I simply don't quite understand the usecase here.

JorTurFer · 2025-02-03T21:32:59Z

tests/scalers/gitlab_runner/gitlab_runner_test.go

+	testSONotActivated(t, kc)
+
+	testSOScaleOut(t, kc, projectID)
+	testSOScaleIn(t, kc, projectID)


Let's use the same name as the majority of the tests:

Suggested change

testSONotActivated(t, kc)

testSOScaleOut(t, kc, projectID)

testSOScaleIn(t, kc, projectID)

testActivation(t, kc)

testScaleOut(t, kc, projectID)

testScaleIn(t, kc, projectID)

JorTurFer · 2025-02-03T21:35:08Z

pkg/scalers/gitlab_runner_scaler.go

+
+	metric := GenerateMetricInMili(metricName, float64(queueLen))
+
+	return []external_metrics.ExternalMetricValue{metric}, queueLen >= s.metadata.TargetPipelineQueueLength, nil


TargetPipelineQueueLength shouldn't be used here. It is used for the target, for the activation we need to include another parameter ActivationTargetPipelineQueueLength with default value 0 and use it with >

return []external_metrics.ExternalMetricValue{metric}, queueLen > s.metadata.ActivationTargetPipelineQueueLength, nil

JorTurFer · 2025-02-03T21:35:50Z

tests/scalers/gitlab_runner/gitlab_runner_test.go

+      gitlabAPIURL: {{.GitlabAPIURL}}
+      projectID: "{{.ProjectID}}"
+      targetPipelineQueueLength: "{{.TargetPipelineQueueLength}}"
+      labels: "e2eSOtester"


The activation parameter is left here for the activation test

JorTurFer · 2025-02-03T21:36:45Z

tests/scalers/gitlab_runner/gitlab_runner_test.go

+
+func testSONotActivated(t *testing.T, kc *kubernetes.Clientset) {


During the activation test, we have to add an item in the queue and check that the activation value blocks the scaling until the item count in the queue is higher

pkg/scalers/gitlab_runner_scaler.go

zroubalik · 2025-02-06T10:47:03Z

pkg/scalers/scalersconfig/typed_config.go

@@ -295,6 +295,22 @@ func setConfigValueURLParams(params Params, valFromConfig string, field reflect.
 	return nil
 }

+// setConfigValueURL is a function that sets the value of the url.URL field
+func setConfigValueURL(valFromConfig string, field reflect.Value) error {


WDYT @wozniakjan ?

Signed-off-by: Fira Curie <[email protected]>

zroubalik · 2025-03-26T19:15:44Z

/run-e2e gitlab
Update: You can check the progress here

fira42073 · 2025-03-26T19:59:30Z

We need gitlab PAT for this to work, have you had a chance to add it?

zroubalik · 2025-03-26T20:30:48Z

We need gitlab PAT for this to work, have you had a chance to add it?

I see, nope. We need to work on that, adding this #6499 (comment) for reference, thanks for letting us know!

fira42073 requested a review from a team as a code owner January 25, 2025 21:23

fira42073 force-pushed the gitlab-runner-scaler branch 2 times, most recently from 1abaa5f to 7af19df Compare January 25, 2025 21:25

fira42073 mentioned this pull request Jan 25, 2025

GitLab Runner Scaler #5616

Open

fira42073 force-pushed the gitlab-runner-scaler branch from 7af19df to b58aaae Compare January 25, 2025 21:42

Add gitlab-runner-scaler

b650bc3

Signed-off-by: Fira Curie <[email protected]>

fira42073 force-pushed the gitlab-runner-scaler branch from b58aaae to b650bc3 Compare January 25, 2025 22:02

JorTurFer reviewed Feb 3, 2025

View reviewed changes

zroubalik reviewed Feb 6, 2025

View reviewed changes

fira42073 and others added 2 commits March 4, 2025 21:16

Merge branch 'kedacore:main' into gitlab-runner-scaler

9503900

apply code review suggestions

a2ccae7

Signed-off-by: Fira Curie <[email protected]>

fira42073 force-pushed the gitlab-runner-scaler branch from 9f9ad27 to a2ccae7 Compare March 4, 2025 22:03

make use of activation target

f92b18c

Signed-off-by: Fira Curie <[email protected]>

fira42073 force-pushed the gitlab-runner-scaler branch from 3fc095e to f92b18c Compare March 4, 2025 22:26

fira42073 requested review from JorTurFer and zroubalik March 4, 2025 22:26

zroubalik mentioned this pull request Mar 26, 2025

Release: 2.17.0 #6502

Open

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add gitlab-runner-scaler #6499

Add gitlab-runner-scaler #6499

fira42073 commented Jan 25, 2025 •

edited

Loading

JorTurFer Feb 3, 2025

JorTurFer Feb 3, 2025

fira42073 Feb 13, 2025

JorTurFer Feb 3, 2025

fira42073 Mar 4, 2025

JorTurFer Feb 3, 2025

JorTurFer Feb 3, 2025

JorTurFer Feb 3, 2025

JorTurFer Feb 3, 2025

zroubalik Feb 6, 2025

zroubalik commented Mar 26, 2025 •

edited by github-actions bot

Loading

fira42073 commented Mar 26, 2025

zroubalik commented Mar 26, 2025


		metric := GenerateMetricInMili(metricName, float64(queueLen))

		return []external_metrics.ExternalMetricValue{metric}, queueLen >= s.metadata.TargetPipelineQueueLength, nil


		func testSONotActivated(t testing.T, kc kubernetes.Clientset) {

Add gitlab-runner-scaler #6499

Are you sure you want to change the base?

Add gitlab-runner-scaler #6499

Conversation

fira42073 commented Jan 25, 2025 • edited Loading

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zroubalik commented Mar 26, 2025 • edited by github-actions bot Loading

fira42073 commented Mar 26, 2025

zroubalik commented Mar 26, 2025

fira42073 commented Jan 25, 2025 •

edited

Loading

zroubalik commented Mar 26, 2025 •

edited by github-actions bot

Loading