Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: eventsource: deterministic volume/volumeMount order #2811

Merged
merged 9 commits into from
Sep 25, 2023

Conversation

popsu
Copy link
Contributor

@popsu popsu commented Sep 18, 2023

The order was non-deterministic earlier and this caused the controller to change the deployment on every reconcile until it the order happened to be same twice in a row.

Fixes: #2810

I've tested this by making the change to v1.7.1 tag (since we were running v1.7.1 when I made the fix), building the image, and deploying it and the issue went away.

Checklist:

@popsu
Copy link
Contributor Author

popsu commented Sep 18, 2023

Let me know if there's something missing, first time doing PR here 🙇

@popsu popsu force-pushed the eventsource-deterministic-volume-order branch from f98c832 to 40d9ba0 Compare September 18, 2023 14:03
@popsu
Copy link
Contributor Author

popsu commented Sep 18, 2023

https://github.com/argoproj/argo-events/blob/v1.8.1/common/util.go#L395-L417

I wonder if I should make the change in these functions instead? I don't know the codebase, so not sure at what level should the fix be made.

The order was non-deterministic earlier and this caused
the controller to change the deployment on every reconcile
until it the order happened to be same twice in a row.

Signed-off-by: Tommi Hovi <[email protected]>
This test is to verify that the order of volume and volumemounts.
Without the sort that was added in the previous commit, the order
is not deterministic and this test would thus fail some of the time
without sorting.

Signed-off-by: Tommi Hovi <[email protected]>
@popsu popsu force-pushed the eventsource-deterministic-volume-order branch from 425ed76 to a6cefa0 Compare September 20, 2023 08:26
@whynowy
Copy link
Member

whynowy commented Sep 25, 2023

@popsu - thanks for submitting the PR! Could you also add the same logic in the sensor controller?

@popsu popsu force-pushed the eventsource-deterministic-volume-order branch from 93744c2 to 786516a Compare September 25, 2023 10:48
@popsu
Copy link
Contributor Author

popsu commented Sep 25, 2023

Edit: seems that I don't need to run the make codegen, so this can be ignored.

I tried adding make codegen, but it failed:

2023/09/25 14:07:48 google/protobuf/descriptor.proto: File not found.                                                                                                                                                                                                                     
github.com/gogo/protobuf/gogoproto/gogo.proto:32:1: Import "google/protobuf/descriptor.proto" was not found or had errors.                                                                                                                                                                
github.com/gogo/protobuf/gogoproto/gogo.proto:38:8: "google.protobuf.EnumOptions" is not defined.                                                                                                                                                                                         
github.com/gogo/protobuf/gogoproto/gogo.proto: "google.protobuf.EnumOptions" is not defined.    
[... similar rows sniped...]
github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/intstr/generated.proto:23:1: Import "github.com/gogo/protobuf/gogoproto/gogo.proto" was not found or had errors.
2023/09/25 14:07:48 protoc -I . -I /tmp/tmp.evfHAmKZF6/src -I ./vendor --gogo_out=/tmp/tmp.evfHAmKZF6/src /tmp/tmp.evfHAmKZF6/src/github.com/argoproj/argo-events/vendor/k8s.io/apimachinery/pkg/util/intstr/generated.proto
2023/09/25 14:07:48 Unable to generate protoc on k8s.io.apimachinery.pkg.util.intstr: exit status 1
Makefile:135: recipe for target 'codegen' failed
make: *** [codegen] Error                                 

I installed protoc from https://github.com/protocolbuffers/protobuf/releases. I'm using Linux so didn't use brew.

Signed-off-by: Tommi Hovi <[email protected]>
Signed-off-by: Tommi Hovi <[email protected]>
@popsu
Copy link
Contributor Author

popsu commented Sep 25, 2023

✔️ added the same sorting for Sensor deployment.

I'm not sure if the Sensor spec has any place (I didn't find with quick look) where the same bug could occur. Since the bug in EventSource case was due to storing webhooks (and other eventsources) in a Map and then when creating volume/volumemounts for the secrets the code iterates through the map and in Go the order is arbitrary.

If there is similar place in Sensor then the testcase I added should be modified (like the one I added for EventSource has.

Copy link
Member

@whynowy whynowy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@whynowy whynowy merged commit 5b8c754 into argoproj:master Sep 25, 2023
8 checks passed
joelcomp1 pushed a commit to joelcomp1/argo-events that referenced this pull request Sep 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Controller keeps updating eventsource pod deployment thousands of times instead of just once
2 participants