Skip to content

✨ Support serviceaccount pull secrets #2005

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

tmshort
Copy link
Contributor

@tmshort tmshort commented Jun 3, 2025

Serviceaccounts reference pull secrets!

  • Determine our serviceaccount (via the new internal/shared/util/sa package).
  • Use a common pull_secret_controller
  • Update the pull_secret_controller to know about the service account
  • Update the pull_secret_controller to watch the namespace-local secrets
  • Update caching to include sa, and use filters for additional secrets
  • Add RBAC to access these secrets and sa
  • Update writing the auth.json file to handle dockercfg and dockerconfigjson
  • Update writing the auth.json file to include multiple secrets

Description

Reviewer Checklist

  • API Go Documentation
  • Tests: Unit Tests (and E2E Tests, if appropriate)
  • Comprehensive Commit Messages
  • Links to related GitHub Issue(s)

@tmshort tmshort requested a review from a team as a code owner June 3, 2025 19:39
@openshift-ci openshift-ci bot requested a review from oceanc80 June 3, 2025 19:39
Copy link

netlify bot commented Jun 3, 2025

Deploy Preview for olmv1 ready!

Name Link
🔨 Latest commit f53a858
🔍 Latest deploy log https://app.netlify.com/projects/olmv1/deploys/684b42424d4a820008369179
😎 Deploy Preview https://deploy-preview-2005--olmv1.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@openshift-ci openshift-ci bot requested a review from OchiengEd June 3, 2025 19:39
@tmshort tmshort force-pushed the use-sa-pull-secret branch from d09d1b8 to 26246b6 Compare June 3, 2025 20:10
Copy link

codecov bot commented Jun 3, 2025

Codecov Report

Attention: Patch coverage is 66.25514% with 82 lines in your changes missing coverage. Please review.

Project coverage is 73.55%. Comparing base (8f81c23) to head (de52a82).
Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
...ernal/shared/controllers/pull_secret_controller.go 66.66% 38 Missing and 11 partials ⚠️
cmd/catalogd/main.go 54.16% 7 Missing and 4 partials ⚠️
cmd/operator-controller/main.go 54.16% 7 Missing and 4 partials ⚠️
...nal/shared/util/pullsecretcache/pullsecretcache.go 72.41% 7 Missing and 1 partial ⚠️
internal/shared/util/sa/serviceaccount.go 84.21% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2005      +/-   ##
==========================================
+ Coverage   69.17%   73.55%   +4.37%     
==========================================
  Files          79       80       +1     
  Lines        7037     7146     +109     
==========================================
+ Hits         4868     5256     +388     
+ Misses       1887     1565     -322     
- Partials      282      325      +43     
Flag Coverage Δ
e2e 44.10% <42.79%> (+1.09%) ⬆️
unit 59.82% <32.09%> (-0.24%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tmshort
Copy link
Contributor Author

tmshort commented Jun 4, 2025

/hold
Until I do some additional manual testing.

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 4, 2025
@tmshort
Copy link
Contributor Author

tmshort commented Jun 4, 2025

/approve

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 4, 2025
@tmshort tmshort force-pushed the use-sa-pull-secret branch 2 times, most recently from 11579bc to 1c0e4dd Compare June 4, 2025 18:05
@tmshort
Copy link
Contributor Author

tmshort commented Jun 4, 2025

/unhold

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 4, 2025
AuthFilePath string
}

func (r *PullSecretReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is req used anywhere other than logging? If not, I think I'd suggest dropping it (e.g. rename to _). And changing the logging to just generically say something like "reconciling pull secrets".

The name/namespace of the request without the type info might be a little confusing to show up in the log. But maybe we could log the events that pass our predicate in our predicate where we do have type information. That way there's still detail about what is causing our reconciler to be triggered?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly... the logger has a lot of additional information added to it, so you can tell what type of resource triggered the event, I wrapped the log below to make it easier to read:

I0606 18:43:42.641045       1 pull_secret_controller.go:113] "found secret"
  logger="pull-secret-reconciler"
  controller="service-account-controller"
  controllerGroup=""
  controllerKind="ServiceAccount"
  ServiceAccount="olmv1-system/operator-controller-controller-manager"
  reconcileID="26404229-a2d3-495d-86ee-0da390a1e8f4"
  name="pull-dockercfg"
  namespace="olmv1-system"

Copy link
Contributor Author

@tmshort tmshort Jun 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And when a pull secret is modified:

I0606 18:58:10.831129       1 pull_secret_controller.go:113] "found secret"
  logger="pull-secret-reconciler"
  controller="pull-secret-controller"
  controllerGroup=""
  controllerKind="Secret"
  Secret="olmv1-system/pull-dockercfg"
  reconcileID="a783ca58-0213-4fe3-a19d-3ed7ca21f5ae"
  name="pull-dockercfg"
  namespace="olmv1-system"

Comment on lines +113 to +141
_, err := ctrl.NewControllerManagedBy(mgr).
For(&corev1.Secret{}).
Named("pull-secret-controller").
WithEventFilter(newSecretPredicate(r)).
Build(r)
if err != nil {
return err
}

_, err = ctrl.NewControllerManagedBy(mgr).
For(&corev1.ServiceAccount{}).
Named("service-account-controller").
WithEventFilter(newNamespacedPredicate(r.ServiceAccountKey)).
Build(r)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I'd setup two separate controllers. IIRC, there's a way to have a single controller with multiple watches. You may need to drop down to the lower-level controller package though (can't remember if For is required).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBH, having two controllers actually seems the cleanest, in that we get different invocations that recognize ServiceAccount vs. Secret, and it's clear what triggered the reconcile (i.e. the logger clearly indicates what resource is the trigger). With a single controller, we have to somehow map the ServiceAccount to a set of Secrets (which we may not yet know about yet), which feels kinda kludgy. Otherwise, the Secret controller is getting a ServiceAccount for the request, rather than a Secret.

@tmshort
Copy link
Contributor Author

tmshort commented Jun 6, 2025

lint doesn't like my logging trick...

Comment on lines 254 to 253
setupLog.Error(err, "Unable to get pod namesapce and serviceaccount")
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
setupLog.Error(err, "Unable to get pod namesapce and serviceaccount")
return err
setupLog.Error(err, "Failed to extract namespace/serviceaccount from JWT token")
return fmt.Errorf("failed to get service account identity from token: %w", err)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a bit redundant to wrap the error, since it's also logged.

return err
}

setupLog.Info("Read token", "serviceaccount", saKey)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
setupLog.Info("Read token", "serviceaccount", saKey)
if saKey.Namespace == "" || saKey.Name == "" {
setupLog.Error(nil, "Extracted service account name or namespace is empty", "saKey", saKey)
return fmt.Errorf("invalid service account identity extracted: %v", saKey)
}
setupLog.Info("Successfully extracted serviceaccount identity from token", "namespace", saKey.Namespace, "name", saKey.Name)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no point in checking for an empty name or namespace, as the sautil.GetServiceAccount() function should (will) error out if both can't be retrieved.

return err
}

setupLog.Info("Read token", "serviceaccount", saKey)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The block below appears identical in both cmd/catalogd/main.go and cmd/operator-controller/main.go.
Could we move this logic into a helper function under pkg/shared/util/sa and call it from both main.go files? That would reduce duplication and help with future changes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, I can try to consolidate a bit.

Copy link
Contributor

@camilamacedo86 camilamacedo86 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few suggestions on possible improvements, mostly around structuring and reusability — nothing blocking though.

The overall implementation looks good for me.
Great work! 🎉

Otherwise, all good from my side — LGTM

@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. and removed lgtm Indicates that a PR is ready to be merged. labels Jun 12, 2025
@tmshort tmshort force-pushed the use-sa-pull-secret branch from 30cded3 to de52a82 Compare June 12, 2025 20:09
tmshort added 7 commits June 12, 2025 17:05
Serviceaccounts reference pull secrets!

* Determine our serviceaccount (via the new internal/shared/util/sa package).
* Use a common pull_secret_controller
* Update the pull_secret_controller to know about the service account
* Update the pull_secret_controller to watch the namespace-local secrets
* Update caching to include sa, and use filters for additional secrets
* Add RBAC to access these secrets and sa
* Update writing the auth.json file to handle dockercfg and dockerconfigjson
* Update writing the auth.json file to include multiple secrets

Signed-off-by: Todd Short <[email protected]>
@tmshort tmshort force-pushed the use-sa-pull-secret branch from de52a82 to 2144fad Compare June 12, 2025 21:05
@camilamacedo86
Copy link
Contributor

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 12, 2025
Copy link

openshift-ci bot commented Jun 12, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, joelanford, tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [camilamacedo86,joelanford,tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot openshift-merge-bot bot merged commit efc6657 into operator-framework:main Jun 12, 2025
20 checks passed
@tmshort tmshort deleted the use-sa-pull-secret branch June 12, 2025 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants