Can DeploymentSpec expose enableSchedulerExtender option?

[ijingo]

According to my knowledge, if I choose the "operator" style to deploy pmem-csi controller, there is no way initially to enable scheduler extender. To enable scheduler extender endpoint, I have to manually edit the cmd or args of the pod template in the Deployment resource created by the pmem-csi-operator.

I am wondering, is it possible to enable the scheduler extender at the beginning by including an option field in the DeploiymentSpec?

[pohly]

This is a valid observation.

We left out support for the scheduler extender in the initial operator release because there's really no good way how the operator could do all of the necessary setup work: deploying the driver is the easy part, but then you also need to reconfigure the Kubernetes scheduler (very dependent on the cluster!) and the apiserver (a bit easier).

@ijingo: if we add an "enable scheduler extensions" in the operator deployment options, will that be enough (i.e. can you then take care of reconfiguring the rest of the cluster)?

[ijingo]

@ijingo: if we add an "enable scheduler extensions" in the operator deployment options, will that be enough (i.e. can you then take care of reconfiguring the rest of the cluster)?

It will be good enough.
And in my opinion, it introduces no side-effect. If the scheduler extender is enabled for registry pod, it just binds a port. Without reconfiguration of the Kubernetes cluster, it would just work normally.

[pohly]

And in my opinion, it introduces no side-effect.

Correct. We therefore might even enable it by default.

[pohly]

Didn't make it into 0.8. Let's push it to 0.9.

[pohly]

I'm working on this.

One conclusion from a discussion with @avalluri was that the operator should at least be able to create the mutating pod webhook configuration if requested in the CRD. The TLS certificates are meant to be provided in a secret by the user.

This leads to some corner cases:

• a missing secret must be detected early because the operator must read it (one value must be copied into the webhook config)
• the operator needs permission to read secrets
• an updated secret must a) lead to an updated webhook config and b) be re-loaded by the webhook once the volume with the secret is updated.

[pohly]

@ijingo PR #838 added a new API for this:

pmem-csi/pkg/apis/pmemcsi/v1beta1/deployment_types.go

Lines 97 to 105 in 1d141c6

	// ControllerTLSSecret is the name of a secret which contains ca.crt, tls.crt and tls.key data
	// for the scheduler extender and pod mutation webhook. A controller is started if (and only if)
	// this secret is specified.
	ControllerTLSSecret string `json:"controllerTLSSecret,omitempty"`
	// MutatePod defines how a mutating pod webhook is configured if a controller
	// is started. The field is ignored if the controller is not enabled.
	// The default is "Try".
	// +kubebuilder:validation:Enum=Always;Try;Never
	MutatePods MutatePods `json:"mutatePods,omitempty"`

Do you perhaps have time to try this out with the current "canary" image of the operator? The updated version with the new API should be available tomorrow.

[pohly]

I think this works, let's close the issue even without explicit confirmation.