Making machine objects immutable makes managing LB's difficult

[mwoodson]

User Story

As an admin of kubernetes I would like to be able to manage the LB's from within kubernetes, similar to how it is managed in machine object.

Detailed Description

This is a concern I have heard, so I am raising it here. It's possible I am misunderstanding everything, as I am not a developer to this repo.

I have heard that machine objects are going to be made immutable. Currently we use the machine object to help manage load balancers in AWS. We update, delete, and add LB's to the loadBalncers list within the object. The way that the machine-api-controllers adds them to the right LB's is fantastic. We can rest assured that they are in the right place.

We also have the need to create new API endpoints via LB's. Again, managing these is a pretty straight forward task.

I am an Openshift v4 admin. In the current version, the machine object is mutable. I have been told that this will change going forward and this object will not be mutable.

By making this immutable, the process to simply add a machine to a LB, will become very hard. Deleting and recreating master nodes just to add them to have them added/removed to a LB is a bit drastic.

If machine objects are made immutable, I would ask that things like LB's, not associated with the actual machines, and things that admins would want to change, can be done in another way.

Goals

• 1. Be able to update/edit/delete machines from loadBalancers

Non-Goals

Anything else you would like to add:

Again, if this is my misunderstanding, please let me know. I'm happy to be corrected.

/kind feature

[detiber]

@mwoodson I think we probably need to clarify the immutability concerns a bit.

We want to ensure immutability wrt handling of the underlying VM instance, so we should not be doing anything that mutates that VM instance after creation.

We do not plan to extend that to the Machine resource itself, which should still allow for the behavior that you are referring to.

[michaelgugino]

My preference is to have a separate controller for provider-specifc loadbalancers to add and remove instances.

There's currently not a mechanism to remove a machine from a load balancer without deleting it.

[ncdc]

For clarity, there is no code in core Cluster API that does anything with load balancers. Do you have examples of what you're doing now to add machines to / remove machines from a load balancer?

[vincepri]

Possibly related with the external effort of a load balancer provider #1250

[michaelgugino]

Possibly related with the external effort of a load balancer provider #1250

I don't see why that issue was closed.

I think it's reasonable to have this discussion here because it's not specific to any particular provider, rather it's an architectural concern.

[ncdc]

That issue was closed mostly coming out of discussions from the September face to face meeting, where we discussed that a component that manages/provides a load balancer doesn't necessarily need to be just for Cluster API. @timothysc suggested doing a prototype.

It's totally fine to have the discussion here. Can you help us understand how you're doing things today? Also keep in mind what I wrote above - CAPI doesn't know anything about or do anything with load balancers.

[michaelgugino]

That issue was closed mostly coming out of discussions from the September face to face meeting, where we discussed that a component that manages/provides a load balancer doesn't necessarily need to be just for Cluster API. @timothysc suggested doing a prototype.

It's totally fine to have the discussion here. Can you help us understand how you're doing things today? Also keep in mind what I wrote above - CAPI doesn't know anything about or do anything with load balancers.

@ncdc
Here's what we are doing downstream with respect to LB's:
https://github.com/openshift/cluster-api-provider-aws/blob/release-4.1/pkg/apis/awsproviderconfig/v1beta1/awsmachineproviderconfig_types.go#L138

I was under the impression we inherited this from upstream, but possibly not.

Generally speaking, our model is 1) Something other than machine-controller has created an ELB 2) machine-controller adds instance to said ELB upon creation, removes it upon deletion.

I think longer term I'd like to see a provider specific controller that matches some data on a machine-object, and adds or removes those machines to a load balancer. This could be labels or annotations or something else.

[ncdc]

Ok, thanks for that. Given that core CAPI (Cluster, Machine, MachineSet, MachineDeployment) is entirely unaware of load balancers, is there anything in CAPI today that you think prevents or somehow inhibits working with load balancers in some way?

It sounds like maybe you might want to open some RFEs in provider repos, such as CAPA?

[michaelgugino]

Ok, thanks for that. Given that core CAPI (Cluster, Machine, MachineSet, MachineDeployment) is entirely unaware of load balancers, is there anything in CAPI today that you think prevents or somehow inhibits working with load balancers in some way?

It sounds like maybe you might want to open some RFEs in provider repos, such as CAPA?

Prevents? No, given what I know now. That said, I think it's still a useful exercise to describe a preferred or reference architecture that's not provider specific (even if the actual code is provider specific).

Probably more of an RFE, I agree.

[ncdc]

Does this apply to core Cluster API, though?

[moshloop]

https://docs.google.com/document/d/17Z_F_lmv4WgXaG9TaayOwwpCGRRoBxLbY070TSXDhvs/edit addresses this

[ncdc]

/close
in favor of #1250

[k8s-ci-robot]

@ncdc: Closing this issue.

In response to this:

/close
in favor of #1250

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.