Add Packet CCM to Packet Lokomotive clusters #548

iaguis · 2020-06-03T13:44:28Z

We should add the Packet CCM to Lokomotive clusters deployed to Packet.

The CCM handles, for example, node changes and is gonna be needed to use the peer autodiscovery feature of MetalLB once it gets merged (we're currently using a patched version of MetalLB to have the feature).

invidian · 2020-06-03T13:53:50Z

When implemented, it will also hit #478 😞

iaguis · 2020-06-03T15:10:24Z

Keep in mind this comment about things needed to bootstrap the CCM, since it'll probably apply to the Packet CCM too.

knrt10 · 2020-09-17T06:34:35Z

Looking into this

johananl · 2020-09-18T11:59:28Z

https://github.com/packethost/packet-ccm#bgp

The Packet CCM enables BGP for the project and enables it on all nodes as they come up.

@knrt10 FYI ^. We should verify that users can still decide not to enable BGP for a worker pool.

knrt10 · 2020-11-02T14:58:46Z

Current status:

v1.1.0 for packet-ccm is buggy and it does not contain MetalLb in the release. It is expected to be fixed in a week.

I tried manually deploying ccm with current master. We had to deploy Lokomotive's MetalLb component for ccm to work. ccm watches and picks up the configMap config from metallb-system namespace. The problems that I faced are:

Once ccm deployment is deployed on cluster, you have to apply MetalLb component. If component is re-applied, ccm does not pick configMap correctly and throws error that configMap is not correct, which should not happen. You have to delete the pod manually to get it to work again.

failed to update and sync nodes: failed to get metallb config map: could not parse config: yaml: unmarshal errors:
  line 1: field peer-autodiscovery not found in type metallb.ConfigFile

BGP is automatically enabled for all nodes. We have to find a way for users it disable it.
Once we deploy Lokomotive's MetalLb component, you have to manually apply Elastic IP to worker nodes on Packet console.

These things should be fixed, once packet-ccm fix in their patch release

- Add helm chart for packet-ccm. - Add varibales api_token and project_id to bootkube. - Run packet-ccm while bootstrapping. closes: #548 Signed-off-by: knrt10 <[email protected]>

- Add helm chart for packet-ccm. - Add varibales api_key and project_id to bootkube. - Run packet-ccm while bootstrapping. closes: #548 Signed-off-by: knrt10 <[email protected]>

- Add helm chart for packet-ccm. - Add varibales auth_token and project_id to bootkube. - Run packet-ccm while bootstrapping. closes: #548 Signed-off-by: knrt10 <[email protected]>

- Add helm chart for packet-ccm. - Run packet-ccm while bootstrapping. closes: #548 Signed-off-by: knrt10 <[email protected]>

- Add helm chart for packet-ccm. - Run packet-ccm while bootstrapping. - Export field auth_token to packet. This is required for packet-ccm. packet-ccm needs secret to be configured before it is applied. In that secret 2 values are required, one is packet project_id and other is api_key. We were already getting project_id and auth_token from user. closes: #548 Signed-off-by: knrt10 <[email protected]>

Which are set by Packet CCM right now. Refs #548 Signed-off-by: Mateusz Gozdek <[email protected]>

This commit enables CCM for Packet platform to allow dropping hacky way of labeling Node objects on Packet via Ignition script for MetalLB autodiscovery to work, which we are doing right now. Closes #548 Signed-off-by: Mateusz Gozdek <[email protected]>

Using CCM on Kubernetes requires --cloud-provider to be set to "external". This commit enables this customization. Default value remains empty string, so no flag will be set for compatibility with other platforms. Refs #548 Signed-off-by: knrt10 <[email protected]>

This commit add packet-ccm Helm chart based on manifests from https://github.com/packethost/packet-ccm/tree/master/deploy/template, but with Lokomotive-specific modifications like using image with BGP Node Selector support, enabled leader election for HA, removed unnecessary RBAC grants etc. This chart will be used to enable CCM on Packet platform. Refs #548 Signed-off-by: Mateusz Gozdek <[email protected]>

So we don't hit a regression when we move the responsibility of enabling BGP for nodes from Terraform to Packet CCM. Refs #548 Signed-off-by: Mateusz Gozdek <[email protected]>