Proof of concept: Use v3 CRDs directly

[caseydavenport]

Description

This is a very early stage WIP PoC for removing the Calico API server and instead using v3 APIs directly as CRDs. This is solely meant as a way to scope out the required work in advance of a design.

Why?

• Calico API server constant source of bugs and burden.
• Differences in CRD vs v3 API
• Importing k8s libraries
• Massive amount of code needed for new APIs
• hostNetwork bugs on EKS, AKS.
• Must be running before APIs are accessible, HA, etc.

Rough work:

• Generate new v3 CRDs instead of crd.pc.org/v1
• Switch libcalico-go to read those CRDs instead of v1
• Assumptions reworked: e.g., tier prefixes
• Skip / disable various translations
• Add extensive kubebuilder validation
• Add tier-based RBAC using webhook
• Add IP Pool deletion controller for dependent resources
• Add short names to CRDs
• Add new v3 CRDs for ipamhandles / ipamblocks

Related issues/PRs

Todos

• Tests
• Documentation
• Release note

Release Note

TBD

Reminder for the reviewer

Make sure that this PR has the correct labels and milestone set.

Every PR needs one docs-* label.

• docs-pr-required: This change requires a change to the documentation that has not been completed yet.
• docs-completed: This change has all necessary documentation completed.
• docs-not-required: This change has no user-facing impact and requires no docs.

Every PR needs one release-note-* label.

• release-note-required: This PR has user-facing changes. Most PRs should have this label.
• release-note-not-required: This PR has no user-facing changes.

Other optional labels:

• cherry-pick-candidate: This PR should be cherry-picked to an earlier release. For bug fixes only.
• needs-operator-pr: This PR is related to install and requires a corresponding change to the operator.