Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: sync controller #167

Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
b30649c
refactor: use annotation for replicas retention
gary-lgy Jul 17, 2023
d06901d
refactor: sync controller
gary-lgy Jul 18, 2023
5b61310
chore: remove unused fedobj condition reasons
gary-lgy Jul 18, 2023
35c7207
fix: bugs introduced during sync controller refactor
gary-lgy Jul 18, 2023
c502ca2
refactor: decouple sync from generic client
gary-lgy Jul 19, 2023
ec43281
refactor: propagated version manager constructors
gary-lgy Jul 19, 2023
ec1de65
docs: propagated version description
gary-lgy Jul 19, 2023
b002e47
fix(sync): enqueue with delay on cluster readiness transition
gary-lgy Jul 19, 2023
77a0eaf
chore(sync): adapt to arch changes
gary-lgy Jul 20, 2023
98cee80
fix: make federate controller compilable
gary-lgy Jul 20, 2023
45f4c46
fix: handle clusters marked for deletion during downtime
gary-lgy Jul 20, 2023
1e4f10b
fix: list PropagatedVersions with rv=0
gary-lgy Jul 20, 2023
b446e2b
chore: remove create-crd-for-ftc flag
gary-lgy Jul 21, 2023
a4a0a99
fix(sync): wait for fedObject informer sync
gary-lgy Jul 21, 2023
fdfffa8
fix(sync): include target name in start reconcile log
gary-lgy Jul 21, 2023
36c4fe6
chore: standardise logging levels in verison manager
gary-lgy Jul 21, 2023
9e26bbe
fix(sync): nil deref in setFederatedStatus
gary-lgy Jul 21, 2023
47a5423
fix(sync): empty GVK for federated object
gary-lgy Jul 21, 2023
4b258c4
fix(sync): cluster enqueue conditions
gary-lgy Jul 22, 2023
636db49
fix(sync): error handling for getting from cluster store
gary-lgy Jul 22, 2023
15161df
fix(sync): error formatting
gary-lgy Jul 22, 2023
1c34a1c
fix(sync): wait for ftcmanager cache sync
gary-lgy Jul 22, 2023
f3961ac
fix(sync): retry when GetResourceFTC fails for cluster
gary-lgy Jul 22, 2023
9927a61
chore(ftc): remove unused fields
gary-lgy Jul 22, 2023
3e68e33
fix(sync): enqueue all fo and cfo on ftc change
gary-lgy Jul 22, 2023
d21907f
fix(queue): add missing Done after Get
gary-lgy Jul 22, 2023
ae7c11b
fix(informermanager): handle DeletedFinalStateUnknown
gary-lgy Jul 22, 2023
396f752
fix(sync): trigger cascading deletion when cluster marked for deletion
gary-lgy Jul 22, 2023
f6eed95
chore(sync): use memberObjectEnqueueDelay
gary-lgy Jul 27, 2023
e0e36de
fix: make override controller compilable
gary-lgy Jul 27, 2023
4af71c6
fix(nsautoprop): make it compilable
gary-lgy Jul 27, 2023
687789d
fix(status): make it compilable
gary-lgy Jul 27, 2023
cf14752
fix(sync): revert deletion of source-generation annotation
gary-lgy Jul 27, 2023
3fed307
chore(status): remove stale structs
gary-lgy Jul 27, 2023
4ecd638
chore: fix wrong import
gary-lgy Jul 27, 2023
fa99420
fix(sync): nil pointer dereference
gary-lgy Jul 27, 2023
fd3e825
fix(override): nil pointer derefenrence
gary-lgy Jul 27, 2023
a1afa06
fix(sync): retain replicas test
gary-lgy Jul 27, 2023
46894c6
fix(overrides): tests
gary-lgy Jul 27, 2023
1f1a6e8
chore: remove remnants of KubeFed in comments
gary-lgy Jul 27, 2023
0ffcd27
fix: import grouping
gary-lgy Jul 31, 2023
0939795
fix(versionmanager): remove redundant ok check
gary-lgy Jul 31, 2023
e42029d
chore: add comments to VersionManager
gary-lgy Jul 31, 2023
da2e3ca
chore(versionmanager): add comments to Get
gary-lgy Jul 31, 2023
fae54e5
chore: FederatedResource comments
gary-lgy Jul 31, 2023
efb2538
refactor(sync): FederatedResource interface implements VersionedResource
gary-lgy Jul 31, 2023
9aba3b8
chore(sync): add comments to FederatedResource methods
gary-lgy Jul 31, 2023
d94304e
chore(sync): update FederatedResourceAccessor comments
gary-lgy Jul 31, 2023
1c35b06
chore: remove unused pkg/util/meta functions and tests
gary-lgy Jul 31, 2023
f26cf0f
refactor: move pkg/util/propagatedversion to pkg/controllers/sync/pro…
gary-lgy Jul 31, 2023
a37b47f
chore(sync): fix import formatting
gary-lgy Jul 31, 2023
3f62427
refactor(sync): avoid redundant get ftc from ftcmanager
gary-lgy Jul 31, 2023
cb1393d
refactor(sync): rename methods for clarity
gary-lgy Jul 31, 2023
e1f722a
fix(scheduler): make it compilable
gary-lgy Jul 31, 2023
3bc1119
fix: compile errors
gary-lgy Jul 31, 2023
9fcd4ee
fix(util): broken test
gary-lgy Jul 31, 2023
eff8b64
feat: forbid overriding invalid paths
gary-lgy Jul 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ all: build
# make build BUILD_PLATFORMS=linux/amd64,darwin/amd64
.PHONY: build
build:
BUILD_FLAGS=$(BUILD_FLAGS) TARGET_NAME=$(TARGET_NAME) GOPROXY=$(GOPROXY) bash hack/make-rules/build.sh
BUILD_FLAGS="$(BUILD_FLAGS)" TARGET_NAME="$(TARGET_NAME)" GOPROXY="$(GOPROXY)" bash hack/make-rules/build.sh

# Start a local kubeadmiral cluster for developers.
#
Expand Down
2 changes: 2 additions & 0 deletions cmd/controller-manager/app/controllermanager.go
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ const (
NamespaceAutoPropagationControllerName = "nsautoprop"
StatusControllerName = "status"
SchedulerName = "scheduler"
SyncControllerName = "sync"
)

var knownControllers = map[string]controllermanager.StartControllerFunc{
Expand All @@ -53,6 +54,7 @@ var knownControllers = map[string]controllermanager.StartControllerFunc{
StatusControllerName: startStatusController,
FederatedClusterControllerName: startFederatedClusterController,
SchedulerName: startScheduler,
SyncControllerName: startSyncController,
}

var controllersDisabledByDefault = sets.New[string]()
Expand Down
34 changes: 32 additions & 2 deletions cmd/controller-manager/app/core.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,13 @@ import (
"github.com/kubewharf/kubeadmiral/pkg/controllermanager"
controllercontext "github.com/kubewharf/kubeadmiral/pkg/controllers/context"
"github.com/kubewharf/kubeadmiral/pkg/controllers/federate"
"github.com/kubewharf/kubeadmiral/pkg/controllers/federatedcluster"
"github.com/kubewharf/kubeadmiral/pkg/controllers/nsautoprop"
"github.com/kubewharf/kubeadmiral/pkg/controllers/override"
"github.com/kubewharf/kubeadmiral/pkg/controllers/policyrc"
"github.com/kubewharf/kubeadmiral/pkg/controllers/status"
"github.com/kubewharf/kubeadmiral/pkg/controllers/federatedcluster"
"github.com/kubewharf/kubeadmiral/pkg/controllers/scheduler"
"github.com/kubewharf/kubeadmiral/pkg/controllers/status"
"github.com/kubewharf/kubeadmiral/pkg/controllers/sync"
)

func startFederateController(
Expand Down Expand Up @@ -207,3 +208,32 @@ func startScheduler(

return scheduler, nil
}

func startSyncController(
ctx context.Context,
controllerCtx *controllercontext.Context,
) (controllermanager.Controller, error) {
syncController, err := sync.NewSyncController(
controllerCtx.KubeClientset,
controllerCtx.FedClientset,
controllerCtx.FedInformerFactory.Core().V1alpha1().FederatedObjects(),
controllerCtx.FedInformerFactory.Core().V1alpha1().ClusterFederatedObjects(),
controllerCtx.InformerManager,
controllerCtx.FederatedInformerManager,
controllerCtx.FedSystemNamespace,
controllerCtx.TargetNamespace,
controllerCtx.ClusterAvailableDelay,
controllerCtx.ClusterUnavailableDelay,
controllerCtx.ComponentConfig.MemberObjectEnqueueDelay,
klog.Background(),
controllerCtx.WorkerCount,
controllerCtx.Metrics,
)
if err != nil {
return nil, fmt.Errorf("error creating sync controller: %w", err)
}

go syncController.Run(ctx)

return syncController, nil
}
4 changes: 0 additions & 4 deletions config/crds/core.kubeadmiral.io_clusterfederatedobjects.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -146,10 +146,6 @@ spec:
- status
type: object
type: array
collisionCount:
description: CollisionCount can be used in conjunction with RevisionHistory to implement rollbacks.
format: int32
type: integer
conditions:
description: Conditions describe the current state of this FederatedObject.
items:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ spec:
- name: v1alpha1
schema:
openAPIV3Schema:
description: ClusterPropagatedVersion holds version information about the state propagated from KubeFed APIs (configured by FederatedTypeConfig resources) to member clusters. The name of a ClusterPropagatedVersion encodes the kind and name of the resource it stores information for (i.e. <lower-case kind>-<resource name>). If a target resource has a populated metadata.Generation field, the generation will be stored with a prefix of `gen:` as the version for the cluster. If metadata.Generation is not available, metadata.ResourceVersion will be stored with a prefix of `rv:` as the version for the cluster.
description: ClusterPropagatedVersion holds version information about the state propagated from ClusterFederatedObject to member clusters. The name of a ClusterPropagatedVersion is the same as its ClusterFederatedObject. If a target resource has a populated metadata.Generation field, the generation will be stored with a prefix of `gen:` as the version for the cluster. If metadata.Generation is not available, metadata.ResourceVersion will be stored with a prefix of `rv:` as the version for the cluster.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
Expand All @@ -40,7 +40,7 @@ spec:
description: The name of the cluster the version is for.
type: string
version:
description: The last version produced for the resource by a KubeFed operation.
description: The last version produced for the resource by a KubeAdmiral operation.
type: string
required:
- clusterName
Expand Down
4 changes: 0 additions & 4 deletions config/crds/core.kubeadmiral.io_federatedobjects.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -146,10 +146,6 @@ spec:
- status
type: object
type: array
collisionCount:
description: CollisionCount can be used in conjunction with RevisionHistory to implement rollbacks.
format: int32
type: integer
conditions:
description: Conditions describe the current state of this FederatedObject.
items:
Expand Down
18 changes: 0 additions & 18 deletions config/crds/core.kubeadmiral.io_federatedtypeconfigs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,24 +68,6 @@ spec:
description: Path to a numeric field that reflects the number of replicas that the object currently has. E.g. `status.replicas` for Deployment and ReplicaSet.
type: string
type: object
revisionHistory:
description: Configuration for RevisionHistory. If left empty, the RevisionHistory feature will be disabled.
properties:
enabled:
description: Whether or not preserve a RevisionHistory for the federated object during updates.
type: boolean
required:
- enabled
type: object
rolloutPlan:
description: Configuration for RolloutPlan. If left empty, the RolloutPlan feature will be disabled.
properties:
enabled:
description: Whether or not to synchronize the rollout process across clusters.
type: boolean
required:
- enabled
type: object
sourceType:
description: The API resource type to be federated.
properties:
Expand Down
4 changes: 2 additions & 2 deletions config/crds/core.kubeadmiral.io_propagatedversions.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ spec:
- name: v1alpha1
schema:
openAPIV3Schema:
description: PropagatedVersion holds version information about the state propagated from KubeFed APIs (configured by FederatedTypeConfig resources) to member clusters. The name of a PropagatedVersion encodes the kind and name of the resource it stores information for (i.e. <lower-case kind>-<resource name>). If a target resource has a populated metadata.Generation field, the generation will be stored with a prefix of `gen:` as the version for the cluster. If metadata.Generation is not available, metadata.ResourceVersion will be stored with a prefix of `rv:` as the version for the cluster.
description: PropagatedVersion holds version information about the state propagated from FederatedObject to member clusters. The name of a PropagatedVersion is the same as its FederatedObject. If a target resource has a populated metadata.Generation field, the generation will be stored with a prefix of `gen:` as the version for the cluster. If metadata.Generation is not available, metadata.ResourceVersion will be stored with a prefix of `rv:` as the version for the cluster.
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
Expand All @@ -40,7 +40,7 @@ spec:
description: The name of the cluster the version is for.
type: string
version:
description: The last version produced for the resource by a KubeFed operation.
description: The last version produced for the resource by a KubeAdmiral operation.
type: string
required:
- clusterName
Expand Down
16 changes: 6 additions & 10 deletions pkg/apis/core/v1alpha1/extensions_federatedobject.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ limitations under the License.
package v1alpha1

import (
"encoding/json"
"reflect"
"sort"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime/schema"
k8sjson "k8s.io/apimachinery/pkg/util/json"
"k8s.io/apimachinery/pkg/util/sets"
)

Expand Down Expand Up @@ -222,16 +222,12 @@ func (spec *GenericFederatedObjectSpec) GetTemplateAsUnstructured() (*unstructur
return template, nil
}

// GetTemplateGVK returns the GVK of the FederatedObject's source object by parsing the FederatedObject's template.
func (spec *GenericFederatedObjectSpec) GetTemplateGVK() (schema.GroupVersionKind, error) {
type partialTypeMetadata struct {
metav1.TypeMeta `json:",inline"`
}
metadata := &partialTypeMetadata{}
if err := json.Unmarshal(spec.Template.Raw, metadata); err != nil {
return schema.GroupVersionKind{}, nil
func (spec *GenericFederatedObjectSpec) GetTemplateMetadata() (*metav1.PartialObjectMetadata, error) {
metadata := &metav1.PartialObjectMetadata{}
if err := k8sjson.Unmarshal(spec.Template.Raw, metadata); err != nil {
return nil, err
}
return metadata.GroupVersionKind(), nil
return metadata, nil
}

// Follower extensions
Expand Down
8 changes: 0 additions & 8 deletions pkg/apis/core/v1alpha1/extensions_federatedtypeconfig.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,14 +76,6 @@ func (f *FederatedTypeConfig) GetPolicyRcEnabled() bool {
return true // TODO: should this be configurable?
}

func (f *FederatedTypeConfig) GetRevisionHistoryEnabled() bool {
return f.Spec.RevisionHistory != nil && f.Spec.RevisionHistory.Enabled
}

func (f *FederatedTypeConfig) GetRolloutPlanEnabled() bool {
return f.Spec.RolloutPlan != nil && f.Spec.RolloutPlan.Enabled
}

func (f *FederatedTypeConfig) GetControllers() [][]string {
return f.Spec.Controllers
}
Expand Down
8 changes: 0 additions & 8 deletions pkg/apis/core/v1alpha1/types_federatedobject.go
Original file line number Diff line number Diff line change
Expand Up @@ -116,10 +116,6 @@ type GenericFederatedObjectStatus struct {
Conditions []GenericFederatedObjectCondition `json:"conditions,omitempty"`
// Clusters contains the propagation status of the Kubernetes object for individual member clusters.
Clusters []PropagationStatus `json:"clusters,omitempty"`

// CollisionCount can be used in conjunction with RevisionHistory to implement rollbacks.
// +optional
CollisionCount *int32 `json:"collisionCount,omitempty"`
}

// PlacementWithController describes the member clusters that a Kubernetes object should be propagated to.
Expand Down Expand Up @@ -211,12 +207,8 @@ type FederatedObjectConditionReason string

const (
AggregateSuccess FederatedObjectConditionReason = ""
SyncRevisionsFailed FederatedObjectConditionReason = "SyncRevisionsFailed"
ClusterRetrievalFailed FederatedObjectConditionReason = "ClusterRetrievalFailed"
ComputePlacementFailed FederatedObjectConditionReason = "ComputePlacementFailed"
PlanRolloutFailed FederatedObjectConditionReason = "PlanRolloutFailed"
CheckClusters FederatedObjectConditionReason = "CheckClusters"
NamespaceNotFederated FederatedObjectConditionReason = "NamespaceNotFederated"
EnsureDeletionFailed FederatedObjectConditionReason = "EnsureDeletionFailed"
)

Expand Down
18 changes: 0 additions & 18 deletions pkg/apis/core/v1alpha1/types_federatedtypeconfig.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,12 +56,6 @@ type FederatedTypeConfigSpec struct {
// Configuration for StatusAggregation. If left empty, the StatusAggregation feature will be disabled.
// +optional
StatusAggregation *StatusAggregationConfig `json:"statusAggregation,omitempty"`
// Configuration for RevisionHistory. If left empty, the RevisionHistory feature will be disabled.
// +optional
RevisionHistory *RevisionHistoryConfig `json:"revisionHistory,omitempty"`
// Configuration for RolloutPlan. If left empty, the RolloutPlan feature will be disabled.
// +optional
RolloutPlan *RolloutPlanConfig `json:"rolloutPlan,omitempty"`
// Configuration for StatusCollection. If left empty, the StatusCollection feature will be disabled.
// +optional
StatusCollection *StatusCollectionConfig `json:"statusCollection,omitempty"`
Expand Down Expand Up @@ -124,18 +118,6 @@ type StatusAggregationConfig struct {
Enabled bool `json:"enabled"`
}

// RevisionHistoryConfig defines the configurations for the RevisionHistory feature.
type RevisionHistoryConfig struct {
// Whether or not preserve a RevisionHistory for the federated object during updates.
Enabled bool `json:"enabled"`
}

// RolloutPlanConfig defines the configurations for the RolloutPlan feature.
type RolloutPlanConfig struct {
// Whether or not to synchronize the rollout process across clusters.
Enabled bool `json:"enabled"`
}

// AutoMigrationConfig defines the configurations for the AutoMigration feature.
type AutoMigrationConfig struct {
// Whether or not to automatically migrate unschedulable pods to a different cluster.
Expand Down
34 changes: 15 additions & 19 deletions pkg/apis/core/v1alpha1/types_propgatedversion.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,13 @@ import (
// +kubebuilder:resource:path=propagatedversions
// +kubebuilder:subresource:status

// PropagatedVersion holds version information about the state
// propagated from KubeFed APIs (configured by FederatedTypeConfig
// resources) to member clusters. The name of a PropagatedVersion
// encodes the kind and name of the resource it stores information for
// (i.e. <lower-case kind>-<resource name>). If a target resource has
// a populated metadata.Generation field, the generation will be
// stored with a prefix of `gen:` as the version for the cluster. If
// metadata.Generation is not available, metadata.ResourceVersion will
// be stored with a prefix of `rv:` as the version for the cluster.
// PropagatedVersion holds version information about the state propagated from
// FederatedObject to member clusters. The name of a PropagatedVersion is the
// same as its FederatedObject. If a target resource has a populated
// metadata.Generation field, the generation will be stored with a prefix of
// `gen:` as the version for the cluster. If metadata.Generation is not
// available, metadata.ResourceVersion will be stored with a prefix of `rv:` as
// the version for the cluster.
type PropagatedVersion struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Expand Down Expand Up @@ -71,7 +69,7 @@ type PropagatedVersionStatus struct {
type ClusterObjectVersion struct {
// The name of the cluster the version is for.
ClusterName string `json:"clusterName"`
// The last version produced for the resource by a KubeFed
// The last version produced for the resource by a KubeAdmiral
// operation.
Version string `json:"version"`
}
Expand All @@ -83,15 +81,13 @@ type ClusterObjectVersion struct {
// +kubebuilder:resource:path=clusterpropagatedversions,scope=Cluster
// +kubebuilder:subresource:status

// ClusterPropagatedVersion holds version information about the state
// propagated from KubeFed APIs (configured by FederatedTypeConfig
// resources) to member clusters. The name of a ClusterPropagatedVersion
// encodes the kind and name of the resource it stores information for
// (i.e. <lower-case kind>-<resource name>). If a target resource has
// a populated metadata.Generation field, the generation will be
// stored with a prefix of `gen:` as the version for the cluster. If
// metadata.Generation is not available, metadata.ResourceVersion will
// be stored with a prefix of `rv:` as the version for the cluster.
// ClusterPropagatedVersion holds version information about the state propagated
// from ClusterFederatedObject to member clusters. The name of a
// ClusterPropagatedVersion is the same as its ClusterFederatedObject. If a
// target resource has a populated metadata.Generation field, the generation
// will be stored with a prefix of `gen:` as the version for the cluster. If
// metadata.Generation is not available, metadata.ResourceVersion will be stored
// with a prefix of `rv:` as the version for the cluster.
type ClusterPropagatedVersion struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Expand Down
47 changes: 0 additions & 47 deletions pkg/apis/core/v1alpha1/zz_generated.deepcopy.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading