Merge pull request #125 from ShwethaKumbla/watcher

implement k8s resource watch and trigger action based on the events

docs/design/k8s-resource-watcher.md

@@ -0,0 +1,164 @@
+# Watcher for K8S Objects
+
+K8S object watchers are great functionality provided by k8s to get efficient change notifications on resources.
+
+The events supported by these watchers are
+1. ADD
+2. MODIFY/UPDATE
+3. DELETE
+
+## Motivation
+Users must implement watchers such a way that whenever any events recieved some action/job has to be triggered. Developer has to write lots of code to do this and its sometimes difficult to manage. 
+
+This can be achieved by using the k8s client built in function but understanding which packages to import or which core type needs to be used might be a complex for the developer.
+
+The below design would make developer life easier. They have to just register their actions for respective events. To stay informed about when these events get triggered just use Watch(), which resides inside klient/k8s/resources package.
+
+## Proposal
+Watch function accepts a `object ObjectList` as a argument. ObjectList type is used to inject the resource type in which Watch has to be applied.
+
+`klient/k8s/resources/resources.go`
+```go=
+import (
+    "sigs.k8s.io/controller-runtime/pkg/client"
+    "k8s.io/apimachinery/pkg/watch"
+)
+
+func (r *Resources) Watch(object k8s.ObjectList, opts ...ListOption) *watcher.EventHandlerFuncs {
+	listOptions := &metav1.ListOptions{}
+
+	for _, fn := range opts {
+		fn(listOptions)
+	}
+
+	o := &cr.ListOptions{Raw: listOptions}
+
+	return &watcher.EventHandlerFuncs{
+		ListOptions: o,
+		K8sObject:   object,
+		Cfg:         r.GetConfig(),
+	}
+}
+```
+
+Watch() in resources.go will return the `watcher` type which helps to call `Start()`. InvokeEventHandler accepts `EventHandlerFuncs` which carries the user registerd function sets.
+
+file : klient/k8s/resources/watch.go
+
+```go=
+// Start triggers the registered methods based on the event recieved for particular k8s resources.
+func (watcher watch.Interface)Start(ctx context.Context) {
+    ...
+    go func() {
+		for {
+			select {
+			case <-ctx.Done():
+				if ctx.Err() != nil {
+					return
+				}
+			case event := <-e.watcher.ResultChan():
+				// retrieve the event type
+				eventType := event.Type
+
+				switch eventType {
+				case watch.Added:
+					// calls AddFunc if it's not nil.
+					if e.addFunc != nil {
+						e.addFunc(event.Object)
+					}
+				case watch.Modified:
+					// calls UpdateFunc if it's not nil.
+					if e.updateFunc != nil {
+						e.updateFunc(event.Object)
+					}
+				case watch.Deleted:
+					// calls DeleteFunc if it's not nil.
+					if e.deleteFunc != nil {
+						e.deleteFunc(event.Object)
+					}
+				}
+			}
+		}
+	}()
+    ...
+}
+
+// EventHandlerFuncs is an adaptor to let you easily specify as many or
+// as few of functions to invoke while getting notification from watcher
+type EventHandlerFuncs struct {
+	addFunc     func(obj interface{})
+	updateFunc  func(newObj interface{})
+	deleteFunc  func(obj interface{})
+	watcher     watch.Interface
+	ListOptions *cr.ListOptions
+	K8sObject   k8s.ObjectList
+	Cfg         *rest.Config
+}
+
+// EventHandler can handle notifications for events that happen to a resource.
+// Start will be waiting for the events notification which is responsible
+// for invoking the registered user defined functions.
+// Stop used to stop the watcher.
+type EventHandler interface {
+	Start(ctx context.Context)
+	Stop()
+}
+
+```
+
+`Start()` is invoked in a goroutine so that whenever watched resource changes the states it will call the registered user defined functions.
+`Stop()` should be explicitly invoked by the user after the watch once the feature is done to ensure no unwanted go routine thread leackage.
+
+If any error while Start() one can retry it for number of times.
+
+This example shows how to use klient/resources/resources.go Watch() func and how to register the user defined functions.
+
+```go=
+
+import (
+    "sigs.k8s.io/e2e-framework/klient/conf"
+    "sigs.k8s.io/e2e-framework/klient/k8s/resources"
+)
+
+func main() {
+    ...
+    cfg, _ := conf.New(conf.ResolveKubeConfigFile())
+    cl, err := cfg.NewClient()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	dep := appsv1.Deployment{
+    	ObjectMeta: metav1.ObjectMeta{Name: "watch-dep", Namespace: cfg.Namespace()},
+	}
+
+	// watch for the deployment and triger action based on the event recieved.
+	cl.Resources().Watch(&appsv1.DeploymentList{}, resources.WithFieldSelector(labels.FormatLabels(map[string]string{"metadata.name": dep.Name}))).
+	WithAddFunc(onAdd).WithDeleteFunc(onDelete).Start(ctx)
+
+    ...
+}
+
+// onAdd is the function executed when the kubernetes watch notifies the
+// presence of a new kubernetes deployment in the cluster
+func onAdd(obj interface{}) {
+    dep := obj.(*appsv1.Deployment)
+    _, ok := dep.GetLabels()[K8S_LABEL_AWS_REGION]
+    if ok {
+        fmt.Printf("It has the label!")
+    }
+}
+
+// onDelete is the function executed when the kubernetes watch notifies 
+// delete event on deployment
+func onDelete(obj interface{}) {
+    dep := obj.(*appsv1.Deployment)
+    _, ok := dep.GetLabels()[K8S_LABEL_AWS_REGION]
+    if ok {
+        fmt.Printf("It has the label!")
+    }
+}
+
+```
+
+The e2e flow of how to use watch is demonsrated in the examples/ folder.

docs/design/test-harness-framework.md

@@ -469,7 +469,7 @@ The framework should automatically inject these filter values into the environme
 ### Skipping features
 The test framework should provide the ability to explicitly exclude features during a test run.  This could be done with the following flags:
 
-* `--skip-feature` - a regular expression that skips features with matching names
+* `--skip-features` - a regular expression that skips features with matching names
 * `--skip-assessment` - a regular expression that skips assessment with matching name
 * `--skip-lables` - a comma-separated list of key/value pairs used to skip features with matching lables
 

examples/watch_resources/README.md

@@ -0,0 +1,54 @@
+# Watching for Resource Changes
+
+The test harness supports several methods for querying Kubernetes object types and watching for resource states. This example shows how to watch particular resource and how to register the functions to act upon based on the events recieved.
+
+
+# Watch for the deployment and triger action based on the event
+
+Watch has to run as goroutine to get the different events based on the k8s resource state changes.
+```go
+func TestWatchForResources(t *testing.T) {
+...
+dep := appsv1.Deployment{
+				ObjectMeta: metav1.ObjectMeta{Name: "watch-dep", Namespace: cfg.Namespace()},
+			}
+
+			// watch for the deployment and triger action based on the event recieved.
+			cl.Resources().Watch(&appsv1.DeploymentList{}, &client.ListOptions{
+				FieldSelector: fields.OneTermEqualSelector("metadata.name", dep.Name),
+				Namespace:     dep.Namespace}, cl.RESTConfig()).WithAddFunc(onAdd).WithDeleteFunc(onDelete).Start(ctx)
+...
+}
+```
+
+# Function/Action definition and registering these actions
+
+```go
+// onAdd is the function executed when the kubernetes watch notifies the
+// presence of a new kubernetes deployment in the cluster
+func onAdd(obj interface{}) {
+	dep := obj.(*appsv1.Deployment)
+	depName := dep.GetName()
+	fmt.Printf("Dep name recieved is %s", depName)
+	if depName == "watch-dep" {
+		fmt.Println("Dep name matches with actual name!")
+	}
+}
+
+// onDelete is the function executed when the kubernetes watch notifies
+// delete event on deployment
+func onDelete(obj interface{}) {
+	dep := obj.(*appsv1.Deployment)
+	depName := dep.GetName()
+	if depName == "watch-dep" {
+		fmt.Println("Deployment deleted successfully!")
+	}
+}
+```
+
+The above functions can be registered using Register functions(WithAddFunc(), WithDeleteFunc(), WithUpdateFunc()) defined under klient/k8s/watcher/watch.go as shown in the example.
+
+# How to stop the watcher
+Create a global EventHandlerFuncs variable to store the watcher object and call Stop() as shown in example TestWatchForResourcesWithStop() test.
+
+Note: User should explicitly invoke the Stop() after the watch once the feature is done to ensure no unwanted go routine thread leackage.

examples/watch_resources/main_test.go

@@ -0,0 +1,43 @@
+/*
+Copyright 2022 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package watch_resources
+
+import (
+	"os"
+	"testing"
+
+	"sigs.k8s.io/e2e-framework/pkg/env"
+	"sigs.k8s.io/e2e-framework/pkg/envconf"
+	"sigs.k8s.io/e2e-framework/pkg/envfuncs"
+)
+
+var testenv env.Environment
+
+func TestMain(m *testing.M) {
+	testenv = env.New()
+	kindClusterName := envconf.RandomName("watch-for-resources", 16)
+	namespace := envconf.RandomName("watch-ns", 16)
+	testenv.Setup(
+		envfuncs.CreateKindCluster(kindClusterName),
+		envfuncs.CreateNamespace(namespace),
+	)
+	testenv.Finish(
+		envfuncs.DeleteNamespace(namespace),
+		envfuncs.DestroyKindCluster(kindClusterName),
+	)
+	os.Exit(testenv.Run(m))
+}