Replay Configs. Build k8s,knative and form cluster

Signed-off-by: Jason Chua <[email protected]>

.github/workflows/build.yml

@@ -7,7 +7,7 @@ on:
     - 'docs/**'
     - '**.md'
   pull_request:
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
     paths-ignore:
     - 'docs/**'
     - '**.md'

.github/workflows/build_setup.yml

@@ -7,7 +7,7 @@ on:
       - 'docs/**'
       - '**.md'
     pull_request:
-      branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+      branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
       paths-ignore:
       - 'docs/**'
       - '**.md'

.github/workflows/codeql-analysis.yml

@@ -16,7 +16,7 @@ on:
     branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
   pull_request:
     # The branches below must be a subset of the branches above
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
   workflow_dispatch:
   schedule:
     - cron: '40 13 * * 2'

.github/workflows/firecracker_cri_tests.yml

@@ -8,7 +8,7 @@ on:
       - '**.md'
       - 'function-images/**'
   pull_request:
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
     paths-ignore:
       - 'docs/**'
       - '**.md'

.github/workflows/gvisor_cri_tests.yml

@@ -8,7 +8,7 @@ on:
     - '**.md'
     - 'function-images/**'
   pull_request:
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
     paths-ignore:
     - 'docs/**'
     - '**.md'

.github/workflows/integration_tests.yml

@@ -8,7 +8,7 @@ on:
     - '**.md'
     - 'function-images/**'
   pull_request:
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
     paths-ignore:
     - 'docs/**'
     - '**.md'

.github/workflows/linters.yml

@@ -3,7 +3,7 @@ on:
   push:
     branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
   pull_request:
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
 
 jobs:
   build:

.github/workflows/stargz_tests.yml

@@ -8,7 +8,7 @@ on:
     - '**.md'
     - 'function-images/**'
   pull_request:
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
     paths-ignore:
     - 'docs/**'
     - '**.md'

.github/workflows/unit_tests.yml

@@ -8,7 +8,7 @@ on:
     - '**.md'
     - 'function-images/**'
   pull_request:
-    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support ]
+    branches: [ main, legacy-firecracker-v0.24.0-with-upf-support, openyurt ]
     paths-ignore:
     - 'docs/**'
     - '**.md'

scripts/openyurt-deployer/README.md

@@ -0,0 +1,120 @@
+# Quick set-up `OpenYurt`
+
+## 1. Introduction
+
+This program extends [`EasyOpenyurt`](https://github.com/flyinghorse0510/easy_openyurt) to automate the set up process of an `OpenYurt` cluster. 
+
+It support setting up a Kubernetes cluster using kubeadm and then deploy `OpenYurt` and Knative on it. It is compatible with vHive stock-only mode.
+
+## 2. Brief overview
+
+**Prerequisite of nodes:**
+1. The scripts has been tested on [cloud-lab](https://www.cloudlab.us/), suggested profile is: [`openyurt-demo`](https://www.cloudlab.us/p/ntu-cloud/openyurt-demo), with one master node, one cloud worker node and one edge worker node
+2. Ensure that SSH authentication is possible from local device to all nodes.
+
+
+**Components:**
+
+|      Files      | Purpose  |
+| :----------: | :---: |
+| main.go | script entry point |
+| `conf.json` | json files that stores cluster's configuration |
+| node | executing commands on remote nodes through ssh |
+| configs | node runtime configurations |
+
+**Description**
+
+1. Prepare system environment for all nodes, installing kubeadm, kubectl, dependencies, etc.
+2. On master node, init the cluster using `kubeadm init` and in each worker node, join the initialized cluster.
+3. On top of the created cluster, init `openyurt` cluster both on master nodes and worker nodes, then expand to all worker nodes from master nodes.
+4. (Optional) Deploy Knative (vHive stock-only mode compatible)
+
+## 3. Usage
+```bash
+./openyurt_deployer deploy # deploy openyurt on the cluster 
+```
+```bash
+./openyurt_deployer clean # clean the openyurt cluster and restore it to initial state 
+```
+
+### 3.1 Preparations 
+1. Prepare a cluster with at least two nodes.
+2. Change the contents in `conf.json` to following format:
+```plaintext
+{
+  "master": "user@master",
+  "workers": {
+    "cloud": [
+      "user@cloud-0"
+    ],
+    "edge": [
+      "user@edge-0"
+    ]
+  }
+}
+```
+
+### 3.2 Run Script
+
+```bash
+go build .
+./openyurt_deployer deploy
+```
+If it gives out error like: 
+```
+FATA[0001] Failed to connect to: username@host
+```
+Please execute:
+```
+eval `ssh-agent -s` && ssh-add ~/.ssh/<your private key>
+```
+For example:
+```
+eval `ssh-agent -s` && ssh-add ~/.ssh/id_rsa
+```
+And try again
+
+
+## 4. Demo: Create `NodePool` And Deploy service on it
+**Referenced from [`OpenYurt`](https://openyurt.io/docs/user-manuals/workload/node-pool-management)*
+
+The demo would deploy a helloworld function to cloud node pool or edge node pool
+
+Deploy the demo:
+```
+./openyurt_deployer demo-c
+```
+or:
+```
+./openyurt_deployer demo-e
+```
+where `demo-c` would deploy the service to the cloud node pool and `demo-e` would deploy the service to the edge node pool.
+
+The demo code will also show information about node pool after deployment.
+The name for `demo-c` would be `helloworld-cloud`, while the name for `demo-e` would be `helloworld-edge`
+It will also show the services' `URL` so you can try to invoke it on the master node.
+
+You can check the node pool information simply by:
+```
+./openyurt_deployer demo-print
+```
+Or delete the services deployed on node pool by:
+```
+./openyurt_deployer demo-clear
+```
+
+The demo code will also show information about node pool after deployment.
+
+### 4.1 Invoke the Services (Optional)
+You can try to invoke the services created by `demo-c` or `demo-e` on master node.
+First, ssh to master node, following commands should all be executed on master node.
+```
+ssh <master-user>@<master-ip>
+git clone https://github.com/vhive-serverless/vSwarm.git
+cd vSwarm/tools/test-client && go build .
+./test-client --addr $URL:80 --name "Hello there"
+```
+
+Here `$URL` should be the `URL` returned in the previous part when deploying cloud and edge services, you can also get it from: `kubectl get ksvc`, but discarding the `http://` at the beginning.
+
+After invoking, you can use `kubectl get pods -o wide` to check whether the pods have been auto-scaled.

scripts/openyurt-deployer/conf.json

@@ -0,0 +1,11 @@
+{
+    "master": "username@masterip",
+    "workers": {
+        "cloud": [
+            "username@cloudip"
+        ],
+        "edge": [
+            "username@edgeip"
+        ]
+    }
+}

scripts/openyurt-deployer/configs/configs.go

@@ -0,0 +1,91 @@
+package configs
+
+import (
+	"encoding/json"
+	"io"
+	"os"
+	"path"
+
+	utils "github.com/vhive-serverless/vHive/scripts/utils"
+)
+
+// Decode specific config files (JSON format)
+func DecodeConfig(configFilePath string, configStruct interface{}) error {
+	// Open & read the config file
+	configFile, err := os.Open(configFilePath)
+	if err != nil {
+		return err
+	}
+	defer configFile.Close()
+
+	// Read file content
+	configContent, err := io.ReadAll(configFile)
+	if err != nil {
+		return err
+	}
+
+	// Decode json into struct
+	err = json.Unmarshal(configContent, configStruct)
+
+	return err
+
+}
+
+// Load knative config files
+func (knative *KnativeConfigStruct) LoadConfig() error {
+	var err error
+	// Check config directory
+	if len(VHive.VHiveSetupConfigPath) == 0 {
+		VHive.VHiveSetupConfigPath, err = utils.GetVHiveFilePath("configs/setup")
+		if err != nil {
+			utils.CleanEnvironment()
+			os.Exit(1)
+		}
+	}
+	// Get the (absolute) path of the config file
+	configFilePath := path.Join(VHive.VHiveSetupConfigPath, "knative.json")
+
+	// Decode json into struct
+	err = DecodeConfig(configFilePath, knative)
+
+	return err
+
+}
+
+// Load kubernetes config files
+func (kube *KubeConfigStruct) LoadConfig() error {
+	// Get the (absolute) path of the config file
+	configFilePath := path.Join(VHive.VHiveSetupConfigPath, "kube.json")
+
+	// Decode json into struct
+	err := DecodeConfig(configFilePath, kube)
+
+	return err
+}
+
+// Load system config files
+func (system *SystemEnvironmentStruct) LoadConfig() error {
+	// Get the (absolute) path of the config file
+	configFilePath := path.Join(VHive.VHiveSetupConfigPath, "system.json")
+
+	// Decode json into struct
+	err := DecodeConfig(configFilePath, system)
+
+	return err
+}
+
+// Load vHive config files
+func (vhive *VHiveConfigStruct) LoadConfig() error {
+	// Get the (absolute) path of the config file
+	configFilePath := path.Join(VHive.VHiveSetupConfigPath, "vhive.json")
+
+	// Decode json into struct
+	err := DecodeConfig(configFilePath, vhive)
+
+	return err
+
+}
+
+const (
+	Version = "0.2.4b" // Version Info
+)

scripts/openyurt-deployer/configs/demo.go

@@ -0,0 +1,21 @@
+package configs
+
+type DemoEnvironment struct {
+	CloudYamlFile      string
+	EdgeYamlFile       string
+	CloudBenchYamlFile string
+	EdgeBenchYamlFile  string
+	YurtAppSetYamlFile string
+	CloudPoolName      string
+	EdgePoolName       string
+}
+
+var Demo = DemoEnvironment{
+	CloudYamlFile:      "cloud.yaml",
+	EdgeYamlFile:       "edge.yaml",
+	CloudBenchYamlFile: "cloud-bench.yaml",
+	EdgeBenchYamlFile:  "edge-bench.yaml",
+	YurtAppSetYamlFile: "yurt.yaml",
+	CloudPoolName:      "cloud",
+	EdgePoolName:       "edge",
+}

scripts/openyurt-deployer/configs/go.mod

@@ -0,0 +1,3 @@
+module github.com/vhive-serverless/vhive/scripts/openyurt_deployer/configs
+
+go 1.20

scripts/openyurt-deployer/configs/knative.go

@@ -0,0 +1,35 @@
+package configs
+
+import "fmt"
+
+type KnativeConfigStruct struct {
+	KnativeVersion                       string
+	IstioVersion                         string
+	IstioDownloadUrlTemplate             string
+	IstioOperatorConfigUrl               string
+	MetalLBVersion                       string
+	MetalLBConfigURLArray                []string
+	LocalRegistryRepoVolumeSize          string
+	LocalRegistryVolumeConfigUrl         string
+	LocalRegistryDockerRegistryConfigUrl string
+	LocalRegistryHostUpdateConfigUrl     string
+	MagicDNSConfigUrl                    string
+	VHiveMode                            bool
+}
+
+var Knative = KnativeConfigStruct{
+	IstioOperatorConfigUrl:   "https://raw.githubusercontent.com/vhive-serverless/vHive/main/configs/istio/istio-minimal-operator.yaml",
+	IstioDownloadUrlTemplate: "https://github.com/istio/istio/releases/download/%s/istio-%s-linux-%s.tar.gz",
+	MetalLBConfigURLArray: []string{
+		"https://raw.githubusercontent.com/vhive-serverless/vHive/main/configs/metallb/metallb-ipaddresspool.yaml",
+		"https://raw.githubusercontent.com/vhive-serverless/vHive/main/configs/metallb/metallb-l2advertisement.yaml"},
+	LocalRegistryVolumeConfigUrl:         "https://raw.githubusercontent.com/vhive-serverless/vHive/main/configs/registry/repository-volume.yaml",
+	LocalRegistryDockerRegistryConfigUrl: "https://raw.githubusercontent.com/vhive-serverless/vHive/main/configs/registry/docker-registry.yaml",
+	LocalRegistryHostUpdateConfigUrl:     "https://raw.githubusercontent.com/vhive-serverless/vHive/main/configs/registry/repository-update-hosts.yaml",     //TODO: uses path
+	MagicDNSConfigUrl:                    "https://raw.githubusercontent.com/vhive-serverless/vHive/main/configs/knative_yamls/serving-default-domain.yaml", //TODO: uses path
+	VHiveMode:                            true,
+}
+
+func (knative *KnativeConfigStruct) GetIstioDownloadUrl() string {
+	return fmt.Sprintf(knative.IstioDownloadUrlTemplate, knative.IstioVersion, knative.IstioVersion, System.CurrentArch)
+}

scripts/openyurt-deployer/configs/kube.go

@@ -0,0 +1,14 @@
+package configs
+
+type KubeConfigStruct struct {
+	K8sVersion                string
+	AlternativeImageRepo      string
+	ApiserverAdvertiseAddress string
+	PodNetworkCidr            string
+	PodNetworkAddonConfigURL  string
+	ApiserverPort             string
+	ApiserverToken            string
+	ApiserverTokenHash        string
+}
+
+var Kube = KubeConfigStruct{}