Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple CRIB config from cl-cluster helm chart #12877

Merged
merged 4 commits into from
Apr 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ tools/clroot/db.sqlite3-wal
.env*
.dbenv
!charts/chainlink-cluster/.env.example
!crib/.env.example
!.github/actions/setup-postgres/.env
.direnv
.idea
Expand Down
156 changes: 3 additions & 153 deletions charts/chainlink-cluster/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,112 +2,9 @@

Example CL nodes cluster for system level tests

Install `kubefwd` (no nixpkg for it yet, planned)

```
brew install txn2/tap/kubefwd
```

If you want to build images you need [docker](https://docs.docker.com/engine/install/) service running

Enter the shell (from the root project dir)

```
nix develop
```

# Develop

## New cluster

We are using [devspace](https://www.devspace.sh/docs/getting-started/installation?x0=3)

Configure the cluster, see `deployments.app.helm.values` and [values.yaml](./values.yaml) comments for more details

Set up your K8s access

```
export DEVSPACE_IMAGE="..."
./setup.sh ${my-personal-namespace-name-crib}
```

Create a .env file based on the .env.sample file

```sh
cp .env.sample .env
# Fill in the required values in .env
```

Build and deploy the current state of your repository

```
devspace deploy
```

Default `ttl` is `72h`, use `ttl` command to update if you need more time

Valid values are `1h`, `2m`, `3s`, etc. Go time format is invalid `1h2m3s`

```
devspace run ttl ${namespace} 120h
```

If you want to deploy an image tag that is already available in ECR, use:

```
devspace deploy --override-image-tag "<image-tag>"
```

If you want to deploy an image tag from a public ECR repo, use:

```
export DEVSPACE_IMAGE=public.ecr.aws/chainlink/chainlink
devspace deploy --override-image-tag 2.9.0
```

Forward ports to check UI or run tests

```
devspace run connect ${my-personal-namespace-name-crib}
```

List ingress hostnames

```
devspace run ingress-hosts
```

Destroy the cluster

```
devspace purge
```

## Running load tests

Check this [doc](../../integration-tests/load/ocr/README.md)

If you used `devspace dev ...` always use `devspace reset pods` to switch the pods back

# Helm

If you would like to use `helm` directly, please uncomment data in `values.yaml`

## Install from local files

```
helm install -f values.yaml cl-cluster .
```

Forward all apps (in another terminal)

```
sudo kubefwd svc -n cl-cluster
```

Then you can connect and run your tests

## Install from release
Note: The setup below doesn't work at the moment.

Add the repository

Expand All @@ -130,6 +27,7 @@ helm install -f values.yaml cl-cluster .
```

## Create a new release
Note: The setup below doesn't work at the moment.

Bump version in `Chart.yml` add your changes and add `helm_release` label to any PR to trigger a release

Expand All @@ -143,52 +41,4 @@ helm test cl-cluster

```
helm uninstall cl-cluster
```

# Grafana dashboard

We are using [Grabana](https://github.com/K-Phoen/grabana) lib to create dashboards programmatically

You can also select dashboard platform in `INFRA_PLATFORM` either `kubernetes` or `docker`

You can select the dashboard panels with `PANELS_INCLUDED` which is a list of panel names separated by comma
If you don't specify it will include core panels by default

```
export LOKI_TENANT_ID=promtail
export LOKI_URL=...
export GRAFANA_URL=...
export GRAFANA_TOKEN=...
export PROMETHEUS_DATA_SOURCE_NAME=Thanos
export LOKI_DATA_SOURCE_NAME=Loki
export INFRA_PLATFORM=kubernetes
export GRAFANA_FOLDER=DashboardCoreDebug
export DASHBOARD_NAME=CL-Cluster

devspace run dashboard_deploy
```

Open Grafana folder `DashboardCoreDebug` and find dashboard `ChainlinkClusterDebug`

# Testing

Deploy your dashboard and run soak/load [tests](../../integration-tests/load/), check [README](../../integration-tests/README.md) for further explanations

```
devspace run dashboard_deploy
devspace run workload
devspace run dashboard_test
```

# Local Testing

Go to [dashboard-lib](../../dashboard) and link the modules locally

```
cd dashboard
pnpm link --global
cd charts/chainlink-cluster/dashboard/tests
pnpm link --global dashboard-tests
```

Then run the tests with commands mentioned above
```
File renamed without changes.
156 changes: 156 additions & 0 deletions crib/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# Crib Devspace Setup

CRIB is a devspace configuration to launch chainlink cluster for system level tests

Install `kubefwd` (no nixpkg for it yet, planned)

```
brew install txn2/tap/kubefwd
```

If you want to build images you need [docker](https://docs.docker.com/engine/install/) service running

Enter the shell (from the root project dir)

```
nix develop
```

# Develop

## New cluster

We are using [devspace](https://www.devspace.sh/docs/getting-started/installation?x0=3)

Configure the cluster, see `deployments.app.helm.values` and [values.yaml](../charts/chainlink-cluster/values.yaml) comments for more details

Set up your K8s access

```
export DEVSPACE_IMAGE="..."
./setup.sh ${my-personal-namespace-name-crib}
```

Create a .env file based on the .env.sample file

```sh
cp .env.sample .env
# Fill in the required values in .env
```

Build and deploy the current state of your repository

```
devspace deploy
```

Default `ttl` is `72h`, use `ttl` command to update if you need more time

Valid values are `1h`, `2m`, `3s`, etc. Go time format is invalid `1h2m3s`

```
devspace run ttl ${namespace} 120h
```

If you want to deploy an image tag that is already available in ECR, use:

```
devspace deploy --override-image-tag "<image-tag>"
```

If you want to deploy an image tag from a public ECR repo, use:

```
export DEVSPACE_IMAGE=public.ecr.aws/chainlink/chainlink
devspace deploy --override-image-tag 2.9.0
```

Forward ports to check UI or run tests

```
devspace run connect ${my-personal-namespace-name-crib}
```

List ingress hostnames

```
devspace run ingress-hosts
```

Destroy the cluster

```
devspace purge
```

## Running load tests

Check this [doc](../../integration-tests/load/ocr/README.md)

If you used `devspace dev ...` always use `devspace reset pods` to switch the pods back

# Helm

If you would like to use `helm` directly, please uncomment data in `values.yaml`

## Install from local files

```
helm install -f values.yaml cl-cluster .
```

Forward all apps (in another terminal)

```
sudo kubefwd svc -n cl-cluster
```

Then you can connect and run your tests

# Grafana dashboard

We are using [Grabana](https://github.com/K-Phoen/grabana) lib to create dashboards programmatically

You can also select dashboard platform in `INFRA_PLATFORM` either `kubernetes` or `docker`

You can select the dashboard panels with `PANELS_INCLUDED` which is a list of panel names separated by comma
If you don't specify it will include core panels by default

```
export LOKI_TENANT_ID=promtail
export LOKI_URL=...
export GRAFANA_URL=...
export GRAFANA_TOKEN=...
export PROMETHEUS_DATA_SOURCE_NAME=Thanos
export LOKI_DATA_SOURCE_NAME=Loki
export INFRA_PLATFORM=kubernetes
export GRAFANA_FOLDER=DashboardCoreDebug
export DASHBOARD_NAME=CL-Cluster

devspace run dashboard_deploy
```

Open Grafana folder `DashboardCoreDebug` and find dashboard `ChainlinkClusterDebug`

# Testing

Deploy your dashboard and run soak/load [tests](../../integration-tests/load/), check [README](../../integration-tests/README.md) for further explanations

```
devspace run dashboard_deploy
devspace run workload
devspace run dashboard_test
```

# Local Testing

Go to [dashboard-lib](../dashboard-lib) and link the modules locally

```
cd dashboard
pnpm link --global
cd crib/dashboard/tests
pnpm link --global dashboard-tests
```

Then run the tests with commands mentioned above
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ commands:
ttl: |-
kubectl label namespace $1 cleanup.kyverno.io/ttl=$2 --overwrite
workload: |-
cd ../../integration-tests/load/ocr && go test -v -run TestOCRLoad || cd -
cd ../integration-tests/load/ocr && go test -v -run TestOCRLoad || cd -
dashboard_deploy: |-
go run dashboard/cmd/deploy.go
dashboard_test: |-
Expand Down Expand Up @@ -98,7 +98,7 @@ deployments:
releaseName: "app"
chart:
name: cl-cluster
path: .
path: ../charts/chainlink-cluster
# for simplicity, we define all the values here
# they can be defined the same way in values.yml
# devspace merges these "values" with the "values.yaml" before deploy
Expand Down
File renamed without changes.
2 changes: 1 addition & 1 deletion charts/chainlink-cluster/go.mod → crib/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ replace (
github.com/mwitkow/grpc-proxy => github.com/smartcontractkit/grpc-proxy v0.0.0-20230731113816-f1be6620749f

github.com/sercand/kuberesolver/v4 => github.com/sercand/kuberesolver/v5 v5.1.1
github.com/smartcontractkit/chainlink/dashboard-lib => ../../dashboard-lib
github.com/smartcontractkit/chainlink/dashboard-lib => ./../dashboard-lib
)
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion dashboard-lib/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ dashboard

Each component should contain rows, logic and unique variables in `component.go`

Components should be imported from this module, see [example](../charts/chainlink-cluster/dashboard/cmd/deploy.go)
Components should be imported from this module, see [example](../crib/dashboard/cmd/deploy.go)

## How to convert from JSON using Grabana codegen utility
1. Download Grabana binary [here](https://github.com/K-Phoen/grabana/releases)
Expand Down
2 changes: 1 addition & 1 deletion integration-tests/load/ocr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ sudo kubefwd svc -n cl-cluster

Change environment connection configuration [here](../../../charts/chainlink-cluster/connect.toml)

If you haven't changed anything in [devspace.yaml](../../../charts/chainlink-cluster/devspace.yaml) then default connection configuration will work
If you haven't changed anything in [devspace.yaml](../../crib/devspace.yaml) then default connection configuration will work

## Usage

Expand Down
Loading