Skip to content

Latest commit

 

History

History
235 lines (150 loc) · 25.3 KB

DEVELOPING.adoc

File metadata and controls

235 lines (150 loc) · 25.3 KB
Raw

Developing Kiali Operator

Setting up your local git repositories

The Kiali Operator source lives in its own git repository, but because it is necessary for Kiali developers to use it, the Kiali Operator repository should be accessible in the Kiali git repo via a softlink in the "operator" directory or just cloned right inside the "operator" directory.

For example:

# Clone Kiali Operator in a local git repository location
cd $HOME/source
git clone [email protected]:kiali/kiali-operator.git

# Go to your existing Kiali local git repository
GOPATH=<your GO path to the Kiali project - something like $HOME/source/kiali>
cd $GOPATH/src/github.com/kiali/kiali

# Provide a link to your Kiali Operator local git repository
ln -s $HOME/source/kiali-operator operator
Note
 If you want to clone directly into your kiali/kiali operator directory, you run something like: git clone [email protected]:kiali/kiali-operator.git $GOPATH/src/github.com/kiali/kiali/operator. However, if you plan on making code changes to the operator, it is highly recommended you use the softlink approach mentioned above. See the WARNING below for an explanation for this recommendation.
Warning
 If you git cloned the operator repo directly into your kiali/kiali git repo’s operator/ directory, and you need to checkout an older branch of your kiali git repo, then you must make sure you push any operator changes to your remote fork before you checkout that older kiali repo branch, otherwise you will lose those operator changes. This is because the operator directory will get replaced with the old kiali repo branch’s original operator content. When you then checkout the master branch again, your operator directory will be deleted. At this point, you will need to either create your softlink again or re-clone the git repo. It is for this reason that it is recommended that the operator directory be a softlink that points to an external directory where you have git cloned the operator repo (with a softlink, your operator code changes remain intact - you merely have to recreate the softlink again).

This now makes it easy for Kiali developers to use the operator when working directly in the Kiali git repo. The Kiali git repo has all the necessary make targets to build, deploy, and test the Kiali Operator along with Kiali itself. This is very similiar to how things worked prior to moving the operator code to a separate git repo.

The Kiali Operator git repo itself has a very small Makefile whose sole job is to build and push the Kiali Operator container image. This Makefile is used mainly for CI and release automation; developers typically will not need to use it.

Developing Kiali Operator While Working In Kiali Git Repo

Assuming you set up your local git repositories as explained in the above section Setting up your local git repositories, then you can develop Kiali Operator features and bug fixes while working in the local Kiali repo directory.

If, however, a developer needs to make changes to the operator, because the Kiali Operator code really lives in a separate repo, the developer will need to create PRs to the Kiali Operator git repo (just like any other git repo you want to commit to). Simply use the same basic workflow and commands that you are used to using already when creating PRs - just remember you need to create a PR from within the Kiali local repo (if changes were made to it) and you need to create another PR from within the "operator" local repo (if changes were made to the operator code).

Running the Kiali Operator

Installing the Operator in a Cluster Using The Helm Charts

The Kiali git repo has a Makefile and a set of Makefile targets that are used in conjunction with the Kiali Operator so you can install and run the operator built from your local git clones. They use the Kiali Operator helm chart to install the operator, so you must clone the helm-chart git repo in the proper place as well.

To install the operator that contains your local changes, run make operator-create. If you are using minikube, you will need to set the environment variable CLUSTER_TYPE=minikube and you need to also set MINIKUBE_PROFILE if your minikube profile name is not minikube. At this point you can create your own Kiali CR to trigger the operator to install Kiali, or use make kiali-create to install a sample Kiali CR for you. Similarly, make ossmconsole-create will create an OSSMConsole CR to install the OSSMC OpenShift Console plugin.

To remove the operator (and any CRs that may have also been created), run make operator-delete.

Installing the Operator in a Cluster Using OLM

If you have a cluster with OLM installed (e.g. OpenShift), you can test the Kiali OLM metadata by installing the Kiali operator using OLM via make.

If you do not have OLM installed, you can easily install it. First determine what OLM release you want to install. Then simply run the command curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${OLM_VERSION}/install.sh | bash -s ${OLM_VERSION}. If you are running minikube, all you need to do is execute the command hack/k8s-minikube.sh olm and the hack script will install the latest OLM for you.

Once OLM is installed, you can install the Kiali Operator (and all the required resources such as the OLM catalog source and subscription) via make olm-operator-create.

Once the operator is running, you can install a Kiali CR normally. One way to do this is to first create a kiali-operator namespace and run the kiali-create make target: kubectl create ns kiali-operator && make kiali-create. Similarly with the ossmconsole-create target for installing OSSMC.

To remove the Kiali operator and all the other OLM resources that came with it (including the CRs if you created some via the kiali-create target or the ossmconsole-create target), run make olm-operator-delete.

Note
 By default, you will test with the latest version of the "kiali-ossm" metadata. You can specify a different version by setting the BUNDLE_VERSION env var on the make command line. If the OC env var points to an oc binary, the metadata from the kiali-community will be used; otherwise, the metadata from the kiali-upstream will be used. If you want to do this, you also must set the env var OLM_BUNDLE_PACKAGE to kiali on the make command line.

Running the Operator Playbook Without Installing in a Cluster

Sometimes you made simple changes to the operator playbook and roles that can be quickly tested by running the Ansible playbook on your local machine. This saves time because it doesn’t require you to install the operator directly in your cluster. To do this, you must have Ansible installed (e.g. ansible-playbook must be in your $PATH). You can run the operator playbook via make run-operator-playbook-kiali (to test the Kiali playbooks) or make run-operator-playbook-ossmconsole (to test the OSSMC playbooks). run-operator-playbook-kiali will run both the kiali-deploy and kiali-remove playbooks so the operator will install and then immediately uninstall a Kiali Server. run-operator-playbook-ossmconsole will run both the ossmconsole-deploy and ossmconsole-remove playbooks so the operator will install and then immediately uninstall a OSSMC plugin. The operator playbooks are configured via the files found in the dev-playbook-config directory.

In order to use these "run-operator-playbook" targets, you must have Python3 in your PATH (having an alias is not enough).

You also must make sure your local Python and Ansible environment matches as closely as possible to the environment of the Kiali operator. To find out the different versions of software within the Kiali operator image, run the following:

  • To get the Python version: podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest python3 --version

  • To get the Python libraries (kubernetes is an important one to look at): podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest pip3 list

  • To get the Ansible version: podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest ansible --version

To get started, try to run python -m pip install --user --upgrade -r operator/molecule/requirements.txt to get Python3 libraries installed. If you need to, you can install specific versions of the libraries like this: python -m pip install --user kubernetes==12.0.1.

You also must get the required Ansible collections installed. To find out the different versions of Ansible collections within the Kiali operator image, run the following:

podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest ansible-galaxy collection list

Note
 Older versions of the operator base image did not support the collection list option to ansible-galaxy. For those older images, you can run this script to determine the versions of the collections installed in the image: for m in $(podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest find .ansible/collections/ansible_collections -name MANIFEST.json); do manifest=$(echo -n ${m} | tr --delete '\r'); echo -n "${manifest}-→"; podman run --rm -i -t --entrypoint "" quay.io/kiali/kiali-operator:latest cat "${manifest}" | jq .collection_info.version; done

To install these, run ansible-galaxy collection install -r operator/requirements.yml --force-with-deps

Running the Operator Without Installing in a Cluster

The previous section tells you how to run the Ansible playbook directly on your local machine. If you want to run the actual operator on your local machine in the same manner in which it runs inside the cluster (that is, within the ansible-operator shell process) then use the make run-operator make target. This will run the ansible-operator executable and point it to the operator’s Ansible playbooks and roles. This operator will watch for Kiali CRs and OSSMConsole CRs in the cluster - when it sees one, it will process it just as if it was running in the cluster. This will allow you to test the Ansible operator infrastructure as well as the operator’s Ansible playbooks themselves.

Running the Operator With the Ansible Profiler

If the ansible.cfg has callbacks_enabled = profile_tasks specified (or the deprecated callback_whitelist for older Ansible installations), a profiler is run and its report is dumped at the end of each reconciliation run. You can tell the operator to use such a configuration by setting the ANSIBLE_CONFIG environment variable to /opt/ansible/ansible-profiler.cfg.

The profiler report is usually good enough to start tracking down performance issues. But it may be difficult to see performance issues over tasks that loop because the profiler report will show duplicate entries for each loop invocation. In that case, copy the profiler report to a file and pipe it to a script with the following content to provide a more cumulative report:

awk -F~ '                   ##Setting field separator as tilde here.
{
  val=$2;                   ##Creating a variable named val whose value is 2nd field of current line.
  $2="@";                   ##Setting value of 2nd field as @ here to keep all lines same(to create index for array a).
  a[$0]+=val                ##Creating an array named a whose index is the current line and its value is the new sum
}
!b[$0]++{                   ##Checking if array b, whose index is the current line, has a value of NULL; if so do following.
  c[++count]=$0}            ##Creating an array named c whose index is variable count increasing value with 1 and value is line.
END{                        ##Starting END block of awk code here.
  for(i=1;i<=count;i++){    ##Starting a for loop whose value starts from 1 to till value of count variable.
     sub("@",a[c[i]],c[i]); ##Substituting @ in value of array c(which is actually lines value) with value of SUMMED $2.
     print c[i]}            ##Printing newly value of array c where $2 is now replaced with its actual value.
}' OFS=\~ <(cat - | sed 's/\(.*\) -\+ \(.*\)s/\1~\2/') | sort -n -t '~' -k 2 -r | column -s~ -t

The Initial Kiali Operator Git Repo Creation

Note
 This section documents how the Kiali Operator git repo was originally created. This only had to be performed one time and one time only. Developers and consumers of Kiali and Kiali Operator do not need to know any of this, so you can skip this section. This is here just for historical purposes.

The new Kiali Operator git repo was created with a empty history. Once a repo was created, it then needed to be populated with the original operator code from the Kiali git repo while retaining all history. Here is how this was done:

  1. Go to a local git repository of the Kiali repo.

  2. Create a new branch to be used to populate the new Kiali Operator git repo: git checkout -b split-operator-master origin/master

  3. Extract only the operator files retaining all history: git filter-branch --subdirectory-filter operator

  4. Push the operator files to the Kiali Operator git repo’s master branch: git push --force -u [email protected]:kiali/kiali-operator.git split-operator-master:master

Analogous steps were done for the other branches v1.0 and v1.12.

The operator directory in the Kiali git repository is no longer used - it thus is removed and ignored, leaving room for developers to link in their working copy of their local Kiali Operator git repository.

  1. Go to a local git repository of the Kiali repo.

  2. Create a branch to create a PR with: git checkout -b remove-operator origin/master

  3. Delete the operator directory and ignore it to get it out of the way permanently: rm -rf operator && echo operator >> .gitignore

  4. Commit the change: git commit -am "Remove and ignore the operator directory"

  5. Push and create a PR from this remove-operator branch, then merge the PR to fully remove the operator from master.

How To Change Resources or Configuration

If you need to alter Kubernetes resources for either the Kiali Server or Operator, or if you need to add, remove, or modify a configuration setting, here are the places you need to check and possibly edit. You must make changes to all of these in order to support the different install mechanisms.

The Operator’s own resources are deployed either via OLM or the Operator Helm Chart. You must make changes to support them all. Today there are three sets of OLM metadata maintained in the kiali-operator repository - we call them the "golden copies" because they are the source of truth for the OperatorHub.io upstream operator, community operator, and the Red Hat OSSM Operator. The Operator Helm Chart templates are here.

The Kiali server resources are deployed by the Kiali Operator Ansible roles or by the Kiali Server Helm Chart. There are generic Kubernetes templates as well as OpenShift templates for the default (i.e. current) release as well as select previous releases.

The configuration settings for the Kiali CR and Kiali ConfigMap are documented in the kiali_cr.yaml example file and are defined in the Kiali Operator Ansible roles and in the Helm Charts, described below in more detail.

The default values for all configuration settings are found in the Kiali Operator defaults/main.yml file (there are analogous defaults/main.yml files for the select previous releases) and in the Kiali Operator Helm Chart and Kiali Server Helm Chart.

If new Kiali Server resources are added or removed, you must make appropriate changes to the Kiali Operator Remove Role as well.

Developer Checklists When Changing Resources or Configuration

Below are checklists that developers should consult when making various kinds of changes.

Maintain Backward Compatibility

Note
 Remember to maintain backward compatiblity for all supported versions. Do not make any changes if those changes would break the older supported versions. For example, if you are removing permissions from the operator roles/rolebindings within the golden copies, and if those permissions are required by the older supported versions such as v1.24, then you must not make those changes. Only when those older versions are no longer supported can you make those changes. In this case, create a github issue as a reminder to make the changes at the appropriate time in the future when those versions are no longer supported.

Run Molecule Tests

Warning
 This may not work if you have SELinux enabled. If you get permission errors, disable SELinux via setenforce 0 and try again.

When making any kind of change, you should also make the necessary changes to the Molecule tests to ensure they still pass. You can easily run the Molecule tests on your local box using Minikube by running the ci-minikube-molecule-tests.sh hack script. If you want to run just a few or one Molecule test rather than the whole suite, you can run something like this (omit CLUSTER_TYPE if running the test on OpenShift; you can omit MINIKUBE_PROFILE if your profile is the default of minikube).

CLUSTER_TYPE=minikube MINIKUBE_PROFILE=ci MOLECULE_USE_DEV_IMAGES=true MOLECULE_SCENARIO="roles-test" make clean build test cluster-push molecule-test

where roles-test is the Molecule test(s) you want to run. The value of MOLECULE_SCENARIO is a space-separated list of Molecule tests - the full list of Molecule tests are found here. There are other environment variables you can set (such as MOLECULE_DEBUG for turning on debug messages) - see the make/Makefile.molecule.mk file in the kiali/kiali repo.

Note
 If you have never run the Molecule tests before, the Molecule container image will be built for you automatically. If you need to rebuild the image for some reason, run FORCE_MOLECULE_BUILD=true make molecule-build.

Are You Altering a Kiali Operator Resource?

Are You Altering a Kiali Server Resource?

Are You Altering a Kiali Server Role Permission that Applies to Resources in All Accessible Namespaces?

Are You Altering a Kiali Server Role Permission that Applies to Resources Only in the Control Plane Namespace?

Are You Altering a Kiali Server Configuration Setting?

Are You Altering a Kiali Monitoring Dashboard Template?

Do You Need To Backport Changes To Support Older Versions?

  • ❏ Changes made to the default Ansible role should be duplicated to the versioned Ansible role (e.g. v1.24)

  • ❏ Cherry-pick changes you made in the default Ansible role and versioned Ansible roles (e.g. v1.24) to the appropriate git branch.

Are You Adding Support For a New Ansible Role Version?

Are You Removing Support For an Old Ansible Role Version?