feat: OU-571 Add Perses and Incidents Feature Flags

[zhuje]

JIRA Issue

https://issues.redhat.com/browse/OU-571

Description

• Adjust COO to include the perses-dashboards flag in the arguments for the monitoring-console-plugin's Deployment; we then can use this flag to render the Perses Dashboard component.
• Add perses proxy information to the list of proxies in the monitoring-console-plugin's ConsolePlugin; this allows the monitoring-console-plugin to proxy to perses an instance.
• Add incidents feature flag to monitoring-console-plugin Deployment ("container.args: "-features='incidents'").

Figure 1. The first screen shows UIPlugin configuration for perses, incidents, and acm. The second screen shows the `--feature='perses-dashboards', 'incidents', 'acm-alerting',` being listed in the Deployment for the monitoring-console-plugin. The third screen shows the `alias: perses` being listed as a proxy within the ConsolePlugin CR for the monitoring-console-plugin

Changes

1. Update schema to include new fields acm, perses, and incidents. In the future, we could add +kubebuilder validations/webhooks to validate the UIPlugin custom resources for type: monitoring.

• bundle/manifests/observability.openshift.io_uiplugins.yaml
• deploy/crds/common/observability.openshift.io_uiplugins.yaml
• types.go

2. Remove ACM-related validation. This is moved to monitoring.go.

• pkg/controllers/uiplugin/compatibility_matrix.go
• pkg/controllers/uiplugin/compatibility_matrix_test.go

3. Validate configuration from UIPlugin CR. If valid, then add a feature flag(s) (e.g., "-feature= 'perses-dashboards', 'incidents'") and proxies (for perses).

• pkg/controllers/uiplugin/monitoring.go
• pkg/controllers/uiplugin/monitoring_test.go

Testing

Cluster Environment Requirements

• OCP v4.18+ ("incidents" feature flag is set only if OCP 4.18+)
• ACM v2.11+ and instance of multiclusterhub ("acm-alerting" feature flag is only set if acm v2.11+)

0. Login into a OCP v4.18+ cluster ("incidents" feature flag is set only if OCP 4.18+)

1. set openshift.enabled to true in main.go

# main.go 
...
	flag.BoolVar(&openShiftEnabled, "openshift.enabled", true, "Enable OpenShift specific features such as Console Plugins.")
...

2. Build the image and push it to your quay.io repo (and regenerate CRD from types.go with make generate and make bundle)

make generate && \
make bundle && \
make operator-image bundle-image operator-push bundle-push  \
    IMAGE_BASE="quay.io/jezhu/observability-operator" \
    VERSION=1.1.0-dev-0.35.0

3. Deploy to cluster (and clean up components from previous observability-operator deploys)

oc delete catalogsource observability-operator-catalog -n openshift-operators && \
operator-sdk cleanup observability-operator -n openshift-operators && \
operator-sdk run bundle \
    quay.io/jezhu/observability-operator-bundle:1.1.0-dev-0.35.0 \
    --install-mode AllNamespaces \
    --namespace openshift-operators \
    --security-context-config restricted

7. Install Advanced Cluster Management (this enables the feature flag "--features=acm-alerting")

• Go to OpenShift Container Platform UI
• Go to the Menu > Administrator Perspective > Operators > OperatorHub
• Search for "Advanced Cluster Management for Kubernetes", and install a version that is > v2.11
• Create a the required MultiClusterHub

8. Apply UIPlugin configuration for monitoring

oc apply -f - <<EOF
apiVersion: observability.openshift.io/v1alpha1
kind: UIPlugin
metadata:
  name: monitoring
spec:
  type: Monitoring
  monitoring:
    acm:
      enabled: true
      alertmanager:
        url: 'https://alertmanager.open-cluster-management-observability.svc:9095'
      thanosQuerier:
        url: 'https://rbac-query-proxy.open-cluster-management-observability.svc:8443'
    perses:
      enabled: true
      serviceName: "perses-api-http"
      namespace: "perses"
    incidents:
      enabled: true
EOF

Troubleshooting

mac specific troubleshooting

sed commands need to be adjusted on macOS update sed ... -i to sed ... -i '' .

# Makefile
...
.PHONY: bundle
...
# before 
	sed -e 's|<IMG_OBSERVABILITY_OPERATOR>|$(OPERATOR_IMG)|g' \
		-i bundle/manifests/observability-operator.clusterserviceversion.yaml
# after
	sed -e 's|<IMG_OBSERVABILITY_OPERATOR>|$(OPERATOR_IMG)|g' \
		-i  '' bundle/manifests/observability-operator.clusterserviceversion.yaml
...

Alternatively, install https://formulae.brew.sh/formula/gnu-sed and add it to your path to replace "sed"

GNU "sed" has been installed as "gsed".If you need to use it as "sed", you can add a "gnubin" directory to your PATH from your bashrc like:     PATH="$HOMEBREW_PREFIX/opt/gnu-sed/libexec/gnubin:$PATH"

If uninstall is hung

1. oc edit crd uiplugins.observability.openshift.io
   n the editor, find the finalizers field under metadata, and remove any finalizers (it will look something like this):

metadata:
  finalizers:
  - kubernetes

After removing the finalizer(s), save and exit the editor. This should allow the CRD to be deleted.

2. Go the the UI > Installed Operator > manually delete the operator

VSCode Local Debugger configurations

# launch.json
{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Run",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "/Users/jezhu/Git/observability-operator/cmd/operator", 
            "args": [
                "--namespace=openshift-operators",
                "--metrics-bind-address=:8080",
                "--images=alertmanager=alertmanager=quay.io/prometheus/alertmanager:v0.26.0",
                "--images=prometheus=prometheus=quay.io/prometheus/prometheus:v2.49.1",
                "--images=thanos=thanos=quay.io/thanos/thanos:v0.33.0",
                "--openshift.enabled",
            ],
            "showLog": false,
            "cwd": "/Users/jezhu/Git/observability-operator/cmd/operator",
            "env": {},
        }
    ]
}

Video Demo (Requires Red Hat SSO)

These videos do not show the initial setup. The setup requires the COO to install an image that includes changes from this PR, AMC installed, and the MultiClusterHub instance created.

Figure 2. Video demo of configuring incidents feature flag, perses feature flag, and perses proxy.
https://drive.google.com/file/d/1ub34XoG8gW6AIBfQyPGC1m6XYQdheVzV/view?usp=sharing

Figure 3. Video demo of testing updating UIPlugin CR
https://drive.google.com/file/d/1Bb-Ji03wXk8L8LbwRrOBVYm1sbUc-ISa/view?usp=sharing

[zhuje]

/hold
I pushed some changes, but they must be tested on a cluster.

[zhuje]

/unhold

tested on cluster

[jan--f]

This looks very good overall, thanks for the detailed PR description!

One question: Iiuc the plan is currently to add the Perses operator as a component to COO (instead of productizing it). That means that we'll add the operator deployment to the COO bundle instead of relying on other install mechanisms.

Is the plan to add this to this upstream project as well, similar to how we handle our prometheus-operator fork? This doesn't block this PR here, but would be good to get clarity over.

[jgbernalp]

This looks very good overall, thanks for the detailed PR description!

One question: Iiuc the plan is currently to add the Perses operator as a component to COO (instead of productizing it). That means that we'll add the operator deployment to the COO bundle instead of relying on other install mechanisms.

Is the plan to add this to this upstream project as well, similar to how we handle our prometheus-operator fork? This doesn't block this PR here, but would be good to get clarity over.

Yes, the plan is to include the deployment of the controller and the APIs similar to what we currently have with prometheus operator.

[jgbernalp]

/lgtm

[jgbernalp]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jgbernalp, zhuje

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jgbernalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment