workloadconfigurationscans not Generating after Reinstalling #607

AdonisOkpidi · 2025-01-30T18:39:04Z

Overview

Reinstalled kubescape to ensure the reports are created/updated effectively when testing scheduled scanning. However the kubescape pod log contains errors (Air-gap only) and reports are not generating with a no resources found (Air-gap and non Air-gap environment) when retrieving using kubectl get workloadconfigurationscans -A.

In AKS (Air Gapped Environment) the errors focuses on Failure to download policies and Host scanner errors (no reports generated). In GKE, Frameworks is pulled successfully and host scanner installed successfully. However the reports are not generated.

AKS (Air Gapped Environment)

Failure to download policies and Host scanner

Using default frameworks scan logs

AKS filtered framework scan logs

GKE (non Air Gapped Environment)

Frameworks is pulled successfully and host scanner installed successfully. However the reports are not generated 'no resources found'.

No Resources Found

The text was updated successfully, but these errors were encountered:

AdonisOkpidi · 2025-01-31T14:31:10Z

Hello @matthyx, can you help with this please.

Did anything change as I cant access the policies in air-gapped env? I have the KS_OFFLINE variable enabled as you instructed but seems to not download successfully since a reinstall?

For GKE the scans seems to work but reports are not generated?

matthyx · 2025-01-31T14:43:41Z

Hello @matthyx, can you help with this please.

Did anything change as I cant access the policies in air-gapped env? I have the KS_OFFLINE variable enabled as you instructed but seems to not download successfully since a reinstall?

For GKE the scans seems to work but reports are not generated?

How did you reinstall kubescape-operator? Can you share the command used and/or the content of the ks-cloud-config ConfigMap?

AdonisOkpidi · 2025-01-31T15:16:04Z

I uninstalled then reinstalled using helm upgrade --install kubescape Documents/Projects/dh/helm-charts/charts/kubescape-operator/ -n kubescape -f Documents/Projects/dh/helm-charts/charts/kubescape-operator/values.yaml

ks-cloud-config

AdonisOkpidi · 2025-01-31T15:26:45Z

Here is another example of the kubescape-scheduler log when a scheduled scan request is posted.

matthyx · 2025-01-31T15:28:36Z

so this was broken in 1.25.3?
I think this is maybe the cause 3ef02bf

AdonisOkpidi · 2025-01-31T16:07:32Z

I think thats correct! I reverted to 1.25.0 and everything is back to normal now. Air gapped environment now pulling the policies and reports generated and Non air-gap environment now generating reports with kubectl get workloadconfigurationscans -A.

matthyx · 2025-01-31T16:09:52Z

ok thanks for checking, sorry for that mess, I will try to solve that next week

AdonisOkpidi · 2025-01-31T16:20:14Z

No worries at all. I appreciate you jumping in! :)

matthyx · 2025-02-03T14:58:17Z

@AdonisOkpidi I have just tried the latest helm on a kind cluster (not air gap):

helm repo add kubescape https://kubescape.github.io/helm-charts/ ; helm repo update ; helm upgrade --install kubescape kubescape/kubescape-operator -n kubescape --create-namespace --set clusterName=`kubectl config current-context` --set capabilities.continuousScan=enable

After a few minutes I get some results:

kubectl get workloadconfigurationscans -A
NAMESPACE            NAME                                                    CREATED AT
local-path-storage   serviceaccount-default                                  2025-02-03T14:56:57Z
local-path-storage   serviceaccount-local-path-provisioner-service-account   2025-02-03T14:56:57Z
local-path-storage   configmap-local-path-config                             2025-02-03T14:56:57Z
local-path-storage   configmap-kube-root-ca.crt                              2025-02-03T14:56:58Z
local-path-storage   deployment-local-path-provisioner                       2025-02-03T14:56:59Z

matthyx · 2025-02-03T14:59:57Z

I am going to check air gaped soon...

matthyx · 2025-02-03T16:40:21Z

Indeed, seems like I have an issue:

{"level":"error","ts":"2025-02-03T16:38:26Z","msg":"scanning failed","ID":"87c50ae7-4221-4858-a511-130f34456b1e","error":"failed to scan. reason: 'framework: security: framework from file not matching'"}

AdonisOkpidi · 2025-02-03T17:43:23Z

@AdonisOkpidi I have just tried the latest helm on a kind cluster (not air gap):

helm repo add kubescape https://kubescape.github.io/helm-charts/ ; helm repo update ; helm upgrade --install kubescape kubescape/kubescape-operator -n kubescape --create-namespace --set clusterName=`kubectl config current-context` --set capabilities.continuousScan=enable

After a few minutes I get some results:

kubectl get workloadconfigurationscans -A
NAMESPACE            NAME                                                    CREATED AT
local-path-storage   serviceaccount-default                                  2025-02-03T14:56:57Z
local-path-storage   serviceaccount-local-path-provisioner-service-account   2025-02-03T14:56:57Z
local-path-storage   configmap-local-path-config                             2025-02-03T14:56:57Z
local-path-storage   configmap-kube-root-ca.crt                              2025-02-03T14:56:58Z
local-path-storage   deployment-local-path-provisioner                       2025-02-03T14:56:59Z

Interesting! Just to confirm to was using 1.25.3?

matthyx · 2025-02-03T17:45:52Z

Actually 1.25.4 (lastest).

…

On Mon, Feb 3, 2025, 18:43 AdonisOkpidi ***@***.***> wrote: @AdonisOkpidi <https://github.com/AdonisOkpidi> I have just tried the latest helm on a kind cluster (not air gap): helm repo add kubescape https://kubescape.github.io/helm-charts/ ; helm repo update ; helm upgrade --install kubescape kubescape/kubescape-operator -n kubescape --create-namespace --set clusterName=`kubectl config current-context` --set capabilities.continuousScan=enable After a few minutes I get some results: kubectl get workloadconfigurationscans -A NAMESPACE NAME CREATED AT local-path-storage serviceaccount-default 2025-02-03T14:56:57Z local-path-storage serviceaccount-local-path-provisioner-service-account 2025-02-03T14:56:57Z local-path-storage configmap-local-path-config 2025-02-03T14:56:57Z local-path-storage configmap-kube-root-ca.crt 2025-02-03T14:56:58Z local-path-storage deployment-local-path-provisioner 2025-02-03T14:56:59Z Interesting! Just to confirm to was using 1.25.3? — Reply to this email directly, view it on GitHub <#607 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE5ZVENAZS5AHOTH3VUWYU32N6TFFAVCNFSM6AAAAABWF5JFYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZRGY3DCMBRGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

matthyx · 2025-02-03T20:44:37Z

@AdonisOkpidi I think I've found what I missed in the chart... can you try to set KS_OFFLINE="true" in the deployment's env for kubescape?
If that works, I will prepare a proper capability in the next chart release.

AdonisOkpidi · 2025-02-04T14:16:42Z

Non Air Gap Environment

This seems to work now in 1.25.4

I am getting errors scanning GCR images. Is the ACCOUNT_ID compulsory in this script when setting this up? https://hub.armosec.io/recipes/setup-gcp-iam-authorization-for-in-cluster-installation-of-kubescape-in-gke-1

matthyx · 2025-02-04T14:59:28Z

I am getting errors scanning GCR images. Is the ACCOUNT_ID compulsory in this script when setting this up? https://hub.armosec.io/recipes/setup-gcp-iam-authorization-for-in-cluster-installation-of-kubescape-in-gke-1

The script is stupid, you don't need to set the ACCOUNT_ID - just make sure you have the proper helm command at the end.

AdonisOkpidi · 2025-02-04T15:41:56Z

I am getting errors scanning GCR images. Is the ACCOUNT_ID compulsory in this script when setting this up? https://hub.armosec.io/recipes/setup-gcp-iam-authorization-for-in-cluster-installation-of-kubescape-in-gke-1

The script is stupid, you don't need to set the ACCOUNT_ID - just make sure you have the proper helm command at the end.

Thanks! I will give this a try and let you know if any issues

AdonisOkpidi · 2025-02-04T15:42:21Z

Air Gapped Environment

I have the KS_OFFLINE variable set to True in the kubescape pod using the 1.25.4 release but seems to be the same issue.

Bumped kubescape version and same result

matthyx · 2025-02-04T15:49:30Z

@AdonisOkpidi can you check that you actually set these 2 parameters?

--set grypeOfflineDB.enabled=true --set grypeOfflineDB.image.tag=latest

matthyx · 2025-02-04T15:50:21Z

This is ninja scripting to enable the offline mode... I promise I will add soon a capability and some docs.

AdonisOkpidi · 2025-02-04T19:31:14Z

Enabling grypeOfflineDB and image.tag seems to do the trick :).

Question: seeing this worked in 1.25.0 in offline mode anyway with grype.anchore.io and toolbox-data.anchore.io whitelisted, why is enabling in this version necessary?

I have noticed although this seems to work, some of the node-agents are crashing unusually but the logs have no errors in logs. I will look into it tomorrow and feedback any details.

matthyx · 2025-02-05T06:17:29Z

That's because of this condition that I've added

helm-charts/charts/kubescape-operator/templates/kubescape/deployment.yaml

Line 175 in faa0d13

    
           {{- if and .Values.grypeOfflineDB.enabled (eq .Values.grypeOfflineDB.image.tag "latest") }}

As I said, I will make an offline capability in the next release (which could happen today).

For node-agents, do you see OOMKills maybe?

AdonisOkpidi · 2025-02-05T18:17:55Z

Awesome! Looking forward to the update :)

Thats correct. It is a OOMKill for the nodes! I have now increased the memory and this is resolved.

I am currently trying to authenticate with GKE and seems the install are using parameters specific to kubescape/kubescape-cloud-operator (deprecated) https://hub.armosec.io/recipes/setup-gcp-iam-authorization-for-in-cluster-installation-of-kubescape-in-gke-1. I have added the service account created to

helm-charts/charts/kubescape-operator/values.yaml

Line 155 in 418bb04

gkeServiceAccount:

but doesnt seem to make any difference with scans. I will test again tomorrow incase I am missing anything.

matthyx · 2025-02-05T18:20:23Z

I am not sure how the whole authentication works... help @dwertent !
In the meantime we have a new parameter in the latest chart:

helm-charts/charts/kubescape-operator/values.yaml

Line 109 in 418bb04

kubescapeOffline: disable

AdonisOkpidi added the bug Something isn't working label Jan 30, 2025

matthyx added this to Kubescaping Feb 3, 2025

matthyx moved this to Triage in Kubescaping Feb 3, 2025

matthyx self-assigned this Feb 4, 2025

matthyx moved this from Triage to High Priority in Kubescaping Feb 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

workloadconfigurationscans not Generating after Reinstalling #607

workloadconfigurationscans not Generating after Reinstalling #607

AdonisOkpidi commented Jan 30, 2025 •

edited

Loading

AdonisOkpidi commented Jan 31, 2025

matthyx commented Jan 31, 2025

AdonisOkpidi commented Jan 31, 2025

AdonisOkpidi commented Jan 31, 2025

matthyx commented Jan 31, 2025 •

edited

Loading

AdonisOkpidi commented Jan 31, 2025

matthyx commented Jan 31, 2025

AdonisOkpidi commented Jan 31, 2025

matthyx commented Feb 3, 2025 •

edited

Loading

matthyx commented Feb 3, 2025

matthyx commented Feb 3, 2025

AdonisOkpidi commented Feb 3, 2025

matthyx commented Feb 3, 2025 via email

matthyx commented Feb 3, 2025 •

edited

Loading

AdonisOkpidi commented Feb 4, 2025

matthyx commented Feb 4, 2025

AdonisOkpidi commented Feb 4, 2025

AdonisOkpidi commented Feb 4, 2025 •

edited

Loading

matthyx commented Feb 4, 2025 •

edited

Loading

matthyx commented Feb 4, 2025

AdonisOkpidi commented Feb 4, 2025

matthyx commented Feb 5, 2025

AdonisOkpidi commented Feb 5, 2025

matthyx commented Feb 5, 2025

workloadconfigurationscans not Generating after Reinstalling #607

workloadconfigurationscans not Generating after Reinstalling #607

Comments

AdonisOkpidi commented Jan 30, 2025 • edited Loading

Overview

AKS (Air Gapped Environment)

Using default frameworks scan logs

AKS filtered framework scan logs

GKE (non Air Gapped Environment)

No Resources Found

AdonisOkpidi commented Jan 31, 2025

matthyx commented Jan 31, 2025

AdonisOkpidi commented Jan 31, 2025

AdonisOkpidi commented Jan 31, 2025

matthyx commented Jan 31, 2025 • edited Loading

AdonisOkpidi commented Jan 31, 2025

matthyx commented Jan 31, 2025

AdonisOkpidi commented Jan 31, 2025

matthyx commented Feb 3, 2025 • edited Loading

matthyx commented Feb 3, 2025

matthyx commented Feb 3, 2025

AdonisOkpidi commented Feb 3, 2025

matthyx commented Feb 3, 2025 via email

matthyx commented Feb 3, 2025 • edited Loading

AdonisOkpidi commented Feb 4, 2025

matthyx commented Feb 4, 2025

AdonisOkpidi commented Feb 4, 2025

AdonisOkpidi commented Feb 4, 2025 • edited Loading

matthyx commented Feb 4, 2025 • edited Loading

matthyx commented Feb 4, 2025

AdonisOkpidi commented Feb 4, 2025

matthyx commented Feb 5, 2025

AdonisOkpidi commented Feb 5, 2025

matthyx commented Feb 5, 2025

AdonisOkpidi commented Jan 30, 2025 •

edited

Loading

matthyx commented Jan 31, 2025 •

edited

Loading

matthyx commented Feb 3, 2025 •

edited

Loading

matthyx commented Feb 3, 2025 •

edited

Loading

AdonisOkpidi commented Feb 4, 2025 •

edited

Loading

matthyx commented Feb 4, 2025 •

edited

Loading