Some tests that are "Manual" as per the text field are not marked "type: manual"

[mrajashree]

Many tests that have the word "Manual" in the text field also have "type: manual". But some of them don't, for instance test 3.2.1. Why is that so?

[mozillazg]

Many tests that have the word "Manual" in the text field also have "type: manual". But some of them don't, for instance test 3.2.1. Why is that so?

If can't audit the tests via shell script, these tests will be marked as "type: manual" otherwise it will not be marked as "type: manual".

More info about 3.2.1: #742

[jorgegutierrez2077]

The thing is it stays as manual despite it is able to check it, the only difference and reason to spot it, is that by filtering you may end with manual test marked as PASS

For example:
# kube-bench --benchmark cis-1.6 --nosummary --noremediations | grep "[PASS]" | grep "(Manual)"
[PASS] 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual)
[PASS] 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual)
[PASS] 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual)
[PASS] 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual)
[PASS] 1.2.33 Ensure that the --encryption-provider-config argument is set as appropriate (Manual)
[PASS] 1.2.34 Ensure that encryption providers are appropriately configured (Manual)
[PASS] 1.2.35 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual)
[PASS] 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual)
[PASS] 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual)
[PASS] 3.2.1 Ensure that a minimal audit policy is created (Manual)
[PASS] 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual)
[PASS] 4.1.4 Ensure that the proxy kubeconfig file ownership is set to root:root (Manual)
[PASS] 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual)
[PASS] 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual)
[PASS] 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual)
[PASS] 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual)
[PASS] 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual)
[PASS] 4.2.8 Ensure that the --hostname-override argument is not set (Manual)
[PASS] 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual)
[PASS] 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual)
[PASS] 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual)
[PASS] 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual)

So maybe a 3rd type of semi-Automated may help to spot those cases when there is a test considered that could fail and go to WARN state instead of FAILED

[yoavrotems]

Hey @mrajashree the simplest explain about it is that type: "manual" is a test that for some reason couldn't be automated (for example unpredictable output or organization specific settings).
Those will always end up with WARN state and could be solved with manual interfering from the user side, if its testing it manually or even writing the test so it will be your needs valid.
Now about the title "manual" CIS also has their own tool called CAT I figure that for their need they decided to change the division from "score/not scored" to "Automated/Manual" probably relate to CAT way of working.
We stick to CIS benchmarks and there for using those titles which most time indicate on the test manually running or automated but not always.