Exit Code Control, and Output Format Options

[AErmie]

Hey all, this is part question and part suggestion.

When running Docker-Bench as part of a CI (in particular, running it from the Docker container method), shouldn't the execution exit with a non-zero code if there are any 'notes' or at least 'warnings'?

Additionally, I would like to be able to control the exit (ie. only exit 1 if there are warnings, for example). Either this, or being able to control a "pass" or "fail" as part of CI if the returned score is +/- a certain value.

Further, is there a way to include output options in specific formats (ie. JUnit, XML, SARIF, etc.), so that it is more easily consumed by CI/CD tools?

[konstruktoid]

Hi @AErmie, well notes are just informal so that shouldn't trigger anything basically in my opinion and exit 1 is triggered if the script fails.

One solution could be setting the exit code to the score if the score if not 0 or 1.

There's no way at the moment to specify any particular format, but the .log.json file should be easily parsed.

[AErmie]

Thanks @konstruktoid, I didn't see a way to reference the score result output though (I may have missed it). Any suggestions/guidance?

[konstruktoid]

if [ $(jq '.score' docker-bench-security.log.json) -le 50 ]; then echo fail; else echo pass; fi
or are you looking for the score per test?

[AErmie]

Thanks, @konstruktoid, but according to this issue, the logging feature is broken. So, I can't actually obtain the log file(s) to parse it (and react to the score).

[konstruktoid]

Yes, the public image is broken in all sorts of ways, including the logging feature (#405).

I recommend you build your own image.

[AErmie]

Thanks, @konstruktoid , but maybe I'm not following. How would building my own image (following the instructions listed here), make any difference? Wouldn't the logging output functionality still be broken?

[konstruktoid]

Because the code in the official Docker image is deprecated, so building your own image ensures that the script is up-to-date.

[AErmie]

Thanks, @konstruktoid, I got the local clone/build of DockerBench working! However, one thing appears to be different...

If I am using the Run with Docker steps, where it pulls the (broken) Docker Hub image, etc., I am able to include the -t (target) command argument, and run Docker Bench against a specific container image (and not all the images on the host).

Note: The -t target command is mentioned in the official Docker Hub image (under Docker Bench for Security options), but not in the GitHub repo Docker Bench for Security options documentation.

But, when I follow the self-build option (to produce a working image), it does not allow the use of the -t (target) command argument. It just prints out the standard info (aka "this is how you're supposed to use this").

Docker Bench for Security - Docker, Inc. (c) 2015-2021
Checks for dozens of common best-practices around deploying Docker containers in production.
Based on the CIS Docker Benchmark 1.3.1.

Usage: docker-bench-security.sh [OPTIONS]

Example:
  - Only run check "2.2 - Ensure the logging level is set to 'info'":
      sh docker-bench-security.sh -c check_2_2
  - Run all available checks except the host_configuration group and "2.8 - Enable user namespace support":
      sh docker-bench-security.sh -e host_configuration,check_2_8
  - Run just the container_images checks except "4.5 - Ensure Content trust for Docker is Enabled":
      sh docker-bench-security.sh -c container_images -e check_4_5

Options:
  -b           optional  Do not print colors
  -h           optional  Print this help message
  -l FILE      optional  Log output in FILE, inside container if run using docker
  -u USERS     optional  Comma delimited list of trusted docker user(s)
  -c CHECK     optional  Comma delimited list of specific check(s) id
  -e CHECK     optional  Comma delimited list of specific check(s) id to exclude
  -i INCLUDE   optional  Comma delimited list of patterns within a container or image name to check
  -x EXCLUDE   optional  Comma delimited list of patterns within a container or image name to exclude from check
  -n LIMIT     optional  In JSON output, when reporting lists of items (containers, images, etc.), limit the number of reported items to LIMIT. Default 0 (no limit).
  -p PRINT     optional  Print remediation measures. Default: Don't print remediation measures.

Complete list of checks: <https://github.com/docker/docker-bench-security/blob/master/tests/>
Full documentation: <https://github.com/docker/docker-bench-security>
Released under the Apache-2.0 License.

Am I missing something? Is the -t target command no longer supported? If not, could we possibly consider putting it back in? It is super useful to run against a specific image (especially in CI/CD on shared build hosts).

[konstruktoid]

The -t command was removed 2 years ago (c53157e) and was replaced by the -i and -x options.

[AErmie]

LOL thanks @konstruktoid, not sure how I missed that!

[konstruktoid]

Closing due to inactivity.

[PeterDaveHello]

I'd like to revisit the discussion on exiting with a non-zero code when warnings are present, especially in CI environments. Considering the importance of immediately identifying issues for prompt action, an exit code reflective of the presence of warnings could significantly enhance the utility of Docker-Bench in CI pipelines. Could we consider reopening this issue to further explore potential implementations or alternatives that could support this functionality?

[konstruktoid]

@PeterDaveHello reopened :)