feat(misconf): Expose misconf engine debug logs with `--debug` option #5550

simar7 · 2023-11-10T01:30:25Z

Description

Exposes useful informational debug logs from the misconf engine when the existing --debug option is passed.

./trivy --debug config  /Users/repos/trivy-issues/5395    
2023-11-09T18:21:57.612-0700    DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-11-09T18:21:57.624-0700    DEBUG   cache dir:  /Users/simarpreetsingh/Library/Caches/trivy
2023-11-09T18:21:57.625-0700    INFO    Misconfiguration scanning is enabled
2023-11-09T18:21:57.625-0700    DEBUG   Policies successfully loaded from disk
2023-11-09T18:21:57.649-0700    DEBUG   The nuget packages directory couldn't be found. License search disabled
2023-11-09T18:21:57.650-0700    DEBUG   Walk the file tree rooted at '/Users/simarpreetsingh/repos/trivy-issues/5395' in parallel
<snip>
2023-11-09T18:21:57.987-0700    DEBUG   [misconf] 21:57.987634000 terraform.executor               Initialised 484 rule(s).
2023-11-09T18:21:57.987-0700    DEBUG   [misconf] 21:57.987638000 terraform.executor               Created pool with 9 worker(s) to apply rules.
2023-11-09T18:21:57.989-0700    DEBUG   [misconf] 21:57.989147000 terraform.scanner.rego           Scanning 1 inputs...
2023-11-09T18:21:57.995-0700    DEBUG   [misconf] 21:57.995493000 terraform.executor               Finished applying rules.
2023-11-09T18:21:57.995-0700    DEBUG   [misconf] 21:57.995512000 terraform.executor               Applying ignores...
2023-11-09T18:21:57.995-0700    DEBUG   [misconf] 21:57.995528000 terraform.executor               Ignored 'google-gke-use-cluster-labels' at 'main.tf:2-5'.
2023-11-09T18:21:58.015-0700    DEBUG   OS is not detected.
2023-11-09T18:21:58.015-0700    INFO    Detected config files: 2
2023-11-09T18:21:58.015-0700    DEBUG   Scanned config file: .
2023-11-09T18:21:58.015-0700    DEBUG   Scanned config file: main.tf

main.tf (terraform)

Tests: 14 (SUCCESSES: 7, FAILURES: 6, EXCEPTIONS: 1)
Failures: 6 (UNKNOWN: 0, LOW: 1, MEDIUM: 3, HIGH: 2, CRITICAL: 0)

This can be useful for a variety of reasons, including knowing if a rule was ignored and why.

Related issues

Close feat(misconf): Improving ignore experience #5395

Related PRs

Support Inline Filtering #2961

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

Addresses: #5395 Signed-off-by: Simar <[email protected]>

knqyf263 · 2023-11-10T02:10:18Z

It's off-topic, but I just remember defsec didn't sync with the logging options in Trivy, like --quiet and --debug. Even when the Trivy logger is disabled with --quiet, the defesc logger is not disabled. It would be great if we support it in trivy-iac.

simar7 · 2023-11-10T04:18:46Z

It's off-topic, but I just remember defsec didn't sync with the logging options in Trivy, like --quiet and --debug. Even when the Trivy logger is disabled with --quiet, the defesc logger is not disabled. It would be great if we support it in trivy-iac.

Yeah you're right - I noticed that too. I think we need to better integrate the logging options between trivy and trivy-iac modules. I will create an issue for it #5551

nikpivkin · 2023-11-10T06:13:50Z

pkg/misconf/scanner.go

@@ -67,6 +68,14 @@ func (o *ScannerOption) Sort() {
 	sort.Strings(o.DataPaths)
 }

+type DebugLogger struct {
+}


Same as above

nikpivkin · 2023-11-10T06:16:44Z

pkg/cloud/aws/scanner/scanner.go

@@ -44,11 +44,11 @@ func (s *AWSScanner) Scan(ctx context.Context, option flag.Options) (scan.Result
 	}

 	if option.Debug {
-		scannerOpts = append(scannerOpts, options.ScannerWithDebug(&defsecLogger{}))
+		scannerOpts = append(scannerOpts, options.ScannerWithDebug(&DebugLogger{}))


How about passing a named logger as a field? Zap will take care of adding the path to the logger name itself

Suggested change

scannerOpts = append(scannerOpts, options.ScannerWithDebug(&DebugLogger{}))

scannerOpts = append(scannerOpts, options.ScannerWithDebug(&DebugLogger{log.Logger.Named("aws")}))

log.Logger.Named() returns a sugaredlogger which doesn't satisfy the interface conditions.

Instead I refactored the DebugLogger into the log package which would help us re-use it across other packages and added a name as a field as you mentioned.

nikpivkin · 2023-11-10T06:23:38Z

It's off-topic, but I just remember defsec didn't sync with the logging options in Trivy, like --quiet and --debug. Even when the Trivy logger is disabled with --quiet, the defesc logger is not disabled. It would be great if we support it in trivy-iac.

In this PR, the components use the Trivy global logger, so they will have the same behaviour as Trivy. I passed the -q flag and Trivy's output was clear.

knqyf263 · 2023-11-13T00:11:44Z

In this PR, the components use the Trivy global logger, so they will have the same behaviour as Trivy. I passed the -q flag and Trivy's output was clear.

Great!

knqyf263 · 2023-11-15T02:54:38Z

pkg/log/logger.go

@@ -121,3 +122,12 @@ func String(key, val string) zap.Field {
 	}
 	return zap.String(key, val)
 }
+
+type DebugLogger struct {


It looks more like PrefixedLogger, doesn't it?

Renamed: d9389f7

knqyf263

Please be mindful that I'm planning to migrate zap to slog after bumping Go to 1.21 in Trivy. I'm not sure slog meets our requirements, though.

knqyf263 · 2023-11-15T03:59:26Z

@nikpivkin Can you please review this PR?

nikpivkin

LGTM

feat(misconf): Expose debug logs with --debug option

8922aa8

Addresses: #5395 Signed-off-by: Simar <[email protected]>

simar7 self-assigned this Nov 10, 2023

simar7 changed the title ~~feat(misconf): Expose debug logs with --debug option~~ feat(misconf): Expose misconf engine debug logs with --debug option Nov 10, 2023

simar7 mentioned this pull request Nov 10, 2023

feat(misconf): Improve misconfig scanner logging experience #5551

Closed

simar7 requested a review from nikpivkin November 10, 2023 04:26

simar7 marked this pull request as ready for review November 10, 2023 04:27

simar7 requested a review from knqyf263 as a code owner November 10, 2023 04:27

nikpivkin reviewed Nov 10, 2023

View reviewed changes

refactor debuglogger into pkg/log

7a0b71a

fix lint

7603035

knqyf263 reviewed Nov 15, 2023

View reviewed changes

rename logger

d9389f7

knqyf263 approved these changes Nov 15, 2023

View reviewed changes

knqyf263 requested a review from nikpivkin November 15, 2023 03:59

nikpivkin approved these changes Nov 15, 2023

View reviewed changes

knqyf263 added this pull request to the merge queue Nov 16, 2023

Merged via the queue into main with commit 1336223 Nov 16, 2023
12 checks passed

knqyf263 deleted the misconf-debug-option branch November 16, 2023 02:50

aqua-bot mentioned this pull request May 29, 2024

release: v0.52.0 [main] #6809

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(misconf): Expose misconf engine debug logs with `--debug` option #5550

feat(misconf): Expose misconf engine debug logs with `--debug` option #5550

simar7 commented Nov 10, 2023 •

edited

Loading

knqyf263 commented Nov 10, 2023

simar7 commented Nov 10, 2023 •

edited

Loading

nikpivkin Nov 10, 2023

nikpivkin Nov 10, 2023

simar7 Nov 11, 2023

nikpivkin commented Nov 10, 2023

knqyf263 commented Nov 13, 2023

knqyf263 Nov 15, 2023

simar7 Nov 15, 2023

knqyf263 left a comment

knqyf263 commented Nov 15, 2023

nikpivkin left a comment

	scannerOpts = append(scannerOpts, options.ScannerWithDebug(&DebugLogger{}))
	scannerOpts = append(scannerOpts, options.ScannerWithDebug(&DebugLogger{log.Logger.Named("aws")}))

feat(misconf): Expose misconf engine debug logs with --debug option #5550

feat(misconf): Expose misconf engine debug logs with --debug option #5550

Conversation

simar7 commented Nov 10, 2023 • edited Loading

Description

Related issues

Related PRs

Checklist

knqyf263 commented Nov 10, 2023

simar7 commented Nov 10, 2023 • edited Loading

nikpivkin Nov 10, 2023

Choose a reason for hiding this comment

nikpivkin Nov 10, 2023

Choose a reason for hiding this comment

simar7 Nov 11, 2023

Choose a reason for hiding this comment

nikpivkin commented Nov 10, 2023

knqyf263 commented Nov 13, 2023

knqyf263 Nov 15, 2023

Choose a reason for hiding this comment

simar7 Nov 15, 2023

Choose a reason for hiding this comment

knqyf263 left a comment

Choose a reason for hiding this comment

knqyf263 commented Nov 15, 2023

nikpivkin left a comment

Choose a reason for hiding this comment

feat(misconf): Expose misconf engine debug logs with `--debug` option #5550

feat(misconf): Expose misconf engine debug logs with `--debug` option #5550

simar7 commented Nov 10, 2023 •

edited

Loading

simar7 commented Nov 10, 2023 •

edited

Loading