Trivy 0.51.0 rootfs scan throwing FATAL during vuln scan on root directory

[dmarcuccio-solace]

Description

After Upgrading Trivy to the latest v0.51.0 version, rootfs scans appear to be crashing with the below FATAL when scanning the root directory "/"

The lstat /proc/3471/fd/12: no such file or directory" error always seems to be the process of the running Trivy scan

The scan works with no issues with the previous v0.50.4 version

Can workaround this issue by adding "--skip-dirs "/proc/"" argument to the scan line

Desired Behavior

The scan should work

Actual Behavior

We are seeing the below FATAL

[root@localhost tmp]# curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /tmp/trivy v0.51.0
aquasecurity/trivy info checking GitHub for tag 'v0.51.0'
aquasecurity/trivy info found version: 0.51.0 for v0.51.0/Linux/64bit
aquasecurity/trivy info installed /tmp/trivy/trivy


[root@localhost tmp]# cd /tmp/trivy/
[root@localhost trivy]# ./trivy --debug rootfs --scanners vuln --vuln-type os --ignore-unfixed /
2024-05-03T10:17:55-04:00       DEBUG   Parsed severities       severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-05-03T10:17:55-04:00       DEBUG   Ignore statuses statuses=[0 1 2 4 5 6 7]
2024-05-03T10:17:55-04:00       DEBUG   Cache dir       dir="/root/.cache/trivy"
2024-05-03T10:17:55-04:00       DEBUG   There is no valid metadata file err="unable to open a file: open /root/.cache/trivy/db/metadata.json: no such file or directory"
2024-05-03T10:17:55-04:00       INFO    Need to update DB
2024-05-03T10:17:55-04:00       INFO    Downloading DB...       repository="ghcr.io/aquasecurity/trivy-db:2"
2024-05-03T10:17:55-04:00       DEBUG   No metadata file
45.83 MiB / 45.83 MiB [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 13.22 MiB p/s 3.7s
2024-05-03T10:17:59-04:00       DEBUG   Updating database metadata...
2024-05-03T10:17:59-04:00       DEBUG   DB info schema=2 updated_at=2024-05-03T12:11:24.085522952Z next_update=2024-05-03T18:11:24.085522672Z downloaded_at=2024-05-03T14:17:59.466619432Z
2024-05-03T10:17:59-04:00       INFO    Vulnerability scanning is enabled
2024-05-03T10:17:59-04:00       DEBUG   Vulnerability type      type=[os]
2024-05-03T10:17:59-04:00       DEBUG   Enabling misconfiguration scanners      scanners=[azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan-json terraformplan-snapshot]
2024-05-03T10:17:59-04:00       DEBUG   [nuget] The nuget packages directory couldn't be found. License search disabled
2024-05-03T10:18:00-04:00       FATAL   Fatal error
  - rootfs scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:430
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:266
  - scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:693
  - failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
        /home/runner/work/trivy/trivy/pkg/scanner/scan.go:148
  - walk filesystem:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/local.Artifact.Inspect
        /home/runner/work/trivy/trivy/pkg/fanal/artifact/local/fs.go:113
  - walk dir error:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk
        /home/runner/work/trivy/trivy/pkg/fanal/walker/fs.go:34
  - unknown error with /proc/3471/fd/12:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk.(*FS).onError.func2
        /home/runner/work/trivy/trivy/pkg/fanal/walker/fs.go:91
  - file info error:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk.(*FS).WalkDirFunc.func1
        /home/runner/work/trivy/trivy/pkg/fanal/walker/fs.go:55
  - lstat /proc/3471/fd/12: no such file or directory

Reproduction Steps

1. Download latest trivy v0.51.0
2. Run rootfs vulnerability scan on root directory

Target

Filesystem

Scanner

Vulnerability

Output Format

Table

Mode

Standalone

Debug Output

2024-05-03T10:17:55-04:00       DEBUG   Parsed severities       severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
2024-05-03T10:17:55-04:00       DEBUG   Ignore statuses statuses=[0 1 2 4 5 6 7]
2024-05-03T10:17:55-04:00       DEBUG   Cache dir       dir="/root/.cache/trivy"
2024-05-03T10:17:55-04:00       DEBUG   There is no valid metadata file err="unable to open a file: open /root/.cache/trivy/db/metadata.json: no such file or directory"
2024-05-03T10:17:55-04:00       INFO    Need to update DB
2024-05-03T10:17:55-04:00       INFO    Downloading DB...       repository="ghcr.io/aquasecurity/trivy-db:2"
2024-05-03T10:17:55-04:00       DEBUG   No metadata file
45.83 MiB / 45.83 MiB [------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 13.22 MiB p/s 3.7s
2024-05-03T10:17:59-04:00       DEBUG   Updating database metadata...
2024-05-03T10:17:59-04:00       DEBUG   DB info schema=2 updated_at=2024-05-03T12:11:24.085522952Z next_update=2024-05-03T18:11:24.085522672Z downloaded_at=2024-05-03T14:17:59.466619432Z
2024-05-03T10:17:59-04:00       INFO    Vulnerability scanning is enabled
2024-05-03T10:17:59-04:00       DEBUG   Vulnerability type      type=[os]
2024-05-03T10:17:59-04:00       DEBUG   Enabling misconfiguration scanners      scanners=[azure-arm cloudformation dockerfile helm kubernetes terraform terraformplan-json terraformplan-snapshot]
2024-05-03T10:17:59-04:00       DEBUG   [nuget] The nuget packages directory couldn't be found. License search disabled
2024-05-03T10:18:00-04:00       FATAL   Fatal error
  - rootfs scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:430
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:266
  - scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:693
  - failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
        /home/runner/work/trivy/trivy/pkg/scanner/scan.go:148
  - walk filesystem:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/local.Artifact.Inspect
        /home/runner/work/trivy/trivy/pkg/fanal/artifact/local/fs.go:113
  - walk dir error:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk
        /home/runner/work/trivy/trivy/pkg/fanal/walker/fs.go:34
  - unknown error with /proc/3471/fd/12:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk.(*FS).onError.func2
        /home/runner/work/trivy/trivy/pkg/fanal/walker/fs.go:91
  - file info error:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*FS).Walk.(*FS).WalkDirFunc.func1
        /home/runner/work/trivy/trivy/pkg/fanal/walker/fs.go:55
  - lstat /proc/3471/fd/12: no such file or directory

Operating System

CentOS7, Oracle Linux 9

Version

Version: 0.51.0
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-05-03 12:11:24.085522952 +0000 UTC
  NextUpdate: 2024-05-03 18:11:24.085522672 +0000 UTC
  DownloadedAt: 2024-05-03 14:17:59.466619432 +0000 UTC

Checklist

• Run trivy image --reset
• Read the troubleshooting

[alhails]

Confirmed same issue on Debian 12

• broken in 0.51.0, working in 0.50.4

[knqyf263]

Thanks for your report. Will fix it soon.
#6627

[knqyf263]

Created PR
#6628

[knqyf263]

Released
https://github.com/aquasecurity/trivy/releases/tag/v0.51.1

[adsick]

Hi, it looks like I'm having a similar problem. It appears when I run Trivy from inside docker container, when I run Trivy locally (native binary, no container) it's fine.

trivy rootfs --offline-scan --skip-db-update --skip-java-db-update --list-all-pkgs --pkg-types os -f cyclonedx /host

Fatal error     rootfs scan error: scan error: scan failed: failed analysis: walk filesystem: walk dir error: unknown error with /host/var/lib/docker/overlay2/196195dade206df7c73bce4129d712d49705b6811ca92b1b80e9857c1e9bef3b/merged/proc/4411/fdinfo/3: file info error: lstat /host/var/lib/docker/overlay2/196195dade206df7c73bce4129d712d49705b6811ca92b1b80e9857c1e9bef3b/merged/proc/4411/fdinfo/3: no such file or directory

I guess I can try skipping /host/var/lib/docker/, but I'm not sure if that's the right thing to do.

[adsick]

upd: created a dedicated bugreport. #7554

[jerrac]

I've been running into similar issues. Mostly inside of docker directories. Not necessarily when running inside of Docker, this specific log was from Trivy installed directly on an Ubuntu 22.04 vm.

FATAL   Fatal error rootfs scan error: scan error: scan failed: failed analysis: walk filesystem: walk dir error: unknown error with /var/lib/docker/volumes/< volume name >/_data/php/twig/< cached twig file >: file info error: lstat /var/lib/docker/volumes/< volume name >/_data/php/twig/< cached twig file >: no such file or directory

Does Trivy get a list of files and then scan them after? So if a file is temporary, and the running process removes it, then Trivy errors out?

Is there a way to tell Trivy that it shouldn't end the run when that happens? I mean, skipping directories is not exactly what you want to do when you are scanning for security issues...