Support for validation of cpu vulnerabilities (#114) needs further consideration...

[stuart-shelton-contino]

Describe the bug

In closing-off issue #114, an additional check was added to search for the string [vV]ulnerable in files beneath /sys/devices/system/cpu/vulnerabilities - however, to be an effective test more nuance is needed.

The current checks are all based on the content of files which are present - but there appears to be no validation of files which must be present for the issue to be mitigated, but which aren't. It could be argued that it's unreasonable to assess an older image against flaws which weren't known when it was created - but that's exactly what's currently being done by reporting failures for hardware issues which can't be (entirely) fixed in software, such as the outstanding issues with Intel's SMT implementation.

See the Additional context section for further discussion.

Expected behavior

Failures are flagged (only) when a software mitigation is not present or not correctly deployed.

Actual behavior

Unfixable hardware errors are reported as failures, the opportunity exists for missing software mitigations to go unreported.

Amazon Linux 2, Ubuntu 20.04, Ubuntu 18.04:

amazon-ebs:   ×  os-12: Detect vulnerabilities in the cpu-vulnerability-directory (3 failed)
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/ is expected to be directory
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/spectre_v2 content is expected not to match "vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/spectre_v2 content is expected not to match "Vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/itlb_multihit content is expected not to match "vulnerable"
amazon-ebs:      ×  File /sys/devices/system/cpu/vulnerabilities/itlb_multihit content is expected not to match "Vulnerable"
amazon-ebs:      expected "KVM: Vulnerable\n" not to match "Vulnerable"
amazon-ebs:      Diff:
amazon-ebs:      @@ -1 +1 @@
amazon-ebs:      -Vulnerable
amazon-ebs:      +KVM: Vulnerable
amazon-ebs: 
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/mds content is expected not to match "vulnerable"
amazon-ebs:      ×  File /sys/devices/system/cpu/vulnerabilities/mds content is expected not to match "Vulnerable"
amazon-ebs:      expected "Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\n" not to match "Vulnerable"
amazon-ebs:      Diff:
amazon-ebs:      @@ -1 +1 @@
amazon-ebs:      -Vulnerable
amazon-ebs:      +Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
amazon-ebs: 
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/l1tf content is expected not to match "vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/l1tf content is expected not to match "Vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/spec_store_bypass content is expected not to match "vulnerable"
amazon-ebs:      ×  File /sys/devices/system/cpu/vulnerabilities/spec_store_bypass content is expected not to match "Vulnerable"
amazon-ebs:      expected "Vulnerable\n" not to match "Vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/tsx_async_abort content is expected not to match "vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/tsx_async_abort content is expected not to match "Vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/spectre_v1 content is expected not to match "vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/spectre_v1 content is expected not to match "Vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/srbds content is expected not to match "vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/srbds content is expected not to match "Vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/meltdown content is expected not to match "vulnerable"
amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/meltdown content is expected not to match "Vulnerable"

Ubuntu 16.04 also reports:

amazon-ebs:      ✔  File /sys/devices/system/cpu/vulnerabilities/tsx_async_abort content is expected not to match "vulnerable"
amazon-ebs:      ×  File /sys/devices/system/cpu/vulnerabilities/tsx_async_abort content is expected not to match "Vulnerable"
amazon-ebs:      expected "Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown\n" not to match "Vulnerable"
amazon-ebs:      Diff:
amazon-ebs:      @@ -1 +1 @@
amazon-ebs:      -Vulnerable
amazon-ebs:      +Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
amazon-ebs: 

OS / Environment

Amazon EC2 executing roles against Ubuntu 16.04LTS, 18.04LTS, 20.04LTS, Amazon Linux 2, CentOS 7 stock images - all of which fail since #138 was committed recently.

Inspec Version

4.23.15-1

Baseline Version

amazon-ebs: Profile: DevSec Linux Security Baseline (linux-baseline)
amazon-ebs: Version: 2.6.0

(from https://github.com/dev-sec/linux-baseline/archive/master.tar.gz as-of 20201208)

Additional context

Especially when validating a system image which might be deployed to a range of hardware, I would suggest that the tests performed should limit themselves to reporting failure cases where a mitigation exists (or make this an option) - the current checks will universally fail on certain (cloud) hardware, simply because there is no possible fix for that hardware (other than not to use it!). I would suggest that this security profile should try to limit itself to ensuring that all possible software work-arounds are deployed, where they exist.

Even this could be improved - for example, https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/Variant4 states that a system is vulnerable if /sys/devices/system/cpu/vulnerabilities/spec_store_bypass doesn't exist - the current checks are only examining those files which do exist. In this case it seems that there a significant performance benefits to allowing Speculative Store, and so EC2 appears to continue to offer this facility to guest OS' - in which case, possibly this should be flagged as a warning rather than a failure? There doesn't seem to be any possible workaround a guest is able to employ, in any case.

It also appears that the mds error cannot be fixed within a VM - suggest passing the test if SMT Host state unknown is present (or operation within a virtualised environment by other means can be confirmed).

The itlb_multihit mitigation (https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/multihit.html#itlb-multihit-system-information) appears to be solely focused on KVM - so if that module is not present (or not loaded?) on the system, then it appears that this test should also pass.

Finally, comparing the output from different images, older kernels lack vulnerability nodes for itlb_multihit, srbds, tsx_async_abort - which I take to mean that these systems have no mitigations in place, but too old a kernel to report this.

I would suggest that the CPU Vulnerability checks should have a list of expected files and should alert if any of these is missing - but should also account for scenarios where the test is being executed on hardware not under the control of the user, and not treat mitigations that cloud providers have chosen not to apply as fatal errors. It feels as if there is the need for a 'warning' category of feedback here which isn't as strong as an error, but which is still notified to the user.

[rndmh3ro]

Hey @stuart-shelton-contino,

thanks for this very thorugh issue!
We see that we have to do some more here.

We're starting with the easy steps: ignoring the tests on docker: #141

In the meantime I'd suggest you skip the control.

[micheelengronne]

You should be able to use #141 result in 2.6.0 release (just fresh made).

[akosveres]

I echo the above sentiment, the only solution right now is to only include the 2.5.0 tests in test kitchen as I'm running on EC2. For anyone else, the syntax is:

suites:
  - name: linux-baseline
    verifier:
      inspec_tests:
        - name: linux-baseline
        # Tests fixed to version 2.5.0 as those introduced CPU vulnerability tests and these do not have a fix from AWS yet.
          url: https://github.com/dev-sec/linux-baseline/archive/2.5.0.zip

[rndmh3ro]

Not the only solution. You can also just include the tests you want and not include the cpu vulnerabilities test.

[akosveres]

I was searching for such a solution, can you provide with any code example, please?

[rndmh3ro]

Sure. I had to search, too. Take a look here: https://github.com/dev-sec/ansible-collection-hardening/blob/356d6ddb9d1048bdbb18bc1067ccbb76b61bd2ce/.travis.yml#L63

[akosveres]

I'm back 3 years later to say that there's a way of including all the controls you want in test-kitchen as well, just follow the docs https://github.com/inspec/kitchen-inspec#custom-outputs

(would love to have an exclude list, that would make it so much better...)