[GENERAL ISSUE] Shouldn't unresolved be not FAIL but ERROR in unit testing?

[yukihiko-shinoda]

Describe the issue

In case when happen unresolved, AWS Config custom policy rule behaves as SKIP.
However, AWS CloudFormation Guard judges as FAIL in unit testing.

This causes missed confirmation in production for the pattern tested in unit testing process.

Any examples

./guard-files/test.guard:

foo == 1

./guard-files/tests/test.yml:

- name: Succeed
  input:
    foo: 1
  expectations:
    rules:
      default: PASS
- name: Developer expects FAIL but mistaking Guard DSL coding in fact
  input:
    bar: 1
  expectations:
    rules:
      default: FAIL

Then, cfn-guard test -d guard-files -v:

Testing Guard File guard-files/test.guard
Test Case #1
Name: Succeed
`- File(, Status=PASS)[Context=File(rules=1)]
   `- Rule(default, Status=PASS)[Context=default]
      `- GuardClauseBlock(Status = PASS)[Context=GuardAccessClause#block foo EQUALS  1]
         `- GuardClauseValueCheck(Status=PASS)[Context= foo EQUALS  1]
  PASS Rules:
    default: Expected = PASS

Test Case #2
Name: Developer expects FAIL but mistaking Guard DSL coding in fact
`- File(, Status=FAIL)[Context=File(rules=1)]
   `- Rule(default, Status=FAIL)[Context=default]
      `- GuardClauseBlock(Status = FAIL)[Context=GuardAccessClause#block foo EQUALS  1]
         `- GuardClauseBinaryCheck(Status=FAIL, Comparison= EQUALS, from=(unresolved, Path=[L:0,C:0] Value={"bar":1}), to=)[Context= foo EQUALS  1]
  PASS Rules:
    default: Expected = FAIL

Operating System:

Debian GNU/Linux (Docker Container)

OS Version

11 (bullseye) (Docker Container)

[yukihiko-shinoda]

#355 (comment)

With the current version a single property value check will complete the exists and value check in one operation.

So, is the behavor of AWS Config wrong?

[joshfried-aws]

Hi @yukihiko-shinoda no, I don't believe this should be an error. This is consistent throughout the guard evaluation engine when using either test or validate command.

I am going to go ahead and close out this issue since this seems to be proper behaviour imo.

Feel free to reopen if need be.

[yukihiko-shinoda]

I think this was AWS Config's bug and it was fixed now.

I understood that it's proper to Gurad evaluation engine marks unresolved as FAIL.
However, AWS Config custom policy rule behaved as SKIP at least in Tokyo region until 2023-07-28.

I already asked AWS support about this issue.
They answered that, at this time, if you try to evaluate a property that is not included in the setting item with the Config custom policy rule, the behavior will not be included in the evaluation results that can be obtained with GetComplianceDetailsByConfigRule.

On 2023-07-28, AWS Config custom policy rule suddenly started to detect the resource which had skiped as noncompliant with following message:

Attempting to retrieve array index or key from map at path = /****/**** , Type was not an array/object map, Remaining Query = Level