Unexpected errors on re-run in test kitchen

[NickRycar]

That's a fun issue title! Let's elaborate some.

When running out of test kitchen in any version later than 1.1.1 (tested and confirmed failure pinned to 1.2.0), there is different behavior when re-running on an already created instance. The first run will always produce the expected result. Attempting to re-run verify on an instantiated instance (whether or not converge has been run again), it will error out when iterating over listening ports like so:

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #verify action: [comparison of NilClass with 3306 failed] on default-ubuntu-1404 >>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

That all said, if you destroy the instance, and then re-run through verify, it will behave as expected and pass/fail based on currently set cihpers.

The cookbook this has been experienced with can be found here: https://github.com/chef-cft/bjc/tree/master/cookbooks/bjc-ecommerce

Kitchen defaults use a locally stashed version of the code that is confirmed working as expected, but outdated. To produce above error, the verifier should be pointed directly to github like so:

    verifier:
      inspec_tests:
        - name: ssl-baseline
          url: https://github.com/dev-sec/ssl-baseline/archive/master.zip

It's a demonstration cookbook with some baked-in SSL insecurities used to evaluate and remediate. Generally speaking, the workflow is: kitchen verify (expecting failures) -> uncomment appropriate lines in the ssl configuration template -> kitchen converge -> kitchen verify (generates above failure). Running kitchen test, or kitchen destroy + kitchen verify, will produce the expected results instead.