Undefined steps does not fail a test

[jacek99]

When I created a scenario and had no steps, it gave me the UNDEFINED error in the output...but all the tests were flagged as passed and nose displayed OK at the end.

Also a 0 return code was returned from the process instead of 1 (or some error code) to indicate failure.

This means it's hard for automated builds to know if the tests actually failed or not, since a test that was not run due to undefined steps does not fail the test run.

if that is the default behavior, freshen should offer an option to override it.

[rlisagor]

Agreed, an option to make it fail if there are undefined steps could be useful. Cucumber allows this with the --strict flag. Adding to the 1.0 milestone.

[dannystaple]

One hack to use this in CI/build system would be to pipe the output (or tee the output) to a file, then this:

grep UNDEFINED nose.log && (echo "ERROR: Undefined found"; exit 1)

[dexterous]

Or simply grep UNDEFINED nose.log; [ $? -ne 0 ]

[jacek99]

That is exactly what we had to do. I wrote a Maven plugin to trigger Freshen runs for our Java apps and we had to add custom logic like that:
https://github.com/jacek99/maven-python-mojos/blob/master/maven-python-mojos/maven-bdd-plugin/src/main/java/com/github/mojo/bdd/NoseMojo.java

[dannystaple]

Ah yes- this would work fine with a Jenkins job or in a build/deploy
script. With bash, you could probably simplify it to:

grep UNDEFINED nose.log && exit 1

On 5 January 2012 14:55, Jacek Furmankiewicz
[email protected]
wrote:

That is exactly what we had to do. I wrote a Maven plugin to trigger Freshen runs for our Java apps and we had to add custom logic like that:
https://github.com/jacek99/maven-python-mojos/blob/master/maven-python-mojos/maven-bdd-plugin/src/main/java/com/github/mojo/bdd/NoseMojo.java

---

Reply to this email directly or view it on GitHub:
#16 (comment)

Danny Staple

Director, ODM Solutions Ltd
w: http://www.odmsolutions.co.uk
Blog: http://orionrobots.co.uk/blog1-Danny-Staple

[abeld]

See abeld@3bca046 for an implementation of the "add an option to turn undefined steps into failures" feature. (The branch is https://github.com/abeld/freshen/tree/undefined-steps-are-errors--option)