Use Pytest for test execution

[schroeding]

This branch changes the test runner to pytest. Fixes #162.

[PhilippWendler]

It would be great if the final diff of the PR (before merging) is somewhat useful and readable (to allow reviewing). I know that there will inevitably be quite some changes due to the test structure and the assertions, but can you maybe aim at making these changes as localized as possible? Such that for example comparing pytest_foo.py against test_foo.py is doable.

We also want to compare the old tests against the new tests, in particular with regards to the question how many tests are executed, in order to find any overlooked things.

For this I wonder what the best thing to do is?

• Copy the test suite in a branch (as you do) such that we can always execute both? And only at the end of the project delete the old test suite and rename the new one?
• Simply replace the test suite in a branch and then compare it with an execution of the old test suite on main? This allows to continually look at the diff between the test suites using GitHub.
• Replace the test suite incrementally, module by module, after checking that the module is rewritten correctly. Makes everything smaller and easier to follow, but with the disadvantage of having two independent test suites in CI for a while.

[schroeding]

It would be great if the final diff of the PR (before merging) is somewhat useful and readable (to allow reviewing). I know that there will inevitably be quite some changes due to the test structure and the assertions, but can you maybe aim at making these changes as localized as possible? Such that for example comparing pytest_foo.py against test_foo.py is doable.

I'm not sure if the localization of the changes is the main problem here, i.e. a git diff --no-index benchexec/pytest_analyze_run_result.py benchexec/test_analyze_run_result.py returns a (IMO) usable diff. The final diff on Github is unusable as those are "new" files, as far as Github is concerned. After playing around with Github's interface, there doesn't appear to be a way to explicitly make it clear that a file is a copy (even though git itself can detect that via git log --follow --find-copies) :/

For this I wonder what the best thing to do is?

Do you have a preference? I'll do what's easiest to review for you.

Copy the test suite in a branch (as you do) such that we can always execute both? And only at the end of the project delete the old test suite and rename the new one?

Another way to do this, in hindsight a better way, would be duplicating and renaming the current testsuite, and changing the existing files to pytest. As a downside, this would again cause commits with "brand-new" files (the existing, nose-based testsuite) during development, however, as those would be deleted with the last commit before merging, they shouldn't need to be reviewed.

In this case, this may be fine though, as the only reason for duplicating the testsuite in the branch would be to make sure we don't miss any changes to the testsuite on main in the meantime (e.g. added tests which may not cause a merge conflict) and to make comparing the results between the two testsuites (especially coverage) more comfortable, right?

Without copying the testsuite (or simply not checking it into git), this would be equivalent to option 2.

[PhilippWendler]

In this case, this may be fine though, as the only reason for duplicating the testsuite in the branch would be to make sure we don't miss any changes to the testsuite on main in the meantime (e.g. added tests which may not cause a merge conflict)

I would expect that almost all changes to the test suite on main in the meantime would cause a merge conflict. And to double check it is easy to get a list of all commits touching test files.

and to make comparing the results between the two testsuites (especially coverage) more comfortable, right?

Without copying the testsuite (or simply not checking it into git), this would be equivalent to option 2.

Then we can simply do option 2, if you like that most.

[schroeding]

Does this mean that we can switch to pytest just by doing this and do not have to rewrite all test structure and all the assertions, can you confirm whether my understanding is correct?

Yes, this is exactly right. Pytest supports unittest-based testsuites and even "mixtures" where some tests are based on unittest, some are based on native assert statements.

[schroeding]

Can you please also check the amount of executed tests before and after this change, to make sure that pytest picks up the same tests and none is forgotten? Ideal would even be to compare a list of executed tests.

Lists are attached (tests_pytest.txt, tests_nose.txt) - they are indeed identical (even though formatting is a bit different). Note that we have one test more than on main, as I added a test to make sure any (existing) changes to the rounding mode are not silently overwritten (but will cause an assertion error) - thinking about it now, this test becomes redundant after #991 is fixed, and should be flipped around: External interference should not cause an assertion error, as we use a local context.

(Commands to verify the lists independently are:
python -m nose benchexec -vv --collect-only --with-print-test-files for pytest and

sed -e 's|<testcase|\
|g' -e 's|</testcase>|\
|g' nosetests.xml | grep classname | awk -F '"' '{ print $2'.'$4 }' | sort -u

for nose)

[PhilippWendler]

Great, thanks for the work and comparing the test results!

Just some minor cleanup now.

[PhilippWendler]

Thanks a lot! I am happy that this is implemented and one of our oldest open issues is resolved.