unit_testing.md

C++ Unit Testing Framework

Problem

The previous unit testing framework started as hack to have a bit more than simple asserts. It is not easy to use (needs explicit enumeration of all test cases) and lacks really important features (e.g. output of the assertion that failed).

Constraints

• Must be BSD licenced
• Must be easy to use
• should be portable
• should support mocking

Assumptions

Considered Alternatives

• Continue with current framework
• Boost Unit Testing framework

Decision

• Keep C framework for C tests and ABI tests
• Google Unit testing framework gtest with code downloaded by CMake for systems where no source is packaged (Debian Wheezy, Arch Linux, Fedora,...) for C++ tests

Rationale

• Having the output of current values when an assertion fails in any case
• No listing of all test cases in main (but instead having test discovery)
• No more commenting out if you only want to run parts of the test-suite
• No more typos in test-suite namings
• xUnit output for jenkins
• value and type-parameterized tests
• Mock-Support (not available in gtest?)
• setup/teardown global+per test
• supports death tests
• writing many parts of it on our own adds to the total amount of code to write and maintain.
• integrations into IDEs

Implications

• It is not ideal to have different frameworks intermixed (C vs. C++ frameworks, but most code is C).
• In the end we have to write a lot of functionality ourselves anyway (e.g. comparing Keys and KeySets).
• Testsuite execution are already handled by cmake and kdb run-all.
• The selection of tests within a test suite does not play well with ctest.
• Rewriting all current tests to have unified behavior is a lot of work.
• Might not work for ABI compatibility tests.
• Mock only by extra framework.

Related Decisions

• Script Testing

Notes

• We had discussions on Mailinglists
• We had discussions in #26