generalizing riak_test [JIRA: RIAK-2092]

[Licenser]

Hi,
I have played a bit with riak_test and noticed it's very tightly tied to riak_kv, which I personally think is a bit of a shame. The framework itself is awesome and I think could be used far beyond the realm of riak_kv, it's qutie easy to adapt for riak_core based applications but I think it would be posible to generalize/moduralize it to a point where it would be useful for testing any kind of distributed erlang application.

That said it's not a blind 'please do this for me NOW!' feature request I'm planning to invest effort into this given there is a general interest into this and a decent chance to get a setup like this merged.

What I was thinking about is making riak_test somewhat of a library with 'extending' libraries that specialise on certain aspects.

A rough idea would be something like this:

• riak_test
  • provides the base infrastructure of running tests, logging validating
  • provides methods to run shell commands, remote commands on nodes, spawn nodes etc.
• riak_core_test
  • requires riak_test
  • provides some general tests that should work on all riak_core applications
  • provide functions to manipulate riak_core applications (build the ring, join nodes, take them down etc)
• riak_kv_test
  • requires riak_core_test
  • provides riak_kv tests
  • provides functions specific to riak_kv

It would then be nice to have something like a riak_test rebar plugin which sets up a basik testing framework in which a user then can just require riak_core_tests or riak_kv_tests and build additional settings based on that. It would allos allow later on to make some more libraires that could dynamically be included that abstract over other aspects of testing.

Cheers,
Heinz

[DeadZen]

+1. I agree there should be a focus on testing riak_core at riak_test's base, and riak_kv as an extension.
Testing riak_core scenarios extensively is somewhat painful (such is distributed computing),
A testing framework for the unique kinds of testing required by riak_core applications would be wonderful.

[Licenser]

Okay I did a bit of experimenting and came up with a first minimal working version of this refactoring mainly a libraryfied version of riak_test and a rebar plugin to create new suites:

https://github.com/Licenser/riak_test_core
https://github.com/Licenser/rebar_riak_test

and an example suite created from this:

https://github.com/project-fifo/sniffle_test

[Licenser]

Moved the original tests to a own repository:
https://github.com/Licenser/riak_kv_test

I've not tested them since I've not set up the infrastructure to test riak_kv but those should be mostly working just the backend stuff is not implemented since I'm not yet 100% sure how to best handle them w/o including them in the core riak_test part I'll update the repo to handle it.

Cheers,
Heinz

[darach]

+1 I've just started following Heinz's recipe as we're adopting riak_core for a project. Contributing fixes/patches would be easier if the core framework was more easily reusable.

[cmeiklejohn]

We've adapted riak_core for both SyncFree's antidote and lasp on the master branch. I'm curious what work is not finished that would need to be.

[darach]

Good question. A little documentation would go a long way. Heinz's blog article is useful, for example. I have fairly vanilla riak_test working with adding a custom harness. Some of the tests would be nice to reuse. For example, the ensemble tests depend on riak_kv and a riak_kv specific ensemble backend. A subset that targeted riak_ensemble over riak_core with the default backend would be useful too.

On the doc front. An example would be the basic_command_line.erl. As I'm using custom start scripts I had to make a few changes to my riak_core based app to make it more compatible with the expectations of riak_test. As riak_test is asserting sensible expectations for riak it made sense to adopt and adapt those so that the habits of a riak user would 'just work' with the app we're building out.

[darach]

PR #742 is an example of some of the minor twiddles and tweaks that were needed in my case.

[cmeiklejohn]

Cool; would love to chat more about this. I can also take a look at #742. For what it's worth, very minimal changes were needed to make Lasp work with riak_test: #609

[darach]

Apart from #742 that was my experience too.

Actually, I've made changes that are in a similar vein to #609. As my app is relx-based it has a very different layout to a typical riak_core based app (history: it was client-server until recently, now it can run as either client-server or distributed as a deployment option). I can prepare a PR for comment if that would help to visualize what/why etc..

But apart from that (ha ha!)... I only had to add a custom rtdev harness. 😁

[DeadZen]

+1 riak_core itself cannot improve very much or very quickly without this being made a priority