idea for systematic internals testing

[jti-lanl]

We could support special filenames (e.g. in a namespace that has a new kind of "testing" indicator), or maybe special xattrs that can't be installed by regular users. When handed to marfs, these filenames or xattrs would cause flags to get set in a test_mode field in the corresponding FileHandle.

Among other things, our code could then use these flags to trigger various simulated timing issues, when interacting with object-servers. This would let us systematically test many situations where e.g. a server stops sending to curl during a GET (or receiving during a PUT). We could systematically provoke every kind of timeout condition that our code recognizes.

We can test normal corner cases with regular test code. The point of these would be to add support for testing specific situations in the internals, which are otherwise difficult to recreate without going into the debugger and creating them explicitly. Some of these could even be probabilistic, or time-delayed, so that we can get a heavy load of traffic going before they occur. Conceivably, we could even set things up to allow multiple independent threads to coordinate in "bad" ways.