Narrow the set of files checked by compaction commit conditional mutation

[keith-turner]

Describe the bug

For a popular tablet with lots of compactions and bulk imports its set of files is rapidly changing. The compaction commit code reads all the tablets files does some checks against those files and then submits a conditional mutation that has a condition that tablets set of files is exactly the same as what it read earlier. On a busy tablet this condition will often fail causing the entire process to retry and fail a bunch of times before finally succeeding.

Expected behavior

The code should be modified to make a more narrow check. The condition does not need to require all the tablets files to be same. The condition on the mutation could instead check that the tablet must have the files involved in the compaction w/o caring about what other files the tablet might have. This more narrow check would make the conditional mutation much less likely to conflict with other changes being made by concurrent bulk imports and compaction to a tablet.

[keith-turner]

To fix this probably need to add a method like requireFiles(Set<StoredTabletFile> files) to ample. After adding that could survey the code to see if other places could benefit from a more narrow check.

[cshannon]

I can take a look at this issue later this week

[keith-turner]

In addition to causing thread contention this issue has the potential to cause memory problems. Ran into the following problem.

Was using the changes in apache/accumulo-testing#287 to generate bulk imports. This bug in 287 apache/accumulo-testing#288 caused some bulk imports to a single tablet to have lots of files. Did not have the 288 bug fix in testing and ended up with a tablet w/ 25K files. There were lots of concurrent compactions running against this tablet. When these finished they would create a conditional mutation which included the 25k files in a condition. These conditional mutations would cause lots of memory pressure on the tablet server.