trunk_has_vacancy assertion failure

[rosenhouse]

Describe the bug

I'm seeing this assertion failure on a bulk insert of keys in YCSB test runs

Assertion failed at src/trunk.c:3333:trunk_memtable_incorporate(): "trunk_has_vacancy(spl, root, 1)".

Reproduction steps

I've got a minimal reproducer in the branch gabe/issue-443-crash-repro

git fetch --all
git checkout gabe/issue-443-crash-repro
make clean
make BUILD_MODE=debug

# download the test data
curl -o test-keys.txt --compressed https://splinterdb-test-data.s3.us-west-1.amazonaws.com/gabe-crash-test-keys.txt

build/debug/bin/unit/splinterdb_key_size_test

Expected behavior

test should pass

Additional context

The crash is sensitive to several things, including the exact keys used, the max_key_size option in the splinterdb_config and the size of the values being inserted.

For example, when I used the built in random.h to create the keys, I couldn't reproduce the problem.

You can play with the environment variables MAX_KEY_SIZE and VALUE_SIZE to see how it changes.

For example,

VALUE_SIZE=100 build/debug/bin/unit/splinterdb_key_size_test

passes OK.

[rosenhouse]

I played with this a bit more today. Some more observations:

1. With this current branch, MAX_KEY_SIZE=103 build/debug/bin/unit/splinterdb_key_size_test will pass
2. But if I double up the key file

   cat test-keys.txt test-keys.txt > test-keys-2.txt

   and then

   KEY_FILE=test-keys-2.txt MAX_KEY_SIZE=103 build/debug/bin/unit/splinterdb_key_size_test

   It will fail again with the same assertion after 900000 inserts.
3. This is unchanged on main (i.e. recent commits haven't changed behavior)