Disable `data.table` multithreading

[kenmawer]

Using multiple CPU threads with data.tables can use a lot of memory when handling large data.table`s.

For example, using 12 threads (the number used when running the code after setDTthreads(0)) instead of 1 core will use up a lot of the SSD (i.e. hard fault) due to running out of RAM. In addition, the memory gets used up after code execution, meaning that memory leaks can be especially problematic.

[brookslogan]

What is "used up" here? In some R and OS versions, R can't return any memory it allocates to the OS, but that doesn't seem to be an issue here; it looks like on the chart that the memory usage returns to about the same point as where the vertical line marking "12 threads" is.

We probably want to still allow multithreading for some archives, as it does seem to speed things up wall-time wise when spare cores are available. Maybe some sort of smart default that considers the size of the archive, how much memory they have, whether they're already multithreading, etc. I think data.table already has that last one built in: I believe it checks whether something's already doing multiprocess stuff and disables multithreading if so.

We'll need to plan this with other performance improvements and implementation changes to epi_archive. We might want to move away from data.tables altogether, or might be able to find a way to limit the size so that this RAM usage issue is no longer a problem.