[CNEUR-379] Use /dev/shm as a cache in multinode simulations

[iomaganaris]

Currently in multinode simulations without /dev/shm all node write the coreneuron_input files in the same folder. This hurts a lot the GPFS performance for all users while this is happening.
Instead of all ranks writing to GPFS this PR adds a CACHE mode for /dev/shm where the coreneuron_input data are first staged to /dev/shm and then written by a single run in different folders in GPFS named coreneuron_input_{node_id}.
Then there symlinks are created from coreneuron_input_{node_id}/*_{1,2,3}.dat to /dev/shm/.../coreneuron_input and then CoreNEURON launches the simulation using /dev/shm/.../coreneuron_input.
The only drawback of this approach is that there extra memory needed to use the /dev/shm cache since everything we dump to /dev/shm is saved in the RAM of the node

[sergiorg-hpc]

(Pasting from Slack)

Thanks to a reminder by @1uc, I completely forgot that I discussed with
@iomaganaris about generating the output in subfolders on GPFS, and that he already had something working recently 😅. Hence, I made some changes to improve it a bit by dividing the coreneuron_input directory into:

coreneuron_input/cycle_X/group_Y/node_Z 

Being cycle_X the current cycle in the model instantiation, node_Z the node ID (i.e., from 0 to 799 in the 800-node simulation), and group_Y a simple division to group the set of nodes in subfolders dividided by 20 (i.e., group_id=floor(node_id / 20)). Why 20? Well, a magic number just to divide the amount of folders into something reasonable.

With this simple approach, inside the coreneuron_input there would be at most 32 folders (i.e., one per cycle). Inside each cycle folder, there would be at most 40 subfolders (i.e., 800/20 = 40, one per subset of nodes). Inside the group of nodes, there would be at most 20 subfolders corresponding to the node IDs (i.e., from 0 to 19 in the first group, 20 to 39 in the second, and so on). Finally, inside the specific node folder, there would be at most 120 files from CoreNEURON (i.e., 3 files_per_rank x 40 ranks_per_node = 120).

In other words, we go from a folder with 3.1M files, to a tree that is much more manageable by GPFS and IME, regardless of the target file system that we use.

[sergiorg-hpc]

Note that there has been a slight update in the code since this comment was written, but the reasoning is still valid.

[iomaganaris]

Shouldn't we delete the symlinks in /dev/shm and folders in GPFS in the end? @sergiorg-hpc

[iomaganaris]

If I'm not mistaken @sergiorg-hpc has been working in an improved version of this fix so this PR can now be closed?

[iomaganaris]

We discussed about this change with Sergio offline and we concluded that it's something that it's not necessary for MMB simulations but might be beneficial so until needed this can be stayed open and Sergio can continue working on it if necessary