some shard operations should be parallel

[jti-lanl]

Several ops work across all shards. For example, when a new sharded directory is created, we have to create that directory on all the shards. Another example is readdir(), where we can read from shards concurrently.

These ops should add a thread pool to MarFS_FileHandle and MarFS_DirHandle (perhaps with a vector of fds, or MDAL contexts). Maybe we want to be able to configure the number of threads used for such parallel shard ops? Or just default to the number of shards.

There's another idea we discussed, where e.g. stat of a sharded dir uses st.st_size to return the number of shards, so a client can run its own parallel readdir, partitioned to match the individual shards. (And we'd also have to understand the offset provided in readdir() as referring to the shard to use?) I think this approach is in addition to the default mode, where readdir() spins up threads to read concurrently.

This idea probably shouldn't extend as far as e.g. read(). Let clients handle concurrent reads.

[brettkettering]

Not started.