Espresso test issues regarding time spent on mesh tuning

[satishskamath]

Description

Mesh tuning currently takes way too long especially on large number of nodes (>= 2 nodes). We can bypass this if a prediction can be made based on a linear interpolation of mesh sizes used in low rank cases on a particular architecture.

TODO

• Predict mesh sizes on a given architecture using 3 points on a dependent reframe test of 1, 2 and 4 MPI ranks and output a linear interpolation.

[jngrad]

Further benchmarks revealed that the linear trend is only observable in dense ionic liquids. For ionic crystals, the ideal mesh size no longer depends on the number of MPI ranks in a linear manner. We opted to instead pin the mesh size and charge assignment order (cao) to values that are suitable for multiple MPI world sizes. This allows to bypass the bisection algorithm during tuning.

[jngrad]

espressomd/espresso#5017 introduces a new mechanism to restrict the range of mesh values that are sampled during tuning. This will be part of the next release of ESPResSo. This change could allow us to introduce some flexibility during tuning by timing mesh size values around the ones we currently pin, if this is desirable (run times from simulations with different mesh sizes or cao are not directly comparable since they have different MPI buffer sizes). In addition, it will simplify the determination of optimal mesh sizes in the CUDA benchmarks, where the tuning algorithm is known to be greedy and spends a lot of time sampling very large mesh sizes that are eventually discarded; here the idea would be to add an upper limit to the mesh size that is more reasonable than the one hardcoded in ESPResSo in the initial benchmark run that is used to determine the optimal P3M parameters that will be used in the testsuite.