DSaaS IB Issues

[philip-davis]

Testing is revealing some issues in the IB transport for DSaaS. I am going to collect these here, and break them out to individual issues as necessary.

1. MPI Integration: The IB client is almost free of MPI, but not quite separable. On the one hand, it would be nice to have the capability of running dataspaces without MPI. On the other hand, we should use MPI when it is available. MPI communication routines are optimized and well-maintained. Fixing this requires several things to be done. First, conditional compilation of MPI support for DataSpaces and DART; MPI support should be a compile-time option, rather than a run-time option. Second, we need functional application-level barrier methods to be provided by DART. This is currently half-baked in the IB transport, and not being used.

2. App registration: There is a coordination bug in app registration. If app B joins while app A is in the process of distributing connection info among its clients, this will cause a crash in app A. The issue is that non-master clients do not build their network contact table until after the initial contact data from the server is disseminated, so data structures that need to be accessed/modified are not yet built.