`invalid IPC handle` when using CUDA-Aware MPI

[cwpearson]

It may be the case that CUDA-Aware MPI stuff is initialized during MPI_Init, so cudaSetDevice must be called before that. Furthermore, cudaSetDevice may need to be called before MPI_Send.

To this end, we can probably only use this with one GPU per rank.

https://devblogs.nvidia.com/benchmarking-cuda-aware-mpi/

[cwpearson]

added cudaSetDevice before MPI_Isend and MPI_Irecv in CudaAwareMpiSender in e9ba952. It's possible this will fix the issue.

[cwpearson]

This fixes the issue for one rank per node, or one rank per GPU, but otherwise multiple GPUs per rank does not work