ofi: follow user specified include/exclude list to select providers #12234
Conversation
wenduwan
force-pushed
the
ofi_provider_include_exclude
branch
2 times, most recently
from
c131639 to
2c348b1
Compare
|
@BrendanCunningham i don't know if opx supports FI_HMEM but esp. if it does can you double check this? |
This PR addresses open-mpi#12233 Since 5.0.x, we introduced an optional FI_HMEM capability in ofi provider selection logic(both mtl and btl) in order to support accelerator memory. As described in the issue, this introduced a bug that can cause the wrong ofi provider to be selected, even if the user explicitly includes/excludes the provider name. This change refactors the selection logic to correctly handle the include/exclude list, and therefore fixes the bug. Signed-off-by: Wenduo Wang <[email protected]>
wenduwan
force-pushed
the
ofi_provider_include_exclude
branch
from
2c348b1 to
29efcef
Compare
|
I think we should still add FI_REMOTE_COMM in the hints' caps. It doesn't make sense to pick shm even if user explicitly include it. |
For BTL, btl/sm should handle same-node communication, and btl/ofi is expected to handle inter-node communication. Here we request the FI_REMOTE_COMM capability to enforce this. Signed-off-by: Wenduo Wang <[email protected]>
|
@shijin-aws I talked to Brian about this and learned that btl/sm is expected to handle same-node communication, so btl/ofi should only worry about inter-node. I added a commit for this. @hppritcha Please see my second commit which adds |
|
Hi @BrendanCunningham did you find any issue with the change? AWS is currently blocked by this. We would like to include it in the upcoming Open MPI 5.0.2 release. |
|
@BrendanCunningham I checked with Howard. We agree to merge this fix first - please let me know if you find out any issues. |
|
Hey @wenduwan I'll take a look on Monday and will let you know if OPX has any issues with this change, thanks! |
I am a bit confused, if opx supports FI_HMEM, it should be picked up by btl/ofi even without the PR. If opx doesn't support FI_HMEM, this PR should fix the bug that will cause shm to be selected (and then excluded) What is the goal of the check here? |
|
@wenduwan @hppritcha Verified this change! OPX provider is found without issue, I tested using OSU MB's, and with the following options, which are typically preferred for OPX runs |
This PR addresses #12233
Since 5.0.x, we introduced an optional
FI_HMEMcapability in ofi provider selection logic(both mtl and btl) in order to support accelerator memory. As described in the issue, this introduced a bug that can cause the wrong ofi provider to be selected, even if the user explicitly includes/excludes the provider name.This change refactors the selection logic to correctly handle the include/exclude list, and therefore fixes the bug.