Tpetra/Ifpack2: Better handling of non-GPU aware MPI

[csiefer2]

Largely from discussions with @vbrunini and @jjellio , we should have a unified way of supporting the equivalent TPETRA_ASSUME_GPU_AWARE_MPI across the solver stack (Ifpack2's TriDi solver ignores this). Moreover, if you are building a GPU node but not using GPU-Aware MPI, it should use Kokkos::SharedHostPinnedSpace for the comm buffer. This abstraction let's us move arch-specific behavior out of Tpetra proper.

This should also probably be done with env controls at a Trilinos-wide level and not at the Tpetra-level.

[github-actions]

Automatic mention of the @trilinos/ifpack2 team

[jhux2]

Related: #4757

[jhux2]

Packages that explicitly reference CudaHostPinnedSpace:

• Adelus
  • option to use
• Ifpack2
  • hard-wired in TriDiag solver
• Panzer (one unit test)
• STK
  • hard-wired
• Tpetra
  • not used
• Kokkos
• Kokkos-Kernels

[csiefer2]

@vqd8a How does Adelus control whether or not to use HostPinnedSpace?

[jhux2]

@vqd8a How does Adelus control whether or not to use HostPinnedSpace?

Adelus has a cmake option.