Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable global particle redistribution based on the grid #707

Merged
merged 2 commits into from
Nov 20, 2023

Conversation

streeve
Copy link
Member

@streeve streeve commented Oct 20, 2023

While it should often be avoided, global communication is occasionally needed in applications. This uses the background grid to determine the rank to send any particle to (including removing those outside the global domain).

Still to do:

  • Consider moving the migrate functionality to a free function or separate class
  • Consider naming scheme - this primarily determines comm destinations, to then create the communication plan and distributor and eventually migrate
  • In this PR or separately, add an interface to extract only those particles that don't "belong" on the current rank from the migration destinations and ghost them instead

@streeve streeve added the enhancement New feature or request label Oct 20, 2023
@streeve streeve self-assigned this Oct 20, 2023
@streeve
Copy link
Member Author

streeve commented Oct 20, 2023

@kwitaechong can you test this as a replacement for your custom migrateAllToAll?

@streeve
Copy link
Member Author

streeve commented Oct 20, 2023

Currently relying on KOKKOS_CLASS_LAMBDA without requiring C++20 causing the CI failures

@streeve streeve force-pushed the global_migrate branch 3 times, most recently from a62ff5e to ddb9409 Compare October 24, 2023 14:02
Copy link
Collaborator

@kwitaechong kwitaechong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

grid/src/Cabana_Grid_GlobalParticleComm.hpp Show resolved Hide resolved
@streeve
Copy link
Member Author

streeve commented Oct 25, 2023

retest this please.

@streeve
Copy link
Member Author

streeve commented Oct 26, 2023

@kwitaechong can you test on Summit or Frontier to find where the device failure is occurring?

@streeve
Copy link
Member Author

streeve commented Nov 16, 2023

Manually checked that tests pass with SYCL. Should be ready to merge once it's removed from Cuda-SYCL tests

@streeve
Copy link
Member Author

streeve commented Nov 20, 2023

I/O timeouts unrelated

@streeve streeve merged commit d883c13 into ECP-copa:master Nov 20, 2023
29 of 33 checks passed
@streeve streeve deleted the global_migrate branch November 20, 2023 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants