Adding bittree interface to improve regridding performance in octree mode #3555

akashdhruv · 2023-09-18T18:18:08Z

Summary

This PR introduces dependency on Bittree library to improve regridding performance in octree mode.

Additional background

Testing and development of this feature is done in sync with Flash-X and is recorded in this reproducibility capsule. This PR is primarily created to consolidate development work and avoid creating multiple branches. At present using AMReX+Bittree improves regridding performance by a factor of 2 at > 20000 ranks. We hope to improve performance further using a bittree-based distribution mapping and therefore adding a new function AmrMesh::MakeDistributionMap as a place-holder.

Continuation of #2893 and #3547

Checklist

The proposed changes:

fix a bug or incorrect behavior in AMReX
add new capabilities to AMReX
changes answers in the test suite to more than roundoff level
are likely to significantly affect the results of downstream AMReX users
include documentation in the code and/or rst files, if appropriate

…mode Co-authored-by: Tom Klosterman <https://github.com/tklos96>

WeiqunZhang

I will add a CI after this a merged.

Src/AmrCore/AMReX_AmrMesh.H

Src/AmrCore/AMReX_AmrMesh.cpp

WeiqunZhang · 2023-09-28T21:53:20Z

I have run the test at amrex/Tests/Amr/Advection_AmrCore/Exec. It works as expected. However, the grids generated are not very efficient.

This is a picture from a run using bittree.

This is from a run using the native non-tree mode. The area covered by level 1 grids is much smaller.

Co-authored-by: Weiqun Zhang <[email protected]>

akashdhruv · 2023-09-28T22:07:18Z

I have run the test at amrex/Tests/Amr/Advection_AmrCore/Exec. It works as expected. However, the grids generated are not very efficient.

The area covered by level 1 grids is much smaller.

I think the refine-derefine criteria maybe too strict. I can look into this after the merge and create a follow-up PR

Src/Extern/Bittree/AMReX_Bittree.cpp

kweide · 2023-09-29T02:00:16Z

the grids generated are not very efficient.

I think we cannot do much better, given the octtree-mimicking parameters that we are using (and have been using way before Bittree) in Flash-X for the AMR/Amrex Grid implementation:

  ! Setup AMReX in octree mode
  call pp_amr%add   ("max_grid_size_x",     NXB)
  call pp_amr%add   ("max_grid_size_y",     NYB)
  call pp_amr%add   ("max_grid_size_z",     NZB)
  call pp_amr%add   ("blocking_factor_x", 2*NXB)
  call pp_amr%add   ("blocking_factor_y", 2*NYB)
  call pp_amr%add   ("blocking_factor_z", 2*NZB)
  call pp_amr%add   ("refine_grid_layout", 0)
  call pp_amr%add   ("grid_eff",  1.0)

  ! According to Weiqun n_proper=1 is an appropriate setting that will result in
  ! correct nesting.
  call pp_amr%add("n_proper", 1)

This is from our gr_amrexInit.F90. Maybe not all the lines are relevant here, maybe only the first 6 are.

(I think at one point at least I understood how the max_grid_size_ and blocking_factor_ AMReX parameters work, and why they have to be set this way in order to limit refinement patters to the octtree-ish ones that can be generated by PARAMESH.)

…mode (AMReX-Codes#3555) ## Summary This PR introduces dependency on [Bittree library](https://github.com/Flash-X/Bittree) to improve regridding performance in octree mode. ## Additional background Testing and development of this feature is done in sync with Flash-X and is recorded in this [reproducibility capsule](https://github.com/Lab-Notebooks/AMReX-Bittree-Performance). This PR is primarily created to consolidate development work and avoid creating multiple branches. At present using AMReX+Bittree improves regridding performance [by a factor of 2](https://github.com/Lab-Notebooks/AMReX-Bittree-Performance/blob/14faa2212c4e5dba7fd99a6526c6937414f9c109/analysis/Performance.ipynb) at > 20000 ranks. We hope to improve performance further using a bittree-based distribution mapping and therefore adding a new function `AmrMesh::MakeDistributionMap` as a place-holder. Continuation of AMReX-Codes#2893 and AMReX-Codes#3547 ## Checklist The proposed changes: - [ ] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate --------- Co-authored-by: Weiqun Zhang <[email protected]> Co-authored-by: Tom Klosterman <https://github.com/tklos96>

akashdhruv · 2024-01-04T19:39:46Z

This is a picture from a run using bittree.

@WeiqunZhang Can you create a time evolution so we can see how derefinement logic is working? It maybe possible that derefinement logic is not very strict.

WeiqunZhang self-requested a review September 18, 2023 18:20

WeiqunZhang self-assigned this Sep 18, 2023

akashdhruv force-pushed the bt-merge branch 5 times, most recently from 8e3cb73 to 74c5b5e Compare September 21, 2023 14:21

Adding bittree interface to improve regridding performance in octree …

b18c8c4

…mode Co-authored-by: Tom Klosterman <https://github.com/tklos96>

akashdhruv force-pushed the bt-merge branch from 74c5b5e to b18c8c4 Compare September 26, 2023 01:00

WeiqunZhang requested changes Sep 28, 2023

View reviewed changes

akashdhruv and others added 7 commits September 28, 2023 17:02

Merge branch 'development' into bt-merge

a9530af

Update Src/AmrCore/AMReX_AmrMesh.H

ab1bfb6

Co-authored-by: Weiqun Zhang <[email protected]>

Update Src/AmrCore/AMReX_AmrMesh.H

0d0d8fb

Co-authored-by: Weiqun Zhang <[email protected]>

Update Src/AmrCore/AMReX_AmrMesh.cpp

8675c4c

Co-authored-by: Weiqun Zhang <[email protected]>

Update Src/AmrCore/AMReX_AmrMesh.cpp

c99c087

Co-authored-by: Weiqun Zhang <[email protected]>

Update Src/AmrCore/AMReX_AmrMesh.cpp

ef3b524

Co-authored-by: Weiqun Zhang <[email protected]>

Update Src/AmrCore/AMReX_AmrMesh.cpp

7638198

Co-authored-by: Weiqun Zhang <[email protected]>

WeiqunZhang reviewed Sep 28, 2023

View reviewed changes

Src/Extern/Bittree/AMReX_Bittree.cpp Outdated Show resolved Hide resolved

WeiqunZhang and others added 2 commits September 28, 2023 15:16

Update Src/Extern/Bittree/AMReX_Bittree.cpp

af0b7cf

Src/Extern/Bittree

b176ddf

akashdhruv requested a review from WeiqunZhang September 28, 2023 22:44

This was referenced Sep 28, 2023

Improve efficiency of octree grids Box-Tools/amrex#10

Open

Improve efficiency of octree grids Flash-X/Bittree#4

Open

WeiqunZhang approved these changes Oct 1, 2023

View reviewed changes

WeiqunZhang merged commit 8515ea8 into AMReX-Codes:development Oct 1, 2023
65 of 67 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding bittree interface to improve regridding performance in octree mode #3555

Adding bittree interface to improve regridding performance in octree mode #3555

akashdhruv commented Sep 18, 2023 •

edited

Loading

WeiqunZhang left a comment

WeiqunZhang commented Sep 28, 2023

akashdhruv commented Sep 28, 2023 •

edited

Loading

kweide commented Sep 29, 2023

akashdhruv commented Jan 4, 2024 •

edited

Loading

Adding bittree interface to improve regridding performance in octree mode #3555

Adding bittree interface to improve regridding performance in octree mode #3555

Conversation

akashdhruv commented Sep 18, 2023 • edited Loading

Summary

Additional background

Checklist

WeiqunZhang left a comment

Choose a reason for hiding this comment

WeiqunZhang commented Sep 28, 2023

akashdhruv commented Sep 28, 2023 • edited Loading

kweide commented Sep 29, 2023

akashdhruv commented Jan 4, 2024 • edited Loading

akashdhruv commented Sep 18, 2023 •

edited

Loading

akashdhruv commented Sep 28, 2023 •

edited

Loading

akashdhruv commented Jan 4, 2024 •

edited

Loading