Adding bittree interface to improve regridding performance in octree mode

[akashdhruv]

Summary

This PR introduces dependency on Bittree library to improve regridding performance in octree mode.

Additional background

Testing and development of this feature is done in sync with Flash-X and is recorded in this reproducibility capsule. This PR is primarily created to consolidate development work and avoid creating multiple branches. At present using AMReX+Bittree improves regridding performance by a factor of 2 at > 20000 ranks. We hope to improve performance further using a bittree-based distribution mapping and therefore adding a new function AmrMesh::MakeDistributionMap as a place-holder.

Continuation of #2893 and #3547

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

[WeiqunZhang]

I will add a CI after this a merged.

[WeiqunZhang]

I have run the test at amrex/Tests/Amr/Advection_AmrCore/Exec. It works as expected. However, the grids generated are not very efficient.

This is a picture from a run using bittree.

This is from a run using the native non-tree mode. The area covered by level 1 grids is much smaller.

[akashdhruv]

I have run the test at amrex/Tests/Amr/Advection_AmrCore/Exec. It works as expected. However, the grids generated are not very efficient.

The area covered by level 1 grids is much smaller.

I think the refine-derefine criteria maybe too strict. I can look into this after the merge and create a follow-up PR

[kweide]

the grids generated are not very efficient.

I think we cannot do much better, given the octtree-mimicking parameters that we are using (and have been using way before Bittree) in Flash-X for the AMR/Amrex Grid implementation:

  ! Setup AMReX in octree mode
  call pp_amr%add   ("max_grid_size_x",     NXB)
  call pp_amr%add   ("max_grid_size_y",     NYB)
  call pp_amr%add   ("max_grid_size_z",     NZB)
  call pp_amr%add   ("blocking_factor_x", 2*NXB)
  call pp_amr%add   ("blocking_factor_y", 2*NYB)
  call pp_amr%add   ("blocking_factor_z", 2*NZB)
  call pp_amr%add   ("refine_grid_layout", 0)
  call pp_amr%add   ("grid_eff",  1.0)

  ! According to Weiqun n_proper=1 is an appropriate setting that will result in
  ! correct nesting.
  call pp_amr%add("n_proper", 1)

This is from our gr_amrexInit.F90. Maybe not all the lines are relevant here, maybe only the first 6 are.

(I think at one point at least I understood how the max_grid_size_ and blocking_factor_ AMReX parameters work, and why they have to be set this way in order to limit refinement patters to the octtree-ish ones that can be generated by PARAMESH.)

[akashdhruv]

This is a picture from a run using bittree.

@WeiqunZhang Can you create a time evolution so we can see how derefinement logic is working? It maybe possible that derefinement logic is not very strict.