Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/multi rhs #1318

Merged
merged 77 commits into from
Oct 25, 2022
Merged

Feature/multi rhs #1318

merged 77 commits into from
Oct 25, 2022

Conversation

maddyscientist
Copy link
Member

@maddyscientist maddyscientist commented Sep 8, 2022

This PR is focussed on framework evolution to get us ready for multi-RHS workflows

  • Creation new vector_ref class which replaces the prior use of std::vector<ColorSpinorField_ref>
  • Added multi-RHS interface to Dirac and DiracMatrix classes
  • Move all regular Krylov solvers to use std::vector<ColorSpinorField> instead of std::vector<ColorSpinorField*>
  • Significant cleanup of eigensolvers to use new multi-RHS framework
    • Block Lanczos eigen solver is now multi-RHS ready
  • Created new class FieldTmp which provides a cache for temporary objects. Deployed in the Dirac operators to negate the need to allocate and maintain explicit temporary vectors
  • Add max and max_deviation reduce kernels, useful for correctness testing
  • Added new test, io_test, for testing both gauge and color-spinor storage when QIO is enabled
  • Cleanup of the blas functions: blas functions are now const correct, and doxygen is now added for all functions.
  • Add deflation to BiCGStab
  • Fix PCG solver which was broken when no preconditioner is present
  • Fix various issues with ARPACK (lack of SVD support, correct e-vector extraction with non-Hermitian matrices)
  • Various other misc fixes and cleanup

Update

Some additional fixes added during review:

  • Clean up PCG and fix some corner cases
  • Tweak max_res_increase default parameter to ensure more reliable convergence in ctest
  • Add PCG and Schwarz preconditioned solvers to gtest list
  • Fix compiler warnings when QUDA_INVERFACE_NVTX=ON
  • Fix initialization error when using split grid
  • Fix issue with BiCGStab

maddyscientist and others added 30 commits August 12, 2022 04:38
…0 seconds. This allows us to clean up various calls to saveTuneCache()
…wrappers with a generic multi-rhs interface and a cleanup of the internals
…and writing to distinct fields. blas::ax is now just a wrapper around this, and introduce blas::axy which exposes this functionality
…Field_cref instead of ColorSpinorField_ref where the vectors are read only
…oexcept - this allows for move assignment instead of copy assignment when std::vector::resize is called
… of crefs, we would endup self recursing if passing a vector of refs: added explicit make_cset helper, to create explicit cref sets to avoid this
…es that allow us to avoid the copying that would occur if we used std::vector::resize
…cantly boosts the performance of the blas_test, removing a lot of copies
… of pointers. As part of this change, the multi-blas function interfaces have been overhauled to utilized a new specialized type vector_ref, to provide a universal interface
@maddyscientist
Copy link
Member Author

There are still a couple of saveTuneCache in the interface. I guess they can all be deleted?

I left those intentionally, but happy to discuss their removal. Those were left to ensure that when the solver finishes, the tunecache is dumped which I naively thought was desirable. Given it would be dumped anyway when endQuda is called, I'm happy to delete for code cleanup.

Thoughts?

Copy link
Member

@mathiaswagner mathiaswagner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Visual review looks good, but I have not done any testing beyond a simple build.

lib/multi_blas_quda.cu Outdated Show resolved Hide resolved
Copy link
Member

@hummingtree hummingtree left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes look great. I will do some actual builds.

Copy link
Contributor

@weinbe2 weinbe2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Visual only, but there's a lot of good cleanup here, and I appreciate the additional unit testing. Since others have been hands-on with this PR, I'm going to give it the approval based on my visual review. Awesome work!

@maddyscientist
Copy link
Member Author

Merging this now. I've verified manually:

  • split grid solvers are working as expected
  • multigrid file i/o

I've updated the description with a few late additions to this PR that were added during review.

@maddyscientist maddyscientist merged commit 0a31b22 into develop Oct 25, 2022
@maddyscientist maddyscientist deleted the feature/multi-rhs branch October 25, 2022 16:44
@maddyscientist maddyscientist restored the feature/multi-rhs branch November 22, 2022 22:02
@maddyscientist maddyscientist deleted the feature/multi-rhs branch March 28, 2023 21:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants