Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

combined inner outer reduction used in layer norm backward #2399

Open
liqiangxl opened this issue Feb 1, 2023 · 0 comments · May be fixed by #2400
Open

combined inner outer reduction used in layer norm backward #2399

liqiangxl opened this issue Feb 1, 2023 · 0 comments · May be fixed by #2400

Comments

@liqiangxl
Copy link
Collaborator

🚀 The feature, motivation and pitch

combine inner and outer reduction into one kernel.

  1. do partial outer reduction while blocks are looping over outer domain doing block inner reduction.
  2. write result of partial outer reduction to gmem
  3. sync and reload from gmem
  4. remap parallel pattern to finalized outer reduciton.

used in ln_backward.

Alternatives

No response

Additional context

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

combined inner outer reduction, add a simple test case csarofeen/pytorch
1 participant
@liqiangxl