Support Half/BFloat16 in native_group_norm (needs accuracy fix)

[swolchok]

No description provided.

[swolchok]

Stack from ghstack (oldest at bottom):

• -> Support Half/BFloat16 in native_group_norm (needs accuracy fix) #7846

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7846

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures

As of commit d3ecd5b with merge base 03cba2c ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner / linux-job (gh)
  >>> Lint for kernels/test/op_native_group_norm_test.cpp:
• pull / unittest / linux / linux-job (gh)
  [ FAILED ] OpNativeGroupNormOutTest.SmokeTest
• pull / unittest / macos / macos-job (gh)
  backends/xnnpack/test/ops/test_conv1d.py::TestConv1d::test_qs8_conv1d_batchnorm_seq

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[manuelcandales]

That's 20%. This doesn't make me feel comfortable.

[swolchok]

group_norm is one of the ops that automatic mixed precision will autocast to float32: https://intel.github.io/intel-extension-for-pytorch/xpu/1.10.200+gpu/tutorials/features/amp.html

I think the norm ops are just particularly prone to roundoff error, but I'm certainly not a numerical analysis person.

(unresolving for posterity)

[swolchok]

Dug in a little further. Here's the PR that originally made PyTorch group_norm support Half: https://github.com/pytorch/pytorch/pull/100234/files#diff-7927db349f568afca2de9b94d74ea5c3b8cb468cb6a433d0cc1e61e65c515a36

It looks like the test is atol=rtol=5e-3. I think it's reasonable to argue that if we can't get the tolerances to be broadly similar then we have a correctness issue and thus don't actually support Half. I'll see what I can do; this one might have to wait for code sharing.

[swolchok]

test doesn't pass with atol=rtol=5e-3. Holding off on group_norm until we have code sharing.