[rocm6.4_internal_testing] [NAVI32] Skipped sdpa_2 test in test_aot_inductor for Navi32 #1882
Conversation
|
Jenkins build for 5d647c36630d8d201cfe8a29820943bc4c2191a2 commit finished as FAILURE |
|
@iupaikov-amd please add PR description explaining the justification for skipping |
|
Jenkins build for 5d647c36630d8d201cfe8a29820943bc4c2191a2 commit finished as FAILURE |
|
Added description |
|
Jenkins build for 5d647c36630d8d201cfe8a29820943bc4c2191a2 commit finished as FAILURE |
|
Jenkins build for 5d647c36630d8d201cfe8a29820943bc4c2191a2 commit finished as FAILURE |
|
Jenkins build for 5d647c36630d8d201cfe8a29820943bc4c2191a2 commit finished as FAILURE |
|
Jenkins build for 5d647c36630d8d201cfe8a29820943bc4c2191a2 commit finished as FAILURE |
The test fails with assertion error "Tensors are not close"
After testing I can confirm that this issue is caused by eager mode execution specific to navi32 during the test_sdpa_2 run. Made a cross reference between navi31, navi32 and mi300. AOTInductor results are all the exact same for all of the archs, only the eager mode fails here for navi32 with 1.5% difference in tensor values from the gpu run. I assume that this happens due to fp16-32-16 conversions in eager mode or missing some if-statements for navi32 specifically.
Simple reproducer to check the values for cpu/gpu/eager/aoti runs.
gfx1101_test_sdpa_2_issue_reproducer.txt