4 Failing test_flash_attn_output_fp8 tests on H100

[BioGeek]

I installed Flash Attention 3 on a system with H100 GPUs and ran the tests. 12436 passed, but 4 failed:

FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-False-True-gqa-dtype0]
FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-True-True-gqa-dtype0]
FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-False-True-gqa-dtype0]
FAILED hopper/test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-True-True-gqa-dtype0]

The differences is small, so could be a random fluke?

E       Mismatched elements: 1 / 3145728 (0.0%)
E       Greatest absolute difference: 0.08984375 at index (3, 3, 0, 80) (up to 0.0725 allowed)
E       Greatest relative difference: 0.1435546875 at index (3, 3, 0, 80) (up to 0.01 allowed)

Full log is attached: experiment.log

[robogast]

Same here:

================================================================================== short test summary info ===================================================================================
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-False-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-False-True-True-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-False-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
FAILED test_flash_attn.py::test_flash_attn_output_fp8[1024-1023-1.0-128-True-True-True-True-gqa-dtype0] - AssertionError: Tensor-likes are not close!
4 failed, 12436 passed in 125.88s (0:02:05)

[jyjfjyjf]

me too