Figure out why MI210 did not work in nightlies

[aprokop]

In #1048, we updated HIP builds to run on MI210 instead of MI100. It worked perfectly fine for continuous. But it did not work for nightly.

There are some differences:

• In continuous, we use Dockerfile.hipcc. In nightly, we just grab the rocm image
• In continuous, we use ROCm 5.6 (changed from 5.3 in Fix label for AMD GPU in the CI #1027). In nightly, we use 5.4

Currently, the patch for nightly was reverted in #1060.

Tasks

Preview Give feedback

No tasks being tracked yet.

[aprokop]

Interestingly, sometimes things pass in the nightlies. The latest build passed.

[Rombur]

It was a different machine. One of the machine is more stable than the other.