Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Attempt to use rFactor for allreduce
``` mpirun -np 1 -x NVFUSER_DUMP=fusion_ir_preseg pytest tests/python/test_communication.py -k allreduce_rfactor -s --only-mpi ``` ``` %kernel { T2_l_float[iS10{2}, ideviceIdx.x13{1}rf, rS14{5}rf, iS12{3}] (DeviceMesh{0}) = reduction( T0_g_float[iS0{2}, ideviceIdx.x6{1}, iS7{5}, iS2{3}] (DeviceMesh{0}), op = add, initial value = float(0), allreduce = false ) T1_g_float[iS15{2}, rS16{1}, iS17{3}] (DeviceMesh{0}) = reduction( T2_l_float[iS10{2}, ideviceIdx.x13{1}rf, rS14{5}rf, iS12{3}] (DeviceMesh{0}), op = add, initial value = float(0), allreduce = false ) TransformPrinter : T0_g_float[iS0{2}, ideviceIdx.x6{1}, iS7{5}, iS2{3}] (DeviceMesh{0}) logical domain : (iS0{2}, iS1{5}, iS2{3}) allocation domain : (iS0{2}, ideviceIdx.x6{1}, iS7{5}, iS2{3}) contiguity: t t t t Outer split: iS1{5} by factor 1 -> ideviceIdx.x6{1}, iS7{5} loop domain : (iS0{2}, ideviceIdx.x6{1}, iS7{5}, iS2{3}) T2_l_float[iS10{2}, ideviceIdx.x13{1}rf, rS14{5}rf, iS12{3}] (DeviceMesh{0}) root domain : (iS10{2}, rS11{5}rf, iS12{3}) Outer split: rS11{5}rf by factor 1 -> ideviceIdx.x13{1}rf, rS14{5}rf logical domain : (iS10{2}, ideviceIdx.x13{1}rf, rS14{5}rf, iS12{3}) allocation domain : (iS10{2}, ideviceIdx.x13{1}rf, rS14{5}rf, iS12{3}) contiguity: t t n t loop domain : (iS10{2}, ideviceIdx.x13{1}rf, rS14{5}rf, iS12{3}) T1_g_float[iS15{2}, rS16{1}, iS17{3}] (DeviceMesh{0}) logical domain : (iS15{2}, rS16{1}, iS17{3}) allocation domain : (iS15{2}, rS16{1}, iS17{3}) contiguity: t n t loop domain : (iS15{2}, rS16{1}, iS17{3}) } // %kernel ``` ``` RuntimeError: INTERNAL ASSERT FAILED at "/opt/pytorch/nvfuser/csrc/scheduler/vectorize_helper.cpp":1063, please report a bug with repro script to NVFuser at https://github.com/NVIDIA/Fuser/issues. Unexpected producer RF ID: iS2{3} Exception raised from getVectorizationBreakPointOfReductionProducer at /opt/pytorch/nvfuser/csrc/scheduler/vectorize_helper.cpp:1063 (most recent call first): ```
- Loading branch information