Why Rotate Again in main.py

[RanchiZhao]

logger.info("* Rotating model in float16 again")
tools.logging.Formatter.indent_inc()
rotate_llm(model, config.quant.rotation, rotation=rotation)

hi! Thanks for your work. I do not know why to rotate again in the above codes?

[wfan1203]

我也想问

[synxlin]

Hi,
We fix this issue in the latest release.
In the previous version, we load all models in FP16 by default. However, some models use BF16 by default. Since rotation is sensitive to the weight precision, we choose to first load model in FP32 which is compatible with both FP16 and BF16, and perform weight rotation. We then load FP16 model and replace corresponding weights with rotated weights.

[wfan1203]

感谢您的来信，我将及时回复您！                                                   ——武凡