Will "Prequant BitsAndBytes models with TP" be supported? #10117
-
|
when seve a BitsAndBytes model with --tensor-parallel-size N |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
Getting same error. Btw, what is PP? |
Beta Was this translation helpful? Give feedback.
-
|
@chenqianfzh What difficulties are there with this support? |
Beta Was this translation helpful? Give feedback.
-
Michael and I had discussed about the lack of support of TP to prequant bnb models, in PR #8434. We agreed PP is a reasonable choice than TP.
|
Beta Was this translation helpful? Give feedback.
Michael and I had discussed about the lack of support of TP to prequant bnb models, in PR #8434. We agreed PP is a reasonable choice than TP.