Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Results for Optimizing OPT Model seems unusually high for 4 bits per tensor #2352

Open
amitrana001 opened this issue Oct 1, 2024 · 1 comment

Comments

@amitrana001
Copy link

Looking at the data for 4 bits per tensor in the results table for optimizing OPT model, the values for c4 Perplexity and wikitext2 Perplexity seem unusually high compared to the other configurations.
Are these values correct or there is some typo?

Code link here:

| 4 bits per tensor | 0.66 | 18763 | 31087 | 21.66 |

OPT-model-results

@junpeiz
Copy link
Collaborator

junpeiz commented Oct 3, 2024

My understanding is that the 4-bit per-tensor hurts that model too much, so the perplexity exploded. By using more fine-grained quantization (per-channel, per-block), the perplexity gets back to normal.
(cc @aseemw to confirm it's not a typo)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants