Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to reproduce RTN results in paper #8

Open
Golden-Wang opened this issue Jun 7, 2024 · 1 comment
Open

Unable to reproduce RTN results in paper #8

Golden-Wang opened this issue Jun 7, 2024 · 1 comment

Comments

@Golden-Wang
Copy link

Hi, thanks for the great work. I'm a big fan of how the configs are laid out.
I've been trying to reproduce the results mentioned in the paper using this repo. It was easy to do so for the methods that has a .yaml file in configs. However, no config is provided for RTN.
The best W4A8KV4 RTN result for Llama-2-7b I can get using this repo is 7.99 PPL on wikitext2. The paper gives 6.51 PPL for W4A8KV4 RTN and 5.99 PPL for W4A8KV4g128 RTN.
How were these results obtained? Thanks in advance.

@synxlin
Copy link
Contributor

synxlin commented Nov 8, 2024

Hi,
For RTN results, please turn off all optimizations in QOQ configurations including smooth, rotation and weight clipping. We keep the same weight data type (INT4 with zero points) for RTN and QOQ experiments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants