Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GPTQ for RWKV #98

Draft
wants to merge 20 commits into
base: main
Choose a base branch
from
Draft

GPTQ for RWKV #98

wants to merge 20 commits into from

Commits on Apr 18, 2023

  1. feat(quantize): measure perplexity on wikitext2

    @3outeille
    3outeille committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    6e556e5 View commit details
    Browse the repository at this point in the history

  2. feat(quantize): add gptq files

    @3outeille
    3outeille committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    bde6374 View commit details
    Browse the repository at this point in the history

  3. feat(quantize): begin to readapt with RWKV

    @3outeille
    3outeille committed Apr 18, 2023
    Configuration menu
    Copy the full SHA
    943af70 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2023

  1. breaking(quantize): draft gptq rwkv

    @3outeille
    3outeille committed Apr 23, 2023
    Configuration menu
    Copy the full SHA
    629fc9b View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2023

  1. fix(quantize): GPTQ hooks now work with RWKV

    @3outeille
    3outeille committed Apr 24, 2023
    Configuration menu
    Copy the full SHA
    4a19476 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2023

  1. feat(quantize): link fasterquant with RWKV + remove 1D tensor quantiz… 

    …ation for now
    @3outeille
    3outeille committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    dba2670 View commit details
    Browse the repository at this point in the history

  2. feat(quantize): full gptq pipeline now integrated with RKWV (quite sl… 

    …ow for some layer + need tests)
    @3outeille
    3outeille committed Apr 25, 2023
    Configuration menu
    Copy the full SHA
    57079e7 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. fix(quantize): add missing part in forward block + support head.weigh… 

    …t quantization
    @3outeille
    3outeille committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    8e78f2d View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2023

  1. feat(sanity-check): begin sanity check for GPTQ on MNIST

    @3outeille
    3outeille committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    f87df05 View commit details
    Browse the repository at this point in the history

  2. breaking(sanity-check): add save & load option for reference gptq

    @3outeille
    3outeille committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    b77715d View commit details
    Browse the repository at this point in the history

  3. breaking(sanity-check): enhance with dummy model

    @3outeille
    3outeille committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    816def4 View commit details
    Browse the repository at this point in the history

  4. fix(sanity-check): dont quantize last layer for dummy example

    @3outeille
    3outeille committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    f141e52 View commit details
    Browse the repository at this point in the history

  5. breaking(sanity-check): adding my implem gptq

    @3outeille
    3outeille committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    a1ea882 View commit details
    Browse the repository at this point in the history

  6. fix(sanity-check): training ref and implem now yield same outputs

    @3outeille
    3outeille committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    8a37fb4 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2023

  1. feat(sanity-check): implem version of gptq now added

    @3outeille
    3outeille committed May 2, 2023
    Configuration menu
    Copy the full SHA
    4233522 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2023

  1. fix(sanity-check): ref and implem now yield the same results at every… 

    … step

 Date:      Tue May 2 18:17:57 2023 +0000
    @3outeille
    3outeille committed May 3, 2023
    Configuration menu
    Copy the full SHA
    e74d72a View commit details
    Browse the repository at this point in the history

  2. feat(quantize): readapt GPTQ for rwkv

    @3outeille
    3outeille committed May 3, 2023
    Configuration menu
    Copy the full SHA
    cf14124 View commit details
    Browse the repository at this point in the history

Commits on May 7, 2023

  1. breaking(gptq): quantizing only 1 layer yield high perplexity

    @3outeille
    3outeille committed May 7, 2023
    Configuration menu
    Copy the full SHA
    c2bbe64 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2023

  1. fix(ppl): measure ppl using sliding window

    @3outeille
    3outeille committed May 23, 2023
    Configuration menu
    Copy the full SHA
    9b9c714 View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2023

  1. update

    @3outeille
    3outeille committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    3399ef0 View commit details
    Browse the repository at this point in the history