New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reranker dynamic quantization #363

Open

rawsh-rubrik opened this issue Sep 16, 2024 · 1 comment

Labels

rawsh-rubrik commented Sep 16, 2024

Feature request

--dtype support for rerankers

Motivation

Easily quantize cross encoder models

Your contribution

. (will look into this)

Owner

michaelfeil commented Sep 16, 2024 •

edited

Loading

Correct, that is currently not possible, but easy to implement. You are welcome to contribute this.

Task:

Add a similar integration as in: https://github.com/michaelfeil/infinity/blob/65afe2b3d68fda10429bf7f215fe645be20788e4/libs/infinity_emb/infinity_emb/transformer/embedder/sentence_transformer.py#L87C9-L90C14
add a test (verifying that the quantization of a reranker model works). Please use BGE-reranker BASE or mixedbread-xsmall reranker for the test

michaelfeil added the help wanted label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment