How to know which Hugging Face Hub models can be used?

[endolith]

As in #45

We can use the ones listed on sbert.net/docs/pretrained_models.html. For instance, --transformer-model sentence-transformers/paraphrase-multilingual-mpnet-base-v2 works for me.

Do others work, too? There is a leaderboard here: https://huggingface.co/spaces/mteb/leaderboard

I'm not clear on whether they all have a compatible interface or can be used in the same way.

[freedmand]

Yes, many HuggingFace models work! I welcome people to drop recommendations of HuggingFace models that have proven effective for them.

From the models guide I had success using sgpt (Muennighoff/SGPT-125M-weightedmean-msmarco-specb-bitfit).

Folks using Semantra in other languages have had success too:

• Semantra with Chinese/multilingual similarity: 请问能否增强对中文的支持？ #30

[freedmand]

I would give it a try! Note that models over 1 GB might require a GPU to run at any reasonable speed.

Note: I have had some success getting Instructor models to run — I have a branch that I'd like to commit soon that allows you to specify the instructor prompt but there are some issues with it (which may be resolved if I run through HuggingFace). I've still found that mpnet and minilm from SentenceTransformers (e.g. some of the default model options packaged with Semantra) seem to hit a sweet spot with versatility and usefulness (and they're fast).

[freedmand]

At the bottom of this thread I linked above, @yych42 had success running e5 but noted performance might be a drawback: #30 (comment)

[endolith]

So instructor models are not compatible yet, is what you're saying? :) Because they require a different "prompt"?

[freedmand]

Yea Instructor models do not work out-of-the-box with Semantra v0.1.7. I'll see if I can get the patch pushed soon though that allows you to specify the secondary prompt for the embedding for v0.1.8

[OmarAHex]

I welcome people to drop recommendations of HuggingFace models that have proven effective for them

I definitely think BAAI/bge-large-en-v1.5 should be "included" by default in semantra. The performance difference for me vs sgpt-1.3B was massive. Query times were neglegible for both, but calculating embeddings was much faster with bge-large (30m vs 3h), and the results were finally relevant to my search with bge-large. (PS. I'm using semantra to search large medical textbooks so it might be biased towards medicine).

[endolith]

This isn't the right way to use Q&A format, but whatever:

Can the question and answer models be used?

https://www.sbert.net/docs/pretrained_models.html#multi-qa-models
https://www.sbert.net/docs/pretrained_models.html#natural-questions-nq-dataset-models

Would that make it better for putting a question in the search box and it shows the answer as the result?

[freedmand]

I think those will work, and yes that seems to be the right interpretation of what it should do! Curious how well it works for you.

[endolith]

I tried to combine both the MTEB Leaderboard and the Sentence-Transformers benchmark into one table and highlighted the Semantra built-in options in green: https://docs.google.com/spreadsheets/d/1uOXb5SOOouFMuUdDTT7FpZAdesfoEE7slyWpri4bac4/edit?usp=sharing

Sorted by "Retrieval" task because I think that's the most applicable to Semantra? Or should it be sorted by STS?

[endolith]

Running on 4 small files:

• openai: 2.2 s / file
• multilingual-e5-small: 16.0 s / file (7.3× slower than openai)
• paraphrase-multilingual-mpnet-base-v2: 38 s / file (2.4× slower than multilingual-e5-small)
• multilingual-e5-large: 135 s / file (8.4× slower than small)

Running on i7-9750H CPU @ 2.60GHz, 2592 Mhz, 6 Core(s), 12 Logical Processor(s). (#36 (comment))

[freedmand]

Woah cool thanks for comparing! This spreadsheet is nice.

Run times probably will vary a lot depending on CPU and possibly GPU.

Re: scores, I don't actually know too much about what they are measuring. I don't know if any score exists that would perfectly capture it being useful for Semantra — a lot of that might be subjective or depend on your specific needs.

[endolith]

Yeah I still don't understand the difference between retrieval and STS. I've asked ChatGPT to explain it to me multiple times, and asked on huggingface forums, but they still sound like the same thing to me...