[BUG] embedding_model is not a valid embedding model', 'Please install the semantic extra to use this feature'

[universewill]

I installed semantic by "pip install chonkie[semantic]" and still get this error

from chonkie import TokenChunker, SemanticChunker, SentenceChunker

chunker = SemanticChunker(
     embedding_model="all-minilm-l6-v2",
     chunk_size=512,
     similarity_threshold=0.7
)

chunks = chunker("Woah! Chonkie, the chunking library is so cool! I love the tiny hippo hehe. The pandas is a creature. It has white and black fur. Its' very cute")

for chunk in chunks:
    print(f"Chunk: {chunk.text}")
    print(f"Tokens: {chunk.token_count}")

[bhavnicksm]

Hey @universewill!

Thanks for opening an issue~ 😊

The issue you are seeing is because, by default the semantic module assumes you are going to use a Model2VecEmbeddings (Chonkie uses model2vec because of their speed and light-weightedness).

When you pass in all-minilm-l6-v2, it requires the sentence-transformer dependency, which can be installed with pip install "chonkie[st]". Here's an image from the DOCS Installation page that describes this behaviour:

The error message does not know which dependency you are missing for the particular model you are trying, so it defaults to saying the above.

Let me know if you're able to use it with chonkie[st]~

Thanks! ☺️

[bhavnicksm]

from chonkie import TokenChunker, SemanticChunker, SentenceChunker

chunker = SemanticChunker(
     embedding_model="all-minilm-l6-v2",
     chunk_size=512,
     similarity_threshold=0.7
)

Also, just wanted to point out that similarity_threshold is no longer used, and instead we are using threshold, please refer to the DOCS for the API reference.

[bhavnicksm]

Update: If you add an unknown keyword argument to the init, then the SentenceTransformerEmbeddings fail. This is problematic.

I'll look into this and add a patch for this, asap.

[bhavnicksm]

Added a patch to have better error message when seeing an unknown kwarg causing failiure; closing issue~