embedjs llama-cpp integration fails to run

[GhostDog98]

🐛 Describe the bug

To reproduce:

import { RAGApplicationBuilder, TextLoader } from '@llm-tools/embedjs'
import { LlamaCppEmbeddings, LlamaCpp } from '@llm-tools/embedjs-llama-cpp';
import { HNSWDb } from '@llm-tools/embedjs-hnswlib'

const app = await new RAGApplicationBuilder()
    .setModel(new LlamaCpp({modelPath:"./models/Llama-3.2-3B-Instruct-f16.gguf"}))
    .setEmbeddingModel(new LlamaCppEmbeddings({modelPath: "./models/dragon-yi-1-5-9.gguf"}))
    .setVectorDatabase(new HNSWDb())
    .build();

This code works if ran with ollama on both setModel and setEmbeddingModel.
It appears that it fails due to the file at models/embedjs-llama-cpp/src/llama-cpp-embeddings.ts not importing the requisite function getEmbeddingFor as called on line 24. Resulting in error;

TypeError: Cannot read properties of undefined (reading 'getEmbeddingFor')
    at file:///home/ghostdog/private-site/models/embedjs-llama-cpp/src/llama-cpp-embeddings.ts:24:50
    at Array.map (<anonymous>)
    at LlamaCppEmbeddings.embedDocuments (file:///home/ghostdog/private-site/models/embedjs-llama-cpp/src/llama-cpp-embeddings.ts:23:33)
    at LlamaCppEmbeddings.getDimensions (file:///home/ghostdog/private-site/models/embedjs-llama-cpp/src/llama-cpp-embeddings.ts:17:35)
    at RAGApplication.init (file:///home/ghostdog/private-site/core/embedjs/src/core/rag-application.ts:71:88)
    at async RAGApplicationBuilder.build (file:///home/ghostdog/private-site/core/embedjs/src/core/rag-application-builder.ts:46:9)
    at async file:///home/ghostdog/private-site/index.js:40:13

Am i missing any imports or obvious errors?

[GhostDog98]

For a more clear sample code:

import { RAGApplicationBuilder, TextLoader } from '@llm-tools/embedjs'
import { LlamaCppEmbeddings, LlamaCpp } from '@llm-tools/embedjs-llama-cpp';
import { HNSWDb } from '@llm-tools/embedjs-hnswlib'
import { OllamaEmbeddings, Ollama } from '@llm-tools/embedjs-ollama';

async function load_app_llamacpp(){
    const app = await new RAGApplicationBuilder()
    .setModel(new LlamaCpp({modelPath:"models/Llama-3.2-3B-Instruct-f16.gguf"}))
    .setEmbeddingModel(new LlamaCppEmbeddings({modelPath: "models/dragon-yi-1-5-9.gguf"}))
    .setVectorDatabase(new HNSWDb())
    .build();

    return app;
}

async function load_app_ollama(){
    const app = await new RAGApplicationBuilder()
    .setModel(new Ollama({modelName: "llama3.2", baseUrl: 'http://localhost:11434'}))
    .setEmbeddingModel(new OllamaEmbeddings({modelName: "nomic-embed-text", baseUrl: 'http://localhost:11434'}))
    .setVectorDatabase(new HNSWDb())
    .build();

    return app;
}

//let app = await load_app_llamacpp(); // Always fails

//let app = await load_app_ollama(); // Works as expected

[GhostDog98]

Update; actually, I think the OllamaEmbeddings doesn't set the modelName correctly, as it still defaults to mxbai-embed-large and thus errors out. The default is defined in @langchain/ollama/dist/embeddings, so a hack could be to change this value, but preferably it'd be set correctly... Shall I create another issue for this?
Edit: It appears this is soley a documentation issue, as while the quickstart has the correct option, the embeddings page does not.

[adhityan]

Yes please. There was another issue that I found in your original post. The way the initialization was happening allowed for a race condition - this has been addressed in the version just published (0.1.23). For the model name, please create a separate issue.

[GhostDog98]

Yes please. There was another issue that I found in your original post. The way the initialization was happening allowed for a race condition - this has been addressed in the version just published (0.1.23). For the model name, please create a separate issue.

I'm still seeing this issue unfortunately. npm list outputs:

├── @llm-tools/[email protected]
├── @llm-tools/[email protected]
├── @llm-tools/[email protected]
├── @llm-tools/[email protected]
└── @llm-tools/[email protected]

So I am up to date, however it seems it is getting slightly further, as the error log is now:

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3060, compute capability 8.6, VMM: yes
TypeError: Cannot read properties of undefined (reading 'getEmbeddingFor')
    at file:///home/private-site/models/embedjs-llama-cpp/src/llama-cpp-embeddings.ts:32:54
    at Array.map (<anonymous>)
    at LlamaCppEmbeddings.embedDocuments (file:///home/private-site/models/embedjs-llama-cpp/src/llama-cpp-embeddings.ts:31:19)
    at LlamaCppEmbeddings.getDimensions (file:///home/private-site/models/embedjs-llama-cpp/src/llama-cpp-embeddings.ts:24:35)
    at RAGApplication.init (file:///home/private-site/core/embedjs/src/core/rag-application.ts:71:88)
    at async RAGApplicationBuilder.build (file:///home/private-site/core/embedjs/src/core/rag-application-builder.ts:46:9)
    at async load_app_llamacpp (file:///home/private-site/mre.js:7:17)
    at async file:///home/private-site/mre.js:16:11

[GhostDog98]

@adhityan if you could reopen this issue that would be fantastic, as I'm still having issues 👍

[github-actions]

This issue is stale because it has been open for 14 days with no activity.

[adhityan]

Sorry, testing this is a little difficult for me. I do most of the programming in a Macbook. I have a windows laptop with Nvidia but it has limitations in being able to run llama-cpp.

[github-actions]

This issue is stale because it has been open for 14 days with no activity.