Chat with Documents

[Utopiah]

Hi there, how can the equivalent of "Chat with Documents" as explained in the documentation https://lmstudio.ai/docs/basics/rag be done here?

[ryan-the-crayon]

This functionality is still in active development. However, that said, it is indeed possible (but not well documented yet). See this script presented at the NYU hackathon event:

import { LMStudioClient } from "@lmstudio/sdk";
import { readFile } from "fs/promises";
import { basename } from "path";
import { createInterface } from "readline/promises";

const client = new LMStudioClient();

const documentPath = "./Frozen Script.txt";
const rl = createInterface({
    input: process.stdin,
    output: process.stdout,
});

const question = await rl.question("Enter a question: ");
rl.close();
const document = await client.files.uploadTempFile(
    basename(documentPath),
    await readFile(documentPath)
);

const nomic = await client.embedding.getOrLoad("text-embedding-nomic-embed-text-v1.5@q8_0");
const results = await client.retrieval.retrieve(question, [document], {
    embeddingModel: nomic,
});

const prompt = `
Given the following citation, please answer user's question:

----- Citation -----
${results.entries[0].content}
----- End of Citation -----

Question: ${question}
`;

const llama = await client.llm.getOrLoad("llama-3.2-3b-instruct");
const prediction = llama.respond([
    {
        role: "user",
        content: prompt,
    },
]);

for await (const { content } of prediction) {
    process.stdout.write(content);
}

const { stats } = await prediction;
console.info("\nStats:");
console.info(stats);