Skip to content

Vercel AI Provider for running Large Language Models locally using LLamaCpp

License

Notifications You must be signed in to change notification settings

blib-la/llamacpp-ai-provider

This branch is up to date with nnance/llamacpp-ai-provider:main.

Folders and files

NameName
Last commit message
Last commit date

Latest commit

935d2fe Â· May 6, 2024

History

64 Commits
Apr 19, 2024
May 6, 2024
Apr 21, 2024
May 6, 2024
Apr 21, 2024
Mar 30, 2024
Apr 7, 2024
Mar 30, 2024
Apr 7, 2024
Apr 19, 2024
Apr 19, 2024
Apr 7, 2024

Repository files navigation

llamacpp-ai-provider

Vercel AI Provider for running Large Language Models locally using LLamaCpp without a server

🚧🚧 WARNING! Under Construction 🚧🚧

This project is under active construction and current depends on the node-llama-cpp library. This will be replaced by a low level API with direct integration to LLamaC++ library. See the roadmap section for more details.

Features

  • Vercel AI full stack support
  • Run local LLMs directly without server dependency
  • Supported by most of the models that are supported by the LLamaC++ library

Roadmap

  • Direct integration with LLamaC++
  • Support Completion Language Model

Installation

npm install --save ai llamacpp-ai-provider

The example below expects the llama-2 model file installed in the models folder in the project root folder. You can download a GGUF compatible file from Hugging Face.

Usage

import { experimental_generateText } from "ai";
import { LLamaCpp } from "../index.js";
import { fileURLToPath } from "url";
import path from "path";

const modelPath = path.join(
  path.dirname(fileURLToPath(import.meta.url)),
  "../../models",
  "llama-2-7b-chat.Q4_K_M.gguf"
);

const llamacpp = new LLamaCpp(modelPath);

experimental_generateText({
  model: llamacpp.completion(),
  prompt: "Invent a new holiday and describe its traditions.",
}).then(({ text, usage, finishReason }) => {
  console.log(`AI: ${text}`);
});

For more examples, see the getting started guide

About

Vercel AI Provider for running Large Language Models locally using LLamaCpp

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • TypeScript 49.1%
  • C++ 44.5%
  • CMake 4.1%
  • Shell 2.3%