Skip to content

Commit

Permalink
Add steps to install from source for llama.cpp (#1396)
Browse files Browse the repository at this point in the history
* Add steps to install from source for llama.cpp

* Formatting.
  • Loading branch information
Vaibhavs10 committed Aug 8, 2024
1 parent cc5be1a commit d7b02af
Showing 1 changed file with 13 additions and 0 deletions.
13 changes: 13 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,22 @@ You can quickly start a locally running chat-ui & LLM text-generation server tha

**Step 1 (Start llama.cpp server):**

Install llama.cpp w/ brew (for Mac):

```bash
# install llama.cpp
brew install llama.cpp
```

or [build directly from the source](https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md) for your target device:

```
git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp && make
```

Next, start the server with the [LLM of your choice](https://huggingface.co/models?library=gguf):

```bash
# start llama.cpp server (using hf.co/microsoft/Phi-3-mini-4k-instruct-gguf as an example)
llama-server --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf --hf-file Phi-3-mini-4k-instruct-q4.gguf -c 4096
```
Expand Down

0 comments on commit d7b02af

Please sign in to comment.