From aeb3f835aef7c80da7ad2ccae433d11449493061 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Wed, 10 Apr 2024 09:07:21 +0200 Subject: [PATCH 1/9] :arrow_up: Update docs version mudler/LocalAI (#1978) Signed-off-by: GitHub Co-authored-by: mudler --- docs/data/version.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/data/version.json b/docs/data/version.json index b6372479d17b..cc0478ca2f6c 100644 --- a/docs/data/version.json +++ b/docs/data/version.json @@ -1,3 +1,3 @@ { - "version": "v2.11.0" + "version": "v2.12.1" } From 951e39d36c06bb14b3b95b27309d7be809f4a3f4 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Wed, 10 Apr 2024 09:07:41 +0200 Subject: [PATCH 2/9] :arrow_up: Update ggerganov/llama.cpp (#1979) Signed-off-by: GitHub Co-authored-by: mudler --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index d42e1a99ffe2..b43541ff9596 100644 --- a/Makefile +++ b/Makefile @@ -5,7 +5,7 @@ BINARY_NAME=local-ai # llama.cpp versions GOLLAMA_STABLE_VERSION?=2b57a8ae43e4699d3dc5d1496a1ccd42922993be -CPPLLAMA_VERSION?=cc4a95426d17417d3c83f12bdb514fbe8abe2a88 +CPPLLAMA_VERSION?=1b67731e184e27a465b8c5476061294a4af668ea # gpt4all version GPT4ALL_REPO?=https://github.com/nomic-ai/gpt4all From 7e2f8bb4083eea3939072dab2cb47261b1b97603 Mon Sep 17 00:00:00 2001 From: "LocalAI [bot]" <139863280+localai-bot@users.noreply.github.com> Date: Wed, 10 Apr 2024 09:08:00 +0200 Subject: [PATCH 3/9] :arrow_up: Update ggerganov/whisper.cpp (#1980) Signed-off-by: GitHub Co-authored-by: mudler --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index b43541ff9596..337ebc6458ef 100644 --- a/Makefile +++ b/Makefile @@ -16,7 +16,7 @@ RWKV_REPO?=https://github.com/donomii/go-rwkv.cpp RWKV_VERSION?=661e7ae26d442f5cfebd2a0881b44e8c55949ec6 # whisper.cpp version -WHISPER_CPP_VERSION?=13c22321d1ac758ce68a429c23104e234b440769 +WHISPER_CPP_VERSION?=8f253ef3af1c62c04316ba4afa7145fc4d701a8c # bert.cpp version BERT_VERSION?=6abe312cded14042f6b7c3cd8edf082713334a4d From d692b2c32a400a4aa0c6df9a51aa4f3cbe73edff Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 10 Apr 2024 10:31:59 +0200 Subject: [PATCH 4/9] ci: push latest images for dockerhub (#1984) Fixes: #1983 Signed-off-by: Ettore Di Giacinto --- .github/workflows/image_build.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/image_build.yml b/.github/workflows/image_build.yml index bd244dcf07cc..b0684a4c66fc 100644 --- a/.github/workflows/image_build.yml +++ b/.github/workflows/image_build.yml @@ -280,6 +280,7 @@ jobs: run: | docker pull localai/localai:${{ steps.meta.outputs.version }} docker tag localai/localai:${{ steps.meta.outputs.version }} localai/localai:${{ inputs.latest-image }} + docker push localai/localai:${{ inputs.latest-image }} docker pull quay.io/go-skynet/local-ai:${{ steps.meta.outputs.version }} docker tag quay.io/go-skynet/local-ai:${{ steps.meta.outputs.version }} quay.io/go-skynet/local-ai:${{ inputs.latest-image }} docker push quay.io/go-skynet/local-ai:${{ inputs.latest-image }} @@ -289,6 +290,7 @@ jobs: run: | docker pull localai/localai:${{ steps.meta_aio_dockerhub.outputs.version }} docker tag localai/localai:${{ steps.meta_aio_dockerhub.outputs.version }} localai/localai:${{ inputs.latest-image-aio }} + docker push localai/localai:${{ inputs.latest-image-aio }} docker pull quay.io/go-skynet/local-ai:${{ steps.meta_aio.outputs.version }} docker tag quay.io/go-skynet/local-ai:${{ steps.meta_aio.outputs.version }} quay.io/go-skynet/local-ai:${{ inputs.latest-image-aio }} docker push quay.io/go-skynet/local-ai:${{ inputs.latest-image-aio }} From d23e73b11828b59a608174dc679eb5a3c2d2f42f Mon Sep 17 00:00:00 2001 From: "Sebastian.W" Date: Wed, 10 Apr 2024 18:36:10 +0800 Subject: [PATCH 5/9] fix(autogptq): do not use_triton with qwen-vl (#1985) * Enhance autogptq backend to support VL models * update dependencies for autogptq * remove redundant auto-gptq dependency * Convert base64 to image_url for Qwen-VL model * implemented model inference for qwen-vl * remove user prompt from generated answer * fixed write image error * fixed use_triton issue when loading Qwen-VL model --------- Co-authored-by: Binghua Wu --- backend/python/autogptq/autogptq.py | 1 - 1 file changed, 1 deletion(-) diff --git a/backend/python/autogptq/autogptq.py b/backend/python/autogptq/autogptq.py index bbafdd920855..c7c350286c70 100755 --- a/backend/python/autogptq/autogptq.py +++ b/backend/python/autogptq/autogptq.py @@ -39,7 +39,6 @@ def LoadModel(self, request, context): self.model_name = "Qwen-VL-Chat" model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=request.TrustRemoteCode, - use_triton=request.UseTriton, device_map="auto").eval() else: model = AutoGPTQForCausalLM.from_quantized(model_path, From 36da11a0ee4cdf575012c669150d5f617362e619 Mon Sep 17 00:00:00 2001 From: Koen Farell Date: Wed, 10 Apr 2024 14:25:26 +0300 Subject: [PATCH 6/9] deps: Update version of vLLM to add support of Cohere Command_R model in vLLM inference (#1975) * Update vLLM version to add support of Command_R Signed-off-by: Koen Farell * fix: Fixed vllm version from requirements Signed-off-by: Koen Farell * chore: Update transformers-rocm.yml Signed-off-by: Koen Farell * chore: Update transformers.yml version of vllm Signed-off-by: Koen Farell --------- Signed-off-by: Koen Farell --- backend/python/common-env/transformers/transformers-nvidia.yml | 2 +- backend/python/common-env/transformers/transformers-rocm.yml | 2 +- backend/python/common-env/transformers/transformers.yml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/backend/python/common-env/transformers/transformers-nvidia.yml b/backend/python/common-env/transformers/transformers-nvidia.yml index e8d8155b508f..e12b5dbbd37c 100644 --- a/backend/python/common-env/transformers/transformers-nvidia.yml +++ b/backend/python/common-env/transformers/transformers-nvidia.yml @@ -116,7 +116,7 @@ dependencies: - sudachipy - sudachidict_core - vocos - - vllm==0.3.2 + - vllm>=0.4.0 - transformers>=4.38.2 # Updated Version - transformers_stream_generator==0.0.5 - xformers==0.0.23.post1 diff --git a/backend/python/common-env/transformers/transformers-rocm.yml b/backend/python/common-env/transformers/transformers-rocm.yml index fa245bf4cec3..48fac8bf9606 100644 --- a/backend/python/common-env/transformers/transformers-rocm.yml +++ b/backend/python/common-env/transformers/transformers-rocm.yml @@ -104,7 +104,7 @@ dependencies: - sudachipy - sudachidict_core - vocos - - vllm==0.3.2 + - vllm>=0.4.0 - transformers>=4.38.2 # Updated Version - transformers_stream_generator==0.0.5 - xformers==0.0.23.post1 diff --git a/backend/python/common-env/transformers/transformers.yml b/backend/python/common-env/transformers/transformers.yml index 3b3b8fe7ed00..843b13fa5b44 100644 --- a/backend/python/common-env/transformers/transformers.yml +++ b/backend/python/common-env/transformers/transformers.yml @@ -108,7 +108,7 @@ dependencies: - sudachipy - sudachidict_core - vocos - - vllm==0.3.2 + - vllm>=0.4.0 - transformers>=4.38.2 # Updated Version - transformers_stream_generator==0.0.5 - xformers==0.0.23.post1 From 93f51d80d41b3b3748da41ad4cb7baf8c762890c Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 10 Apr 2024 16:29:46 +0200 Subject: [PATCH 7/9] Update gpt-vision.md Signed-off-by: Ettore Di Giacinto --- docs/content/docs/features/gpt-vision.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/content/docs/features/gpt-vision.md b/docs/content/docs/features/gpt-vision.md index 3afcab16e1bd..827e2c08dfa4 100644 --- a/docs/content/docs/features/gpt-vision.md +++ b/docs/content/docs/features/gpt-vision.md @@ -22,6 +22,17 @@ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/jso "messages": [{"role": "user", "content": [{"type":"text", "text": "What is in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}' ``` +Grammars and function tools can be used as well in conjunction with vision APIs: + +```bash + curl http://10.1.0.36:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "llava", "grammar": "root ::= (\"yes\" | \"no\")", + "messages": [{"role": "user", "content": [{"type":"text", "text": "Is there some grass in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}' +``` + ### Setup +All-in-One images have already shipped the llava model as `gpt-4-vision-preview`, so no setup is needed in this case. + To setup the LLaVa models, follow the full example in the [configuration examples](https://github.com/mudler/LocalAI/blob/master/examples/configurations/README.md#llava). + From 636d487dc84c6f1d99ba7630d8851865091c42cb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 10 Apr 2024 16:30:03 +0200 Subject: [PATCH 8/9] Update gpt-vision.md Signed-off-by: Ettore Di Giacinto --- docs/content/docs/features/gpt-vision.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/content/docs/features/gpt-vision.md b/docs/content/docs/features/gpt-vision.md index 827e2c08dfa4..9e0212734abe 100644 --- a/docs/content/docs/features/gpt-vision.md +++ b/docs/content/docs/features/gpt-vision.md @@ -25,7 +25,7 @@ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/jso Grammars and function tools can be used as well in conjunction with vision APIs: ```bash - curl http://10.1.0.36:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ + curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "llava", "grammar": "root ::= (\"yes\" | \"no\")", "messages": [{"role": "user", "content": [{"type":"text", "text": "Is there some grass in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}' ``` From 92005b9c0285f31e7f29ca4f37e6afa194745cf0 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 10 Apr 2024 16:30:57 +0200 Subject: [PATCH 9/9] Update openai-functions.md Signed-off-by: Ettore Di Giacinto --- docs/content/docs/features/openai-functions.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/content/docs/features/openai-functions.md b/docs/content/docs/features/openai-functions.md index 843524f40623..435101ee8722 100644 --- a/docs/content/docs/features/openai-functions.md +++ b/docs/content/docs/features/openai-functions.md @@ -144,6 +144,15 @@ curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/jso }' ``` +Grammars and function tools can be used as well in conjunction with vision APIs: + +```bash + curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{ + "model": "llava", "grammar": "root ::= (\"yes\" | \"no\")", + "messages": [{"role": "user", "content": [{"type":"text", "text": "Is there some grass in the image?"}, {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg" }}], "temperature": 0.9}]}' +``` + + ## 💡 Examples A full e2e example with `docker-compose` is available [here](https://github.com/go-skynet/LocalAI/tree/master/examples/functions).