How to use this in continue with Ollama?

[dayekuaipao]

No description provided.

[Shadowfita]

I have the same question! Auto complete doesn't seem to be working for some reason.

[SlavikCA]

Here is my config:

  "models": [
    {
      "title": "qwen2.5-coder",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b-instruct-q6_K",
      "apiBase": "http://192.168.0.44:11434/"
    }
  ],

Of course, the model has to be downloaded first.

Auto-complete doesn't work for me, too.

[huybery]

Please give us some time to give you all a best practice.

[dayekuaipao]

Here is my config:

  "models": [

    {

      "title": "qwen2.5-coder",

      "provider": "ollama",

      "model": "qwen2.5-coder:7b-instruct-q6_K",

      "apiBase": "http://192.168.0.44:11434/"

    }

  ],

Of course, the model has to be downloaded first.

Auto-complete doesn't work for me, too.

you should use base model for autocompleting, but it doesn't work too.

[SlavikCA]

So, the the rule is to use base model for autocompleting and instruct model for chat?
Or can base model be used for chat, too?

[dayekuaipao]

So, the the rule is to use base model for autocompleting and instruct model for chat?

Or can base model be used for chat, too?

you should use instruct model for chat, it works. And you should use base model for autocompleting,it works for other model such as starcoder, but not for this model

[Shadowfita]

It's interesting because the readme talks about autocomplete support. I wonder if this is an issue with the continue.dev extension and the way it implements it, rather than the model.

[asmeurer]

FIM does seem to work with qwen2.5-coder-base:

$ollama run qwen2.5-coder:7b-base
>>> <|fim_prefix|>def fib(n):<|fim_suffix|>    return b<|fim_middle|>

   a = 0
   b = 1

   for i in range(2,n+1):
      c=a+b
      a=b
      b=c
<|endoftext|>class Solution(object):

(I control-C'd there because the ollama model doesn't seem to be configured to stop on <|endoftext|>).

I believe this needs to be added to the continue templates file https://github.com/continuedev/continue/blob/main/core/autocomplete/templates.ts

However, there may be a way to configure a template directly with continue https://docs.continue.dev/customize/config (maybe the tabAutocompleteOptions/template key?)

[rjmacarthy]

Hey all, FYI I've added support for this model using Ollama recently in https://github.com/twinnydotdev/twinny. Hope it helps someone. All the best.

[SlavikCA]

Looks like autocomplete for Qwen2.5-coder will be added to continue.dev, too:

continuedev/continue#2330

[LNTH]

This is a workaround solution for autocomplete with vllm (I don't use ollama).
I also found that Instruct model can work with autocomplete so I just have to load 1 model for both tasks (chat & autocomplete).

vllm serve Orion-zhen/Qwen2.5-Coder-7B-Instruct-AWQ --quantization awq --disable-log-requests --disable-log-stats

Continue config.json

{
  "models": [
    {
      "title": "Qwen2.5-Coder-7B-Instruct",
      "provider": "openai",
      "apiBase": "http://192.168.1.19:8000/v1/",
      "model": "Orion-zhen/Qwen2.5-Coder-7B-Instruct-AWQ"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5-Coder-7B-Instruct",
    "provider": "openai",
    "apiKey": "None",
    "completionOptions": {
      "stop": [
        "<|endoftext|>",
        "\n"
      ]
    },
    "apiBase": "http://192.168.1.19:8000/v1/",
    "model": "Orion-zhen/Qwen2.5-Coder-7B-Instruct-AWQ"
  },
  "tabAutocompleteOptions": {
    "multilineCompletions": "never",
    "template": "You are a helpful assistant.<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
  },
  "customCommands": [],
  "allowAnonymousTelemetry": false,
  "docs": []
}

continuedev/continue#2388

Edit: correct config.json