LLaVA with GGUF files (and --mmproj argument)?

[benmayersohn]

Hi everyone! I'm coming from llama.cpp which currently doesn't support serving multi-modal models (it was removed and hopefully it will return soon).

I'm used to the GGUF format, which is only recently supported by vLLM. In llama.cpp you would do something like this:

./llama-server -t ${N_THREADS} -m ${LLAVA_MODEL} -c ${N_CTX} --host ${HOST} \
  --port ${PORT} -ngl ${N_GPU_LAYERS} --mmproj ${MMPROJ} --temp 0.1 \
  -p $'### User: What do you see?\n### Assistant: '

where

• ${MMPROJ} is a multi-modal projector that aligns the vision and text data (per this overview)
• ${LLAVA_MODEL} is the Llava model itself

and both are two distinct GGUF files. Does anyone know what the equivalent command/arguments be when serving via python -m vllm.entrypoints.openai.api_server?

Thanks!