Some questions regarding the WebLLM tool

[jasonsu123]

Dear WebLLM Team,

I recently came across a Facebook group where someone shared your achievements, and I found it very intriguing. However, I have a few questions about WebLLM that I’d like to ask:

Which country or company/team developed WebLLM?

Does WebLLM work based on the principle of running LLMs locally on a computer? However, it seems to allow the use of WebLLM's default models. In this case, is it considered local operation? What about the privacy risks of conversation data?

Can WebLLM utilize locally downloaded GGUF models, similar to tools like Ollama or LM Studio?
Additionally, does it support integration with OpenAI API keys for usage?

Thank you!

[jeromeetienne]

(just saw your issue, just a web-llm user too )

local model ala ollama would be significantly faster than their web counter part.
so the algo would be

• if ollama is present, use ollama
• else use webgpu

Done this way it seems it would be an optimisation.

i would love it too

[erikh2000]

(I'm not a maintainer)

Does WebLLM work based on the principle of running LLMs locally on a computer?

Sorta. It runs LLMs in the browser (not native). Charlie Ruan descibes the architecture in more detail in this video: https://www.youtube.com/watch?v=MhTCzq7iTy0&pp=ygUMd2ViYWkgd2VibGxt

However, it seems to allow the use of WebLLM's default models. In this case, is it considered local operation?

That depends on your definition of "local". The models are retrieved from a server on the Internet (not local). They are then cached to browser persistent storage and can be executed entirely offline (local).

What about the privacy risks of conversation data?

By design, user prompts and responses aren't sent anywhere outside your device. It's good to be vigilant as a developer though. For example a supply chain attack in one of WebLLM's dependencies might add a backdoor that sends user prompts to a capturing endpoint. The community would probably figure that out pretty quickly.

does it support integration with OpenAI API keys for usage?

No, and that seems outside of its mission to provide local LLM functionality.

[erikh2000]

local model ala ollama would be significantly faster than their web counter part. so the algo would be: if ollama is present, use ollama else use webgpu

Yeah, that's what I did with my project (create-decent-app). I think it's a good approach though it's a little difficult to know you have the exact same model in Ollama or WebLLM.

[jasonsu123]

Can WebLLM utilize locally downloaded GGUF models, similar to tools like Ollama or LM Studio?

Thank you for your response! However, I still have a few questions, such as:

If WebLLM uses default models from the internet, can I still use it completely offline after disconnecting from the network?

Can WebLLM utilize locally downloaded GGUF models from HuggingFace, similar to tools like Ollama or LM Studio?

In what scenarios would using WebLLM be the preferred choice? Otherwise, wouldn't it be better to directly use Ollama for running LLMs locally, especially when combined with OpenWebUI?

Thank you!

[Propfend]

is there some API to infra state? Same as llama.cpp with slots endpoint? would be useful to have that due to prod env.