FastAPI-BitNet

[grctest]

I've created the project FastAPI-BitNet which uses Uvicorn + FastAPI (Python) and Docker to provide a reliable REST API for testing BitNet out locally!

It supports running the inference framework, running bitnet model benchmarks and calculating bitnet model perplexity values.

https://github.com/grctest/FastAPI-BitNet

[CementZhang]

@grctest
your project is the same as the way run_inference.pyrun_inference.py executes system commands. Your project simply encapsulates run_command. If you provide API interfaces as a server, you can consider using the resident memory mode instead of loading the model into memory from scratch every time (this is time-consuming and inefficient). For details, you can refer to the "llama.cpp" method and use "llama_cpp.server" to provide services。

[abstratium-dev]

llama_server isn't built when building this repo from sources as described in the readme, so I used Claude 3.7 to work out what to change and turned it into a docker image. Instructions are here for anyone wanting to replicate this: https://github.com/abstratium-dev/bitnet-llm?tab=readme-ov-file