Open
Description
Prerequisites
- I am running the latest code. Mention the version if possible as well.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new and useful enhancement to share.
Feature Description
New Wina tech which could further improve the performance of LLM model
https://github.com/microsoft/wina
Motivation
This will made realtime inference feasible with even lower inference hardware, and made GPU support even more concurrent inferences
Possible Implementation
No response