Add instructions for running vLLM backend #8
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
dyastremsky
changed the title
dyastremsky
changed the title
Co-authored-by: Neelay Shah <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>
| ``` | ||
| mkdir -p /opt/tritonserver/backends/vllm | ||
| wget -P /opt/tritonserver/backends/vllm https://raw.githubusercontent.com/triton-inference-server/vllm_backend/main/src/model.py | ||
| ``` |
There was a problem hiding this comment.
Not an action item here, but a random food for thought that could be nice for both users and developers. If we standardize on a certain python-based-backend git repository structure, we can do something like:
git clone https://github.com/triton-inference-server/vllm_backend.git /opt/tritonserver/backends
- Single command
- Developers could iterate on the backend directly in the git repo and just reload triton without copying files/builds around (developer experience)
- More support for multi-file implementations. The
wgetis nice, but won't scale past a single file. Ex: Imaginemodel.pyimplementsTritonPythonModelbut importsimplementation.pythat has all the gorey details for certain features.
Just some random Tuesday ideas in my head. Core would just be updated to also look for src/model.py or whatever standard we set instead of just model.py.
There was a problem hiding this comment.
this will not work with git clone, since required model.py is in sub-directory of vllm_backend, plus clone will clone tests as well.
We can discuss the best solution at some point.
There was a problem hiding this comment.
For the ease of development, I think your earlier idea of symlinks makes more sense.
There was a problem hiding this comment.
this will not work with git clone, since required model.py is in sub-directory of vllm_backend, plus clone will clone tests as well.
I know it won't work as-is and would require minor changes. Not necessarily asking for this feature at this time, just food for thought.
There was a problem hiding this comment.
We have a separate goal of improving python backend developer experience (more for things like debugging, ipdb, etc) somewhere in the pipeline, so this came to mind as a tangential idea.
There was a problem hiding this comment.
I see, by any chance, do you know in what ticket this is tracked? If you don't remember, then no worries
Co-authored-by: Ryan McCormick <[email protected]>
Co-authored-by: Ryan McCormick <[email protected]>
|
LGTM besides a minor suggestion. Great work @dyastremsky ! |
Co-authored-by: Tanmay Verma <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>
Co-authored-by: Neelay Shah <[email protected]>
|
Amazing work on this, @dyastremsky ! |
Co-authored-by: Neelay Shah <[email protected]> Co-authored-by: Olga Andreeva <[email protected]> Co-authored-by: Ryan McCormick <[email protected]> Co-authored-by: Tanmay Verma <[email protected]>
Draft documentation to allow users to quickly use the vLLM backend to run their models.