Python API Beta #299

nnshah1 · 2023-12-07T01:39:38Z

Triton Inference Server In-Process Python API [BETA]

python/CMakeLists.txt

python/test/test_api.py

python/tritonserver/_api/__init__.py

python/tritonserver/_api/_datautils.py

python/CMakeLists.txt

python/build_wheel.py

python/tritonserver/__init__.py

python/tritonserver/_api/wrapper.py

python/test/test_api.py

python/tritonserver/_api/_datautils.py

python/tritonserver/_api/wrapper.py

python/test/test_api.py

Tabrizian

Some minor comments. I think apart from the tensor abstraction and the default allocator changes that we discussed offline it looks good to me.

We definitely need to add more testing but can be part of future PRs.

python/tritonserver/_api/wrapper.py

Co-authored-by: Ryan McCormick <[email protected]>

python/test/test_api.py

python/tritonserver/_c/tritonserver_pybind.cc

Co-authored-by: Ryan McCormick <[email protected]>

oandreeva-nv · 2024-01-11T20:25:55Z

python/tritonserver/_api/_allocators.py

+    @staticmethod
+    def from_dlpack(owner: Any) -> MemoryBuffer:
+        if not hasattr(owner, "__dlpack__"):
+            raise InvalidArgumentError("Object does not support DLpack protocol")


Just a nit: Maybe change to "Object does not have __dlpack__ method". Older versions of DLPack protocol allow passing a capsule irectly to from_dlpack

nnshah1 added 11 commits December 4, 2023 00:02

initial commit of python wrapper api

8848ae8

adding initial api level test

1ce4899

adding api unit test

bd44a33

update to make cupy optional

5f2c3c0

updated with formatting fixes

76baf08

documentation updates

8b52407

updated documentation

fe0368a

updates

513abaa

updating documentation

9e1d813

updating incremental typing improvements

f1fe520

updates for latest - fix cancel

318ebe9

nnshah1 marked this pull request as draft December 7, 2023 01:39

nnshah1 requested review from GuanLuo, rmccorm4, Tabrizian and tanmayv25 December 7, 2023 01:39

mc-nv reviewed Dec 7, 2023

View reviewed changes

python/CMakeLists.txt Outdated Show resolved Hide resolved

nnshah1 added 3 commits December 6, 2023 22:27

updated cmake commands for consistency

02b4125

updated for based on coding style

759dabf

adding parameter setting and error handling

b5782e7

nv-hwoo reviewed Dec 7, 2023

View reviewed changes

python/test/test_api.py Show resolved Hide resolved

python/tritonserver/_api/__init__.py Outdated Show resolved Hide resolved

python/tritonserver/_api/_datautils.py Show resolved Hide resolved

Tabrizian reviewed Dec 7, 2023

View reviewed changes

python/test/test_api.py Outdated Show resolved Hide resolved

nnshah1 added 4 commits December 8, 2023 06:15

copyright headers and tweaks to makefile

9efd589

added metric

fe3cb7a

adding support for blocking unload

b9b5885

updating naming

68ecf20

Tabrizian reviewed Dec 13, 2023

View reviewed changes

python/tritonserver/_api/wrapper.py Outdated Show resolved Hide resolved

python/tritonserver/_api/wrapper.py Outdated Show resolved Hide resolved

python/tritonserver/_api/wrapper.py Outdated Show resolved Hide resolved

nnshah1 marked this pull request as ready for review December 14, 2023 19:46

nnshah1 marked this pull request as draft December 14, 2023 19:47

nnshah1 and others added 10 commits January 10, 2024 14:21

Update python/tritonserver/_api/_allocators.py

5c7bfe3

Co-authored-by: Ryan McCormick <[email protected]>

update for consistency

250c2f9

Update python/tritonserver/_api/_model.py

8d25b00

Co-authored-by: Ryan McCormick <[email protected]>

Update python/tritonserver/_api/_dlpack.py

333e08b

Co-authored-by: Ryan McCormick <[email protected]>

Update python/tritonserver/_api/_dlpack.py

20b9aa5

Co-authored-by: Ryan McCormick <[email protected]>

Update python/tritonserver/_api/_model.py

3e85540

Co-authored-by: Ryan McCormick <[email protected]>

Update python/tritonserver/_api/_model.py

e983404

Co-authored-by: Ryan McCormick <[email protected]>

Update python/tritonserver/_api/_request.py

9e1cc13

Co-authored-by: Ryan McCormick <[email protected]>

removing placeholders for examples

1d60e9b

removing paramter from config file - is passed in via explicit load

121808f

rmccorm4 reviewed Jan 10, 2024

View reviewed changes

python/test/test_api.py Outdated Show resolved Hide resolved

rmccorm4 reviewed Jan 10, 2024

View reviewed changes

python/tritonserver/_c/tritonserver_pybind.cc Show resolved Hide resolved

rmccorm4 previously approved these changes Jan 10, 2024

View reviewed changes

nnshah1 and others added 2 commits January 10, 2024 15:01

Merge branch 'main' into nnshah1-python-api

8f6dce1

Update python/test/test_api.py

fa43784

Co-authored-by: Ryan McCormick <[email protected]>

nnshah1 dismissed rmccorm4’s stale review via fa43784 January 10, 2024 23:03

nnshah1 added 3 commits January 10, 2024 15:18

adding model load retry count to options

befe2a3

adding type hint for model load retry count

0d95d7b

changing to copy over directories instead of using file glob

cbf37e0

rmccorm4 previously approved these changes Jan 11, 2024

View reviewed changes

added test dependencies to wheel, removed typing_extensions

750c655

nnshah1 dismissed rmccorm4’s stale review via 750c655 January 11, 2024 07:43

nnshah1 added 3 commits January 11, 2024 04:26

updated to skip tests requiring torch

8c51e67

update for formatting

ed78404

updating description to Triton Inference Server In-Process Python API

731a722

nnshah1 requested review from Tabrizian and rmccorm4 January 11, 2024 15:23

rmccorm4 approved these changes Jan 11, 2024

View reviewed changes

nnshah1 merged commit 6d0cd38 into main Jan 11, 2024
1 check passed

oandreeva-nv reviewed Jan 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python API Beta #299

Python API Beta #299

nnshah1 commented Dec 7, 2023 •

edited

Loading

Tabrizian left a comment

oandreeva-nv Jan 11, 2024

Python API Beta #299

Python API Beta #299

Conversation

nnshah1 commented Dec 7, 2023 • edited Loading

Tabrizian left a comment

Choose a reason for hiding this comment

oandreeva-nv Jan 11, 2024

Choose a reason for hiding this comment

nnshah1 commented Dec 7, 2023 •

edited

Loading