Added TRTWrapper #7990

borisfom · 2024-08-05T06:20:38Z

Description

Added alternative class to ONNX->TRT export and wrap TRT engines for inference.
It encapsulates filesystem persistence and does not rely on torch-tensortrt for execution.
Also can be used to run ONNX with onnxruntime.

Types of changes

Non-breaking change (fix or new feature that would not break existing functionality).
Breaking change (fix or new feature that would cause existing functionality to change).
New tests added to cover the changes.
Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
In-line docstrings updated.
Documentation updated, tested make html command in the docs/ folder.

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

yiheng-wang-nv · 2024-08-05T06:50:10Z

/black

monai/utils/trt_utils.py

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

monai/utils/export_utils.py

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <[email protected]>

…ppers

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <[email protected]>

…ppers Signed-off-by: Boris Fomitchev <[email protected]>

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Boris Fomitchev <[email protected]>

monai/utils/cast_utils.py

monai/networks/trt_compiler.py

monai/networks/utils.py

…dded Signed-off-by: Boris Fomitchev <[email protected]>

monai/networks/trt_compiler.py

Signed-off-by: Yiheng Wang <[email protected]>

KumoLiu · 2024-08-28T10:37:03Z

/build

Signed-off-by: Boris Fomitchev <[email protected]>

monai/networks/trt_compiler.py

Signed-off-by: Boris Fomitchev <[email protected]>

…ppers Signed-off-by: Boris Fomitchev <[email protected]>

Signed-off-by: Boris Fomitchev <[email protected]>

for more information, see https://pre-commit.ci

KumoLiu · 2024-08-29T05:46:48Z

I'll try to merge this after the base image is updated in this PR. Still working through the failed test cases.

Nic-Ma

Looks good to me.

Thanks.

borisfom · 2024-08-30T14:53:16Z

@KumoLiu : looks like #8049 is ready to merge ?

KumoLiu · 2024-08-30T15:11:59Z

@KumoLiu : looks like #8049 is ready to merge ?

Hi @borisfom, yes, it may need #8058 merge since after updating the base image, there are some cuda sync issue need to be addressed. Will merge these three prs after the test on the blossom all pass. Thanks!

KumoLiu · 2024-08-31T14:26:27Z

/build

KumoLiu · 2024-08-31T15:15:07Z

/build

Sample implementation of using new trt_wrap() from MONAI Depends on Project-MONAI/MONAI#7990 --------- Signed-off-by: Boris Fomitchev <[email protected]> Signed-off-by: Yiheng Wang <[email protected]> Signed-off-by: binliu <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Yiheng Wang <[email protected]> Co-authored-by: binliu <[email protected]>

borisfom and others added 3 commits August 4, 2024 23:17

Added TRTWrapper

054a2b8

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

3ab9c83

for more information, see https://pre-commit.ci

Merge branch 'dev' into trt-wrappers

4ec1d3b

borisfom mentioned this pull request Aug 5, 2024

TRT wrappers for Vista3D Project-MONAI/VISTA#28

Open

4 tasks

yiheng-wang-nv reviewed Aug 5, 2024

View reviewed changes

monai/utils/trt_utils.py Outdated Show resolved Hide resolved

yiheng-wang-nv reviewed Aug 5, 2024

View reviewed changes

monai/utils/trt_utils.py Outdated Show resolved Hide resolved

borisfom and others added 2 commits August 5, 2024 16:18

Addressing code review comments, adding docustrings, cleanup

fe71030

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

6a9727f

for more information, see https://pre-commit.ci

yiheng-wang-nv reviewed Aug 6, 2024

View reviewed changes

monai/utils/export_utils.py Outdated Show resolved Hide resolved

borisfom and others added 18 commits August 5, 2024 23:37

Added TRT 10.3RC to Dockerfile

29d9725

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

5b8b4f2

for more information, see https://pre-commit.ci

Workaround for format check

f31d6dd

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

9303c32

for more information, see https://pre-commit.ci

More format check workarounds

c1d0b19

Signed-off-by: Boris Fomitchev <[email protected]>

More format check workarounds

63c4b70

Signed-off-by: Boris Fomitchev <[email protected]>

More format check workarounds

9a3d6a6

Signed-off-by: Boris Fomitchev <[email protected]>

Using optional exports for trt_utils

8bf0300

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

c03e49b

for more information, see https://pre-commit.ci

Fixing lint errors

39c94c2

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt-wrappers' of github.com:borisfom/MONAI into trt-wra…

35dffcc

…ppers

Format fixed

9d867a7

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

6e2733a

for more information, see https://pre-commit.ci

Fixing flake errors

848a42d

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt-wrappers' of github.com:borisfom/MONAI into trt-wra…

9ade6af

…ppers Signed-off-by: Boris Fomitchev <[email protected]>

Fixing CI

cf2c3b1

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

e8b51f4

for more information, see https://pre-commit.ci

Fixed mypy, Engine refactor

ddb5bc8

Signed-off-by: Boris Fomitchev <[email protected]>

borisfom requested a review from yiheng-wang-nv August 7, 2024 03:32

yiheng-wang-nv reviewed Aug 7, 2024

View reviewed changes

monai/utils/cast_utils.py Outdated Show resolved Hide resolved

KumoLiu reviewed Aug 28, 2024

View reviewed changes

monai/networks/trt_compiler.py Outdated Show resolved Hide resolved

KumoLiu reviewed Aug 28, 2024

View reviewed changes

monai/networks/utils.py Outdated Show resolved Hide resolved

optional propagation of dynamo arg fixed, onnx_graphsurgeon package a…

9481d9f

…dded Signed-off-by: Boris Fomitchev <[email protected]>

binliunls reviewed Aug 28, 2024

View reviewed changes

monai/networks/trt_compiler.py Outdated Show resolved Hide resolved

yiheng-wang-nv reviewed Aug 28, 2024

View reviewed changes

monai/networks/trt_compiler.py Show resolved Hide resolved

yiheng-wang-nv reviewed Aug 28, 2024

View reviewed changes

monai/networks/trt_compiler.py Show resolved Hide resolved

yiheng-wang-nv and others added 2 commits August 28, 2024 17:46

add vista test cases

6a11581

Signed-off-by: Yiheng Wang <[email protected]>

Merge branch 'dev' into trt-wrappers

6e8bd6b

yiheng-wang-nv mentioned this pull request Aug 28, 2024

Added trt_compile configs for vista2d and vista3d Project-MONAI/model-zoo#632

Merged

8 tasks

Merge branch 'dev' into trt-wrappers

3d221cb

Code review input addressed

792a721

Signed-off-by: Boris Fomitchev <[email protected]>

Nic-Ma reviewed Aug 29, 2024

View reviewed changes

monai/networks/trt_compiler.py Outdated Show resolved Hide resolved

borisfom and others added 4 commits August 28, 2024 20:03

Fixed torch-tensorrt path of trt_compile, added test

73ac717

Signed-off-by: Boris Fomitchev <[email protected]>

Merge branch 'trt-wrappers' of github.com:borisfom/MONAI into trt-wra…

ea879f2

…ppers Signed-off-by: Boris Fomitchev <[email protected]>

Fixing tests

1e7e76d

Signed-off-by: Boris Fomitchev <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

47e676e

for more information, see https://pre-commit.ci

borisfom requested review from yiheng-wang-nv and binliunls August 29, 2024 05:35

binliunls approved these changes Aug 29, 2024

View reviewed changes

Merge branch 'dev' into trt-wrappers

dd4d2d6

Nic-Ma approved these changes Aug 29, 2024

View reviewed changes

Merge branch 'dev' into trt-wrappers

6b47a8b

KumoLiu merged commit c9f8d32 into Project-MONAI:dev Sep 1, 2024
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added TRTWrapper #7990

Added TRTWrapper #7990

borisfom commented Aug 5, 2024

yiheng-wang-nv commented Aug 5, 2024

KumoLiu commented Aug 28, 2024

KumoLiu commented Aug 29, 2024

Nic-Ma left a comment

borisfom commented Aug 30, 2024

KumoLiu commented Aug 30, 2024

KumoLiu commented Aug 31, 2024

KumoLiu commented Aug 31, 2024

Added TRTWrapper #7990

Added TRTWrapper #7990

Conversation

borisfom commented Aug 5, 2024

Description

Types of changes

yiheng-wang-nv commented Aug 5, 2024

KumoLiu commented Aug 28, 2024

KumoLiu commented Aug 29, 2024

Nic-Ma left a comment

Choose a reason for hiding this comment

borisfom commented Aug 30, 2024

KumoLiu commented Aug 30, 2024

KumoLiu commented Aug 31, 2024

KumoLiu commented Aug 31, 2024