Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADD: support of fp16 python inference backend #773

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

skprot
Copy link

@skprot skprot commented Nov 18, 2021

Enviroment:

CUDA Version: 11.4
TensorRT version: 8.2.0.6-1+cuda11.4
ONNX version: 1.10.1
Python version: 3.8.10

Issue:

While working around with converted to onnx model with fp16 precision i have found the following issue:

Error Code 4: Internal Error (fp16 precision has been set for a layer or layer output, but fp16 is not configured in the builder)

This error occurs when building an engine in python when calling the command:
engine = backend.prepare(model, device='CUDA:0', verbose=True)

Then it was found that in the backend.py file there was no initialization of the fp16 flag in the config for building the engine.

Solution:

The solution is inspired by the still open pull request #345, but it's for the old version of TensorRT. The current solution works on the latest version of TensorRT, which at the time of the pull request is 8.2.0.6.

@skprot skprot force-pushed the master branch 2 times, most recently from ebef1c0 to 67f35a2 Compare November 18, 2021 13:06
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Sayan Protasov seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants