Performance Testing Strategy

[ncomly-nvidia]

Performance Testing Strategy for Torch-TensorRT

Questions needed answered:

• Model set to perf test, ideally some public available set
• frequency of testing, at minimum at the time of each release
• where / if results will be publish publicly
• regression threshold
• etc.

@dheerajperi to update this ticket with the answers to the above by 7/14

[peri044]

Perf testing

Torchbench analysis

Torchbench : https://github.com/pytorch/benchmark (benchmarks both training and inference performance)

• using models directly from torchvision
• Installation might run into issues if you don't have latest torch, torchvision, torchtext
• Doesn't support torch==1.11 which Torch-TRT currently supports. symbol mismatch errors.

Models of interest currently in Torchbench:

• alexnet (directly from torchvision)

• vgg16 (directly from torchvision)

• resnet 18/50 (directly from torchvision)

• mobilenet_v1/v2 (directly from torchvision)

• efficient_net (directly from timm)

• efficient_det (directly from timm)

• HF_BERT (from hugging face)

List of all models : https://github.com/pytorch/benchmark/tree/main/torchbenchmark/models

The above models are directly inherited from torchvision or timm which is same as what we are doing in our perf benchmark today

Integrating Torch bench pain points

There won't be any code in https://github.com/pytorch/TensorRT/tree/master/tools/perf. Instead the

workflow would look like

git clone https://github.com/pytorch/benchmark.git
cd benchmark
python install.py
python run.py alexnet -d cuda --torch_trt # for a specific model

• Adding the whole torchbench as a dependency is something to consider. It comes with extra dependencies/modules that might not be used by Torch-TRT
• torchbench doesn't have support or releases for respective pytorch versions. It supports the latest and this will be incompatible with Torch-TRT
• Lot of contribution to the pytorch/benchmark repo is needed to accept compile_spec as input to torch_trt and supporting inference in native TRT given a TRT engine. Both of which are currently supported by our perf utility.

Alternative

We can expand on our perf utility with the same model list as used by Meta. More control and hermetic. We can add fx2trt as a backend to our existing utility.

@ncomly-nvidia @narendasan Let me know your thoughts or any missing items we need to explore.

[narendasan]

I'd rather use existing infra that lets us do 1:1 comparisons easier. Adding components to benchmark (provided they are okay with it) for TorchTRT-[TS/FX] and TRT based execution seems much easier than maintaining our own tool

[ncomly-nvidia]

@narendasan TRT engg & product will need to be involved if we create a public BM for TensorRT.

[ncomly-nvidia]

There won't be any code in https://github.com/pytorch/TensorRT/tree/master/tools/perf. Instead the

workflow would look like [...]

Then is the plan to submit PRs to TB to add such functionality?

torchbench doesn't have support or releases for respective pytorch versions. It supports the latest and this will be incompatible with Torch-TRT

By latest is that PyT master or the PyT main?

Lot of contribution to the pytorch/benchmark repo is needed to accept compile_spec as input to torch_trt and supporting inference in native TRT given a TRT engine. Both of which are currently supported by our perf utility.

What are some trade-off of just using the TB models in our perf utility, instead of modifying TB directly? Can we characterize TB perf / evaluation vs our per utility's?

[peri044]

Then is the plan to submit PRs to TB to add such functionality?

Yes, we would have to basically submit PRs to TB adding features very similar to our perf utility.

By latest is that PyT master or the PyT main?

I installed pytorch 1.12 and it worked. So it is latest pytorch release as per my understanding. Even 1.12 support is not explicity listed btw.

Can we characterize TB perf / evaluation vs our per utility's?

Our perf utility can accept multiple settings (eg: truncate_long_and_double, device, native_trt execution). All these are missing in TB currently.

[narendasan]

By latest is that PyT master or the PyT main?
We are switching to nightly after this release anyway, so this doesnt seem like a blocker

[peri044]

At this moment, pytorch latest release (not nightly) worked for me during installation. It is not very clear which one torchbench uses.

[frank-wei]

Add TB PoC @xuzhao9 for some discussions. A good question for @xuzhao9 , does TB consider roll out with release tag?

I think it is good to maintain our own benchmark as it is controllable and debug-able. It could be a playground for our users to have a quick try without installation of TB.
But I also suggest we could consider TB as our benchmark target for a few reasons: 1) TB is adopted in Meta infra team as key indicator for our tools' model coverage. 2) TB developer has devoted a lot of efforts to maintain this benchmark and it will save us a lot of time. 3) TB model coverage is pretty wide 4) It has integrated Torch-TensorRT(and FX). Overall, we could consider TB as an advanced benchmarks for testing our coverage and model efficiency. We could use it for two purposes at least 1) model coverage guide 2) perf demo with our future release.

[ncomly-nvidia]

I'll +1 to this. TB will need to be tested against for standardization, however, we'll want to BM models outside of TB too.

[ncomly-nvidia]

@dheerajperi can you please update current status? Integration w/ Torchbench, stand-alone bm tool, etc.