Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 [Bug] Segmentation Fault in TorchTensorRTModule with PyTorch 2.5.0.dev Due to full_like Converter Replacement #3104

Closed
chohk88 opened this issue Aug 20, 2024 · 0 comments · Fixed by #3077
Assignees
Labels
bug Something isn't working

Comments

@chohk88
Copy link
Collaborator

chohk88 commented Aug 20, 2024

Bug Description

We are encountering a segmentation fault when running the test case for the full_like converter. We have identified some potential causes, but a definitive solution has not yet been found.

Code Context

File: py/torch_tensorrt/dynamo/runtime/_TorchTensorRTModule.py

outputs: List[torch.Tensor] = torch.ops.tensorrt.execute_engine(
    list(input_tensors), self.engine
)

This line is part of the forward method, defined as follows:

def forward(self, *inputs: Any) -> torch.Tensor | Tuple[torch.Tensor, ...]:

Class: TorchTensorRTModule(torch.nn.Module)

Behavior Based on PyTorch Version

  • PyTorch 2.4.0a: The function executes without any issues.

  • PyTorch 2.5.0.dev20240807+cu124: When running the full_like converter test case, the program terminates abruptly with a segmentation fault, and no error message is displayed.

Additional Details

This issue consistently reproduces both in the CI environment and in a local setup using torch 2.5.0.dev20240807+cu124.
When converting, the shape information is extracted from the inputs, but the input tensor itself is lost.

To Reproduce

With PyTorch 2.5.0, run the following command:

python -m pytest tests/py/dynamo/lowering/test_decompositions.py -k test_lowering_full_like_module -s

This will reproduce the segmentation fault.

image

Expected Behavior

With PyTorch 2.4.0a, running the same test case should complete successfully without any issues:

python -m pytest tests/py/dynamo/lowering/test_decompositions.py -k test_lowering_full_like_module -s

image

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

  • pytorch-triton: 3.0.0+dedb7bdf33
  • torch: 2.5.0.dev20240807+cu124
  • torch_tensorrt: 2.5.0.dev0+cf390fa55 /TensorRT/py
  • torchvision: 0.19.0.dev20240617+cu124
@chohk88 chohk88 added the bug Something isn't working label Aug 20, 2024
@chohk88 chohk88 self-assigned this Aug 20, 2024
@chohk88 chohk88 linked a pull request Aug 20, 2024 that will close this issue
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant