Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[E2E] Torchbench models load weight got failed #1220

Closed
Tracked by #1223
mengfei25 opened this issue Dec 26, 2024 · 2 comments
Closed
Tracked by #1223

[E2E] Torchbench models load weight got failed #1220

mengfei25 opened this issue Dec 26, 2024 · 2 comments

Comments

@mengfei25
Copy link
Contributor

🐛 Describe the bug

xpu train functorch_maml_omniglot
Traceback (most recent call last):
File "/home/sdp/actions-runner/_work/torch-xpu-ops/pytorch/benchmarks/dynamo/common.py", line 4886, in run
) = runner.load_model(
File "/home/sdp/actions-runner/_work/torch-xpu-ops/pytorch/benchmarks/dynamo/torchbench.py", line 312, in load_model
benchmark = benchmark_cls(
File "/home/sdp/actions-runner/_work/torch-xpu-ops/benchmark/torchbenchmark/util/model.py", line 39, in call
obj = type.call(cls, *args, **kwargs)
File "/home/sdp/actions-runner/_work/torch-xpu-ops/benchmark/torchbenchmark/models/functorch_maml_omniglot/init.py", line 73, in init
self.meta_inputs = torch.load(f'{root}/maml_omniglot/batch.pt')
File "/home/sdp/miniforge3/envs/e2e_ci/lib/python3.10/site-packages/torch/serialization.py", line 1493, in load
raise pickle.UnpicklingError(_get_wo_message(str(e))) from None
_pickle.UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, �[1mdo those steps only if you trust the source of the checkpoint�[0m.
(1) In PyTorch 2.6, we changed the default value of the weights_only argument in torch.load from False to True. Re-running torch.load with weights_only set to False will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
(2) Alternatively, to load with weights_only=True please check the recommended steps in the following error message.
WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray._reconstruct was not an allowed global by default. Please use torch.serialization.add_safe_globals([_reconstruct]) or the torch.serialization.safe_globals([_reconstruct]) context manager to allowlist this global if you trust this class/function.

Check the documentation of torch.load to learn more about types accepted by default with weights_only https://pytorch.org/docs/stable/generated/torch.load.html.

Versions

env:
python: 3.10
XPU_OPS: 9ed0a1a
TRITON_COMMIT_ID: e98b6fcb8df5b44eb0d0addb6767c573d37ba024
TORCH_COMMIT_ID: 4f8b7c4272db521f7ffc4070ce1bdece513d1183
TORCHBENCH_COMMIT_ID: 03cde49eba0580ed17f9ae2250832fd8af4ed756
TORCHVISION_COMMIT_ID: d23a6e1664d20707c11781299611436e1f0c104f
TORCHAUDIO_COMMIT_ID: a6b0a140cc13216975e8922093459019537bb80a
TRANSFORMERS_VERSION: 243e186efbf7fb93328dd6b34927a4e8c8f24395
TIMM_COMMIT_ID: ac3470188b914c5d7a5058a7e28b9eb685a62427
DRIVER_VERSION: 1.23.10.49.231129.50
KERNEL_VERSION: 5.15.0-73-generic #80-Ubuntu SMP Mon May 15 15:18:26 UTC 2023
BUNDLE_VERSION: 2025.0.1.20241113
OS_PRETTY_NAME: Ubuntu 22.04.2 LTS
GCC_VERSION: 11

@mengfei25
Copy link
Contributor Author

Same as #510

@mengfei25
Copy link
Contributor Author

Use pytorch pinned torchbench as CUDA and should be fixed in #1226

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants