Inference Qwen2-0.5b + Medusa failed

[shangshng]

System Info

A100

[TensorRT-LLM] TensorRT-LLM version: 0.16.0.dev2024120300

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. convert qwen 0.5b + 5 medusa heads to fp16 checkpoints
2. build trtllm engine, max_batch_size = 1, no special configs
3. using summarize.py inference

Expected behavior

get the answers

actual behavior

The engine is successfully loaded but the generation fails.

additional notes

The trtllm version is 0.16.0.dev2024120300 and this is the last version to successfully build qwen2 + medusa engines. In the subsequent versions, when engines are built, there is an error message shown as bellow. If I added the missing configs manually, the inference results are messy.

[nv-guomingz]

Hi @shangshng thanks for your patience.
Could u plz share us the original model download path and reproduce step?