Adding custom sampling config

[buddhapuneeth]

System Info

Branch: v0.15
HW

EC2: g6e.12x
GPU: L40S
CUDA: 550.127.05

I am trying to extend the executor sampling config with a custom config and try to plumb it. But running into error while compiling. I am using the cpp backend (model_runner_cpp.py).

I used Llama2 7b model.

compilation commands:

python convert_checkpoint.py --model_dir /code/tensorrt_llm/Llama-2-7b-hf/ --output_dir /tmp/ws/models/compiled/chpt --dtype float16

trtllm-build --checkpoint_dir   /tmp/ws/models/compiled/chpt --output_dir /tmp/ws/models/compiled/engine/fp16/1-gpu --gemm_plugin float16 --gather_all_token_logits

runtime command:

/code/tensorrt_llm/examples/llama# python ../run.py --max_output_len=20 --tokenizer_dir /code/tensorrt_llm/Llama-2-7b-hf/ --engine_dir /tmp/ws/models/compiled/engine/fp16/1-gpu --input_text "In Bash, how do I list all text files?" --output_logits_npy logits.npy --temperature 1.2 --repetition_penalty 1.4

Issue:

Here are my changes:

lto-wrapper: warning: using serial compilation of 39 LTRANS jobs
/usr/bin/ld: /tmp/ccEhVY9h.ltrans12.ltrans.o: in function `tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)':
<artificial>:(.text+0x4ed0): undefined reference to `tensorrt_llm::executor::SamplingConfig::SamplingConfig(int, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<unsigned long> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<int> const&, std::optional<int> const&)'
/usr/bin/ld: /tmp/ccEhVY9h.ltrans17.ltrans.o: in function `pybind11::cpp_function::initialize<pybind11::detail::initimpl::factory<tensorrt_llm::pybind::executor::InitBindings(pybind11::module_&)::{lambda(int, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<unsigned long>, std::optional<unsigned long> const&, std::optional<float> const&, std::optional<int>, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<int> const&, std::optional<int> const&)#6}, pybind11::detail::void_type (*)(), std::unique_ptr<tensorrt_llm::executor::SamplingConfig, std::default_delete<tensorrt_llm::executor::SamplingConfig> > (int, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<unsigned long>, std::optional<unsigned long> const&, std::optional<float> const&, std::optional<int>, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<int> const&, std::optional<int> const&), pybind11::detail::void_type ()>::execute<pybind11::class_<tensorrt_llm::executor::SamplingConfig>, pybind11::arg_v, pybind11::kw_only, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<tensorrt_llm::executor::SamplingConfig>&, pybind11::arg_v const&, pybind11::kw_only const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&) &&::{lambda(pybind11::detail::value_and_holder&, int, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<unsigned long>, std::optional<unsigned long> const&, std::optional<float> const&, std::optional<int>, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<int> const&, std::optional<int> const&)#1}, void, pybind11::detail::value_and_holder&, int, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<unsigned long>, std::optional<unsigned long> const&, std::optional<float> const&, std::optional<int>, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<int> const&, std::optional<int> const&, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::detail::is_new_style_constructor, pybind11::arg_v, pybind11::kw_only, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::class_<tensorrt_llm::executor::SamplingConfig>&&, void (*)(pybind11::detail::value_and_holder&, int, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<unsigned long>, std::optional<unsigned long> const&, std::optional<float> const&, std::optional<int>, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<int> const&, std::optional<int> const&), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::detail::is_new_style_constructor const&, pybind11::arg_v const&, pybind11::kw_only const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) [clone .lto_priv.0]':
<artificial>:(.text+0x3c25): undefined reference to `tensorrt_llm::executor::SamplingConfig::SamplingConfig(int, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<unsigned long> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<float> const&, std::optional<int> const&, std::optional<int> const&, std::optional<int> const&)'

executor.txt
samplingConfig.txt

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I added already the diffs in the above description.

Expected behavior

Should able to compile.

actual behavior

Compilation error.

additional notes

N/A.

[nv-guomingz]

hi @buddhapuneeth I think it'd be better to file one PR with your changes, so our engineer could review the code and rootcause the issue easily.