[bug] forwardAsync assertion failed: Unable to get batch slot for reqId

[akhoroshev]

Affected versions: https://github.com/NVIDIA/TensorRT-LLM/tree/31ac30e928a2db795799fdcab6be446bfa3a3998 and https://github.com/NVIDIA/TensorRT-LLM/tree/548b5b73106aaf7374955e1c37aad677678ebc7b

[TensorRT-LLM][ERROR] Could not find sequence id in allocated sequence slots
[TensorRT-LLM][ERROR] Encountered an error in forwardAsync function: [TensorRT-LLM][ERROR] Assertion failed: Unable to get batch slot for reqId (/home/jenkins/agent/workspace/LLM/main/L0_MergeRequest/llm/cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.cpp:1161)
1             0x411e2b tensorrt_llm::common::throwRuntimeError(char const*, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 78
2       0x7f85d9684401 /opt/distr/wmcore/wmcore_3.4/lib/libtensorrt_llm.so(+0x721401) [0x7f85d9684401]
3       0x7f85db7b7e7c tensorrt_llm::batch_manager::TrtGptModelInflightBatching::forwardAsync(std::__cxx11::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > > const&) + 1484
4       0x7f85db7e2661 tensorrt_llm::executor::Executor::Impl::forwardAsync(std::__cxx11::list<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest>, std::allocator<std::shared_ptr<tensorrt_llm::batch_manager::LlmRequest> > >&) + 353
5       0x7f85db7e756f tensorrt_llm::executor::Executor::Impl::executionLoop() + 895
6       0x7f85c59e1a80 /opt/distr/wmcore/wmcore_3.4/lib/libtensorrt_llm_nvrtc_wrapper.so(+0x32c5a80) [0x7f85c59e1a80]
7       0x7f857c9a41ca /opt/distr/wmcore/wmcore_3.4/lib/libpthread.so.0(+0x81ca) [0x7f857c9a41ca]
8       0x7f857bcd08d3 clone + 67

Same error #1869