Clarification on Context Length for Code Generation

[ramaneswaran]

Hi team,

Thank you for providing the benchmark and inference examples—they’ve been incredibly helpful.

I have a question regarding the context length used during generation. In inference/example.py, it seems that the context is not truncated and is used in its entirety, even in cases where it exceeds 16K tokens.

In my experiments, I’ve set the context length to 12K tokens, but I haven’t been able to generate any code that successfully passes the test cases.

Could you please clarify, what is the maximum context length used in your inference setup?

[shanchaoL]

Hi ramaneswaran,

Thank you for your question and for sharing your observations.

The maximum context length in our inference setup is 12K tokens. The example.py file serves as a demonstration of the inference process and doesn't enforce strict truncation logic.

During actual inference, we ensure that both the system prompt and the target function's description are always included in the context. If the total context exceeds the 12K token limit, we truncate other contexts to ensure that the system prompt and target function's description remain intact within the allowed token length.