awscurl: WARN maxLength is not explicitly specified, use modelMaxLength: 512

[CoolFish88]

Description

When requesting tokens per second in benchmark metrics (-t option specified) while providing the path to the tokenizer.json file as well as a payloads dataset, aws curl return the warning: WARN maxLength is not explicitly specified, use modelMaxLength: 512

I used a vLLM engine for which OPTION_MAX_MODEL_LEN was set to a higher value than 512.

(1) Are modelMaxLength and OPTION_MAX_MODEL_LEN?
(2) Is modelMaxLength overwriting OPTION_MAX_MODEL_LEN?
(3) Are token metrics impacted by this truncation?

Remark: A "tps" metric (requests per second) is included in the benchmark response. Since "tps" is usually used as an acronym for Tokens per second within the inference domain, requests per second could benefit from another abbreviation. Does "tokenThroughput" represent in this case tokens per second?

Expected Behavior

Use model server parameter as opposed to setting it to a default value

Error Message

WARN maxLength is not explicitly specified, use modelMaxLength: 512

How to Reproduce?

TOKENIZER=<path_to_tokenizer> ./awscurl -c 1 -N 10 -X POST -n sagemaker <endpoint_url> --dataset <path_to_dataset> -H 'Content-Type: application/json' -P --connect-timeout 60 -t

[frankfliu]

The warning is just for the awscurl tool. awscurl uses the tokenizer to calculate the number of output token.

I created a PR that allows you to configure the max length with env var: #2341

[sindhuvahinis]

@CoolFish88 The PR is now merged. Let us know if you need anything else.