Batch prefill tokens uses max input tokens as default#320

Open

noyoshi wants to merge 2 commits intomainfrom batch-prefill-tokens-better-default

+12-3

Commits on Mar 11, 2024

use default value derived from max input tokensgst
noyoshi
committed
using the value of max input tokens to derive default max batch prefill tokens
noyoshi
committed