Skip to content

Commit

Permalink
bugfix: Fix the default value of data_type in batch decode plan fun…
Browse files Browse the repository at this point in the history
…ction (#544)

As mentioned in #543 , the behavior of batch decode plan function is
problematic if we use the combo of `q_data_type` and `kv_data_type`.

cc @cyang49
  • Loading branch information
yzh119 authored Oct 22, 2024
1 parent 41ebe6d commit b2a9e16
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion python/flashinfer/decode.py
Original file line number Diff line number Diff line change
Expand Up @@ -499,9 +499,9 @@ def plan(
pos_encoding_mode: str = "NONE",
window_left: int = -1,
logits_soft_cap: Optional[float] = None,
data_type: Optional[Union[str, torch.dtype]] = "float16",
q_data_type: Optional[Union[str, torch.dtype]] = "float16",
kv_data_type: Optional[Union[str, torch.dtype]] = None,
data_type: Optional[Union[str, torch.dtype]] = None,
sm_scale: Optional[float] = None,
rope_scale: Optional[float] = None,
rope_theta: Optional[float] = None,
Expand Down

0 comments on commit b2a9e16

Please sign in to comment.