Add fp8 support for llama model family on Navi4x #245

qli88 · 2024-10-25T05:32:50Z

[MISC] Add FP8 support for llama model family on Navi4x

vllm/utils.py

gshtras · 2024-10-25T16:40:19Z

Great job!
Please fix linters and consider the proposed navi check change, and then it's GTG

2. change implementation of is_navi4x ( from env variable to cuda query)

vllm/model_executor/models/llama.py

vllm/utils.py

2. Remove unnecessary detection of Navi4x platform;

qli88 added 2 commits October 24, 2024 17:08

Add fp8 support for Llama model family on Navi4x

99bb48d

Fix a typo in example

68b9476

qli88 requested review from charlifu, gshtras and maleksan85 October 25, 2024 05:32

gshtras reviewed Oct 25, 2024

View reviewed changes

vllm/utils.py Show resolved Hide resolved

[misc] 1. format updates (split long lines);

eb931f4

2. change implementation of is_navi4x ( from env variable to cuda query)

qli88 requested a review from gshtras October 25, 2024 19:47

gshtras requested changes Oct 25, 2024

View reviewed changes

[misc] 1. Add platform detection before using torch.cuda;

ae81124

2. Remove unnecessary detection of Navi4x platform;

qli88 requested a review from gshtras October 25, 2024 21:26

gshtras approved these changes Oct 25, 2024

View reviewed changes

qli88 merged commit 4bba092 into main Oct 25, 2024
16 of 17 checks passed

gshtras deleted the qiang-navi4x-fp8-llama branch October 25, 2024 21:40

Provide feedback