Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix quark fp8 format loading. #395

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

fxmarty-amd
Copy link

@fxmarty-amd fxmarty-amd commented Jan 31, 2025

This PR fixes an issue in the order requantize_with_max_scale is called when loading a checkpoint using quark config.json/checkpoint format.

Compare the following:

normalize_e4m3fn_to_e4m3fnuz should be called before requantize_with_max_scale.

@gshtras
Copy link
Collaborator

gshtras commented Jan 31, 2025

A separate PR is not required. Once the upstream one is accepted, we'll get it within 1 week usually

@fxmarty-amd
Copy link
Author

@gshtras I think ROCm/vllm is used more widespread internally so if this can get included here before merging upstream, it is nice, but I can close this if you prefer!

input_scale=layer.input_scale)
if input_scale is not None:
layer.input_scale = Parameter(input_scale,
requires_grad=False)

max_w_scale, weight = requantize_with_max_scale(
weight=weight,
weight_scale=max_w_scale,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

both weight and max_w_scale are undefined if current_platform.is_rocm() is False.

@BowenBao
Copy link

BowenBao commented Jan 31, 2025

Do you observe better accuracy with this change? Mathematically the order should not matter.

edit: taking this back, the order does matter since otherwise the requantize_with_max_scale is requanting into fnuz using fn scales. Let's point this out in the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants