[experimental-webgpu] - Configuring Encoder/Decoder Precision with dtype for Local Models

[kostia-ilani]

Hello,

I’m using whisper-web (experimental-webgpu branch) with local models,
(env.allowLocalModels = true and env.localModelPath = "./models"), and facing challenges in setting distinct dtype values for encoder_model and decoder_model_merged with a - small model.

The error I see -

Uncaught (in promise) Error: Can't create a session. ERROR_CODE: 7, ERROR_MESSAGE: Failed to load model because protobuf parsing failed.

Is there a specific convention for key names or values when setting dtype for encoder/decoder precision levels (according to the models ONNX files?

const transcriber = await pipeline(
  "automatic-speech-recognition",
  "my-whisper-model",
  {
    dtype: {
      encoder_model: "fp32",
      decoder_model_merged: "q4"
    },
    device: "webgpu"
  }
);

[xenova]

Might be related to huggingface/transformers.js#1025 (comment)
(Have you pulled the git lfs files into your local folder?)

[kostia-ilani]

Thanks for the fast reply. @xenova
I use local files stored under ./models, so it's not a git issue.

The files were taken from
https://huggingface.co/Xenova/whisper-small/tree/main/onnx

Do you have any assumptions about what might be the issue?