cross attention hook

[pkyoung]

I've tested code as in Multilingual_ASR.ipynb,
only got the following error while reading kv values in cross attention cache

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[19], line 19
     16 with torch.no_grad():
     17     logits = model(mel.unsqueeze(0), tokens.unsqueeze(0))
---> 19 weights = torch.cat(QKs)  # layers * heads * tokens * frames    
     20 weights = weights[:, :, :, : duration // AUDIO_SAMPLES_PER_TOKEN].cpu()
     21 weights = median_filter(weights, (1, 1, 1, medfilt_width))

TypeError: expected Tensor as element 0 in argument 0, but got NoneType

How to solve the problem?

[jason-adriel]

By any chance, did you have SDPA enabled? The implementation currently returns the attention weights as the None type when SDPA is enabled. Add the disable_sdpa() context manager before you run the model inference and see if that fixes the issue.

[Ahmad-Haffez]

changed:
with torch.no_grad():
logits = model(mel.unsqueeze(0), tokens.unsqueeze(0))
To:
from whisper.model import disable_sdpa
with torch.no_grad(),disable_sdpa():
logits = model(mel.unsqueeze(0), tokens.unsqueeze(0))

still getting the error, any ideas? and its working now, thanks