Phi3-mini-INT4 performance speedup with numpy 2.0 or above

[bkaruman]

Hi!

I am observing significant speedup in tokenization and prompt processing throughput with numpyv2 on Intel CPUs.
Can someone help understand how GenAI makes use of numpy during tokenization/prompt phase?

[natke]

Hi @bkaruman, thanks for bringing to our attention - which Intel CPU are you running on?

[bkaruman]

Hi @bkaruman, thanks for bringing to our attention - which Intel CPU are you running on?

Intel Core Ultra Meteor Lake

[natke]

Thanks. And the only difference is you upgraded to 0.4.0?

How long is the prompt you are running? (Or range of lengths)

[bkaruman]

Thanks. And the only difference is you upgraded to 0.4.0?
Yes, here's the environment I use.
ONNX RT (1.19.0) + ORT GEN(0.4.0) + numpy v2.0.0
I have seen significant difference in perf running numpy 2.0 and numpy 1.24.4
How long is the prompt you are running? (Or range of lengths)
I am running for prompt length = 1024, 2048 and generation length=1024,2048

[natke]

To clarify, you use exactly the same configuration and the only difference is the numpy version?

Can you quantify the speedup?

[natke]

Hi @bkaruman, we can't reproduce the speedup with numpy 2. Closing this issue for now. Please re-open if you continue to observe this behavior.