From dfaf9a4552d67bb54880e71a0fd498e012a88acd Mon Sep 17 00:00:00 2001 From: jiqing-feng Date: Fri, 20 Sep 2024 05:00:37 -0400 Subject: [PATCH] cpu benchmark --- docs/source/non_cuda_backends.mdx | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/source/non_cuda_backends.mdx b/docs/source/non_cuda_backends.mdx index fca586534..5abe4c1b4 100644 --- a/docs/source/non_cuda_backends.mdx +++ b/docs/source/non_cuda_backends.mdx @@ -24,4 +24,18 @@ Thank you for your support! ### Intel +The following performance data is collected from Intel 4th Gen Xeon (SPR) platform. The tables show speed-up and memory compared with different data types of [Llama-2-7b-chat-hf](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf). + +For inference: +| CPU | BF16 | INT8 | NF4 | FP4 | +|---|---|---|---|---| +| speed-up | 1.0x | 0.6x | 2.3x | 0.03x | +| memory | 13.1G | 7.6G | 5.0G | 4.6G | + +For fine-tune: +| CPU | AMP BF16 | INT8 | NF4 | FP4 | +|---|---|---|---|---| +| speed-up | 1.0x | 0.38x | 0.07x | 0.07x | +| memory | 40G | 9G | 6.6G | 6.6G | + ### AMD