An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet)
Architecture | Workflow | LLMs Recipes | Results | Documentations
Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks such as TensorFlow, PyTorch, ONNX Runtime, and MXNet, as well as Intel extensions such as Intel Extension for TensorFlow and Intel Extension for PyTorch. In particular, the tool provides the key features, typical examples, and open collaborations as below:
-
Support a wide range of Intel hardware such as Intel Gaudi Al Accelerators, Intel Core Ultra Processors, Intel Xeon Scalable Processors, Intel Xeon CPU Max Series, Intel Data Center GPU Flex Series, and Intel Data Center GPU Max Series with extensive testing; support AMD CPU, ARM CPU, and NVidia GPU through ONNX Runtime with limited testing; support NVidia GPU for some WOQ algorithms like AutoRound and HQQ.
-
Validate popular LLMs such as LLama2, Falcon, GPT-J, Bloom, OPT, and more than 10,000 broad models such as Stable Diffusion, BERT-Large, and ResNet50 from popular model hubs such as Hugging Face, Torch Vision, and ONNX Model Zoo, with automatic accuracy-driven quantization strategies
-
Collaborate with cloud marketplaces such as Google Cloud Platform, Amazon Web Services, and Azure, software platforms such as Alibaba Cloud, Tencent TACO and Microsoft Olive, and open AI ecosystem such as Hugging Face, PyTorch, ONNX, ONNX Runtime, and Lightning AI
- [2024/07] From 3.0 release, framework extension API is recommended to be used for quantization.
- [2024/07] Performance optimizations and usability improvements on client-side.
# Install 2.X API + Framework extension API + PyTorch dependency
pip install neural-compressor[pt]
# Install 2.X API + Framework extension API + TensorFlow dependency
pip install neural-compressor[tf]Note: Further installation methods can be found under Installation Guide. check out our FAQ for more details.
Setting up the environment:
pip install "neural-compressor>=2.3" "transformers>=4.34.0" torch torchvisionAfter successfully installing these packages, try your first quantization program.
Following example code demonstrates Weight-Only Quantization on LLMs, it supports Intel CPU, Intel Gaudi2 AI Accelerator, Nvidia GPU, best device will be selected automatically.
To try on Intel Gaudi2, docker image with Gaudi Software Stack is recommended, please refer to following script for environment setup. More details can be found in Gaudi Guide.
# Run a container with an interactive shell
docker run -it --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --net=host --ipc=host vault.habana.ai/gaudi-docker/1.14.0/ubuntu22.04/habanalabs/pytorch-installer-2.1.1:latest
# Install the optimum-habana
pip install --upgrade-strategy eager optimum[habana]
# Install INC/auto_round
pip install neural-compressor auto_roundRun the example:
from transformers import AutoModel, AutoTokenizer
from neural_compressor.config import PostTrainingQuantConfig
from neural_compressor.quantization import fit
from neural_compressor.adaptor.torch_utils.auto_round import get_dataloader
model_name = "EleutherAI/gpt-neo-125m"
float_model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
dataloader = get_dataloader(tokenizer, seqlen=2048)
woq_conf = PostTrainingQuantConfig(
approach="weight_only",
op_type_dict={
".*": { # match all ops
"weight": {
"dtype": "int",
"bits": 4,
"algorithm": "AUTOROUND",
},
}
},
)
quantized_model = fit(model=float_model, conf=woq_conf, calib_dataloader=dataloader)Note:
To try INT4 model inference, please directly use Intel Extension for Transformers, which leverages Intel Neural Compressor for model quantization.
from torchvision import models
from neural_compressor.config import PostTrainingQuantConfig
from neural_compressor.data import DataLoader, Datasets
from neural_compressor.quantization import fit
float_model = models.resnet18()
dataset = Datasets("pytorch")["dummy"](shape=(1, 3, 224, 224))
calib_dataloader = DataLoader(framework="pytorch", dataset=dataset)
static_quant_conf = PostTrainingQuantConfig()
quantized_model = fit(model=float_model, conf=static_quant_conf, calib_dataloader=calib_dataloader)| Overview | |||||||
|---|---|---|---|---|---|---|---|
| Architecture | Workflow | APIs | LLMs Recipes | Examples | |||
| PyTorch Extension APIs | |||||||
| Overview | Static Quantization | Dynamic Quantization | Smooth Quantization | ||||
| Weight-Only Quantization | MX Quantization | Mixed Precision | |||||
| Tensorflow Extension APIs | |||||||
| Overview | Static Quantization | Smooth Quantization | |||||
| Other Modules | |||||||
| Auto Tune | Benchmark | ||||||
Note: From 3.0 release, we recommend to use 3.X API. Compression techniques during training such as QAT, Pruning, Distillation only available in 2.X API currently.
- Blog by Intel: Neural Compressor: Boosting AI Model Efficiency (June 2024)
- Blog by Intel: Optimization of Intel AI Solutions for Alibaba Cloud’s Qwen2 Large Language Models (June 2024)
- Blog by Intel: Accelerate Meta* Llama 3 with Intel AI Solutions (Apr 2024)
- EMNLP'2023 (Under Review): TEQ: Trainable Equivalent Transformation for Quantization of LLMs (Sep 2023)
- arXiv: Efficient Post-training Quantization with FP8 Formats (Sep 2023)
- arXiv: Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs (Sep 2023)
Note: View Full Publication List.
- GitHub Issues: mainly for bug reports, new feature requests, question asking, etc.
- Email: welcome to raise any interesting research ideas on model compression techniques by email for collaborations.
- Discord Channel: join the discord channel for more flexible technical discussion.
- WeChat group: scan the QA code to join the technical discussion.
