Foundation Model Support Matrix¶
Notes:
- You can click a link in the Model Type column to go to the best practices page recommended by msModelSlim, which describes the quantization commands and configuration in detail.
- √ indicates that the quantization strategy has been officially verified by msModelSlim. It features complete functions and stable performance and is recommended for use.
- - indicates that the quantization strategy has not been officially verified by msModelSlim. You can configure and try the strategy as needed, but the quantization effect and function stability cannot be guaranteed.
- The "model name-quantization mode (such as w8a8s)" combination marked with "one-click quantization" can be used to execute the following one-click quantization command line to perform model quantization after installation.
- Because newer versions of the Qwen series with more powerful capabilities have been released, the Qwen1.5-14B/32B/72B models have exceeded the maintenance period. Old models in this series will be sunset, and maintenance support for the quantization modes of their live-network versions will be discontinued.
- Qwen3.5 / Qwen3.6 have native multimodal (Vision) capabilities. For calibration modality descriptions, see the Qwen3.5 Quantization Guide - Calibration Modality Support.
msmodelslim quant --model_path ${MODEL_PATH} --save_path ${SAVE_PATH} --device npu --model_type ${MODEL_TYPE} --quant_type ${QUANT_TYPE} --trust_remote_code true
- For best practices not marked with "one-click quantization", read the best practices page of the corresponding model type and execute the command in the subdirectory of the corresponding model type in the example directory, such as DeepSeek or Qwen3.
Quantization Mode Naming Convention¶
The format of the quantization mode name is W{weight_bit}A{activation_bit}[C{cache_bit}][S], where
{weight_bit}indicates weight quantization bits (such as 8, 4, or 16).{activation_bit}indicates activation quantization bits (such as 8 or 16).{cache_bit}(optional) indicates KV cache quantization bits (such as 8).S(optional) indicates sparse quantization.
Supported LLMs¶
Note: This table contains a large amount of data. If it is not fully visible, use the scroll bar at the bottom or hold the mouse wheel to scroll horizontally.
| Model Type | Model Name | Dependency Library | w8a161 | w8a8 | w4a8 | w4a16 | w8a8c82 | w4a8c82 | w8a8s (Sparse Quantization)3 | w16a16s (Floating-point Sparse Quantization)3 | w4a4 | w4a4c82 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek series | DeepSeek-V2-16B | - | √ | √ | - | - | - | - | - | - | - | - |
| DeepSeek-V2-236B | - | √ | √ | - | - | - | - | - | - | - | - | |
| DeepSeek-Coder-33B | - | √ | √ | - | - | √ | - | - | - | - | - | |
| DeepSeek-V3 | transformers==4.48.2 | - | √ | - | - | - | - | - | - | - | - | |
| DeepSeek-V3.1 | transformers==4.48.2 | - | √ | √ | - | √ | √ (one-click quantization) | - | - | - | - | |
| DeepSeek-V3.1-Terminus | transformers==4.48.2 | - | - | - | - | √ (one-click quantization) | - | - | - | - | √ (one-click quantization) | |
| DeepSeek-V3.2-Exp | transformers==4.48.2 | - | √ (one-click quantization) | √ (one-click quantization) | - | - | - | - | - | - | - | |
| DeepSeek-V3.2 | transformers==4.48.2 | - | √ (one-click quantization) | - | - | - | - | - | - | - | - | |
| DeepSeek-V4-Flash | transformers==4.48.2 | - | √ (one-click quantization) | - | - | - | - | - | - | - | √ (one-click quantization) | |
| DeepSeek-V4-Flash-0731 | transformers==4.48.2 | - | - | - | - | - | - | - | - | - | √ (one-click quantization)5 | |
| DeepSeek-V4-Pro | transformers==4.48.2 | - | √ (one-click quantization) | √ (one-click quantization) | - | - | - | - | - | - | √ (one-click quantization) | |
| DeepSeek-V4-Pro-0813 | transformers==4.48.2 | - | - | - | - | - | - | - | - | - | √ (one-click quantization)5 | |
| DeepSeek-R1 | transformers==4.48.2 | - | √ | √ | - | √ | - | - | - | - | - | |
| DeepSeek-R1-0528 | transformers==4.48.2 | - | √ | √ (one-click quantization) | - | √ | √ (one-click quantization) | - | - | - | - | |
| DeepSeek-R1-Distill series | DeepSeek-R1-Distill-Llama-8B | - | - | √ | - | - | - | - | √ | - | - | - |
| DeepSeek-R1-Distill-Llama-70B | - | - | √ | - | - | - | - | - | - | - | - | |
| DeepSeek-R1-Distill-Qwen-1.5B | - | - | √ | - | - | - | - | √ | - | - | - | |
| DeepSeek-R1-Distill-Qwen-7B | - | - | √ | - | - | - | - | √ | - | - | - | |
| DeepSeek-R1-Distill-Qwen-14B | - | - | √ | - | - | - | - | √ | - | - | - | |
| DeepSeek-R1-Distill-Qwen-32B | - | - | √ | - | - | - | - | √ | - | - | - | |
| Qwen3 series | Qwen3-8B | transformers==4.51.0 | - | - | - | - | - | - | √ (one-click quantization) | - | - | - |
| Qwen3-14B | transformers==4.51.0 | - | √ (one-click quantization, supported only by MindIE)4 | - | - | - | - | √ (one-click quantization) | - | - | - | |
| Qwen3-32B | transformers==4.51.0 | - | √ (one-click quantization, supported only by MindIE)4 | - | - | √ (one-click quantization) | - | √ (one-click quantization) | √ (one-click quantization) | √ | - | |
| Qwen3-MOE series | Qwen3-30B-A3B | transformers==4.57.6 | - | √ (one-click quantization) | √ (one-click quantization) | - | - | - | - | - | - | - |
| Qwen3-235B-A22B | transformers==4.57.6 | - | √ | √ (one-click quantization) | - | - | - | - | - | - | - | |
| Qwen3-Coder-480B-A35B | transformers==4.57.6 | - | - | √ (one-click quantization) | - | - | - | - | - | - | - | |
| Qwen3.5 series | Qwen3.5-397B-A17B | transformers==5.2.0 | - | √ (one-click quantization) | √ (one-click quantization) | - | - | - | - | - | - | - |
| Qwen3.5-122B-A10B | transformers==5.2.0 | - | √ (one-click quantization) | - | - | - | - | - | - | - | - | |
| Qwen3.5-35B-A3B | transformers==5.2.0 | - | √ (one-click quantization) | - | - | - | - | - | - | - | - | |
| Qwen3.5-27B | transformers==5.2.0 | - | √ (one-click quantization) | - | - | - | - | - | - | - | - | |
| Qwen3.5-4B | transformers==5.2.0 | - | √ (one-click quantization) | - | - | - | - | - | - | - | - | |
| Qwen3.6-27B | transformers==5.2.0 | - | √ (one-click quantization) | - | - | √ (one-click quantization) | - | - | - | - | - | |
| Qwen3.6-35B-A3B | transformers==5.2.0 | - | - | - | - | √ (one-click quantization)4 | - | - | - | - | - | |
| Qwen3-Next series | Qwen3-Next-80B-A3B-Instruct | transformers>=4.57.0 | - | √ (one-click quantization, supported only by vLLM Ascend) | - | - | - | - | - | - | - | - |
| Qwen2.5 series | Qwen2.5-7B-Instruct | - | - | √ (one-click quantization) | - | - | - | - | √ | - | - | - |
| Qwen2.5-14B-Instruct | - | - | √ | - | - | - | - | √ | - | - | - | |
| Qwen2.5-32B-Instruct | - | - | √ (one-click quantization) | - | - | - | - | - | - | - | - | |
| Qwen2.5-72B-Instruct | - | - | - | - | √ | √ (one-click quantization) | - | - | - | - | - | |
| Qwen2.5-Coder-7B-Instruct | - | - | - | - | - | - | - | √ (one-click quantization) | - | - | - | |
| Qwen2 series | Qwen2-7B | - | - | √ | - | - | - | - | √ | - | - | - |
| Qwen2-72B | - | √ | √ | - | - | √ | - | √ | - | - | - | |
| Qwen series | Qwen-7B | - | - | √ | - | - | - | - | - | - | - | - |
| Qwen-14B | - | - | √ | - | - | - | - | - | - | - | - | |
| Qwen-72B | - | √ | - | - | - | - | - | - | - | - | - | |
| Qwen1.5-14B | - | - | √ | - | - | - | - | √ | - | - | - | |
| Qwen1.5-32B | - | - | √ | - | - | - | - | - | - | - | - | |
| Qwen1.5-72B | - | √ | - | - | - | - | - | - | - | - | - | |
| Qwen1.5-110B | - | √ | - | - | - | - | - | - | - | - | - | |
| QwQ series | QwQ-32B | - | - | √ (one-click quantization) | - | - | - | - | √ (one-click quantization) | - | - | - |
| GLM5-MOE series | GLM-5 | transformers==5.4.0 | - | √ | √ | - | - | - | - | - | - | - |
| GLM-5.1 | transformers==5.4.0 | - | √ (one-click quantization) | √ (one-click quantization) | - | √ (one-click quantization) | √ (one-click quantization) | - | - | - | √ (one-click quantization) | |
| GLM-5.2 | transformers==5.12.0 | - | √ (one-click quantization) | - | - | √ (one-click quantization) | - | - | - | √ (one-click quantization) | √ (one-click quantization) | |
| GLM-5.3 | transformers==5.15.0 | - | √ (one-click quantization) | - | - | √ (one-click quantization) | - | - | - | - | √ (one-click quantization) | |
| GLM-5.3-Flash | transformers==5.16.1 | - | √ (one-click quantization) | - | - | √ (one-click quantization) | - | - | - | - | - | |
| GLM series | GLM-4-9B | - | - | √ | - | - | √ | - | √ | - | - | - |
| GLM4-MOE series | GLM-4.7 | transformers==4.57.3 | - | √ (one-click quantization, supported only by vLLM Ascend) | - | - | - | - | - | - | - | - |
| GLM4-MOE-LITE series | GLM-4.7-Flash | transformers==5.3.0 | - | √ (one-click quantization, supported only by sglang) | - | - | - | - | - | - | - | - |
| Hy3 series | Hy3 | transformers==5.6.0 | - | √ (one-click quantization) | - | - | - | - | - | - | - | - |
| HunYuan series | Hunyuan-A52B-Instruct | transformers>=4.48.2 | - | √ | - | - | - | - | - | - | - | - |
| MiniMax-M2 series | MiniMax-M2.7 | - | - | √ (one-click quantization) | - | - | √ (one-click quantization) | - | - | - | - | - |
| MiMo series | MiMo-V2-Flash | transformers==4.57.6 | - | √ (one-click quantization) | - | - | - | - | - | - | - | - |
| LongCat-Flash series | LongCat-Flash-Chat | transformers==4.55.0 | - | - | - | - | - | - | - | - | √ (one-click quantization) | - |
| stepfun series | Step-3.5-Flash | - | - | √ (one-click quantization) | - | - | - | - | - | - | - | - |
| InternLM series | InternLM2-20B | - | √ | √ | - | - | √ | - | - | - | - | - |
| LLaMA series | LLaMA-33B | - | - | - | - | - | - | - | √ | - | - | - |
| LLaMA-65B | - | √ | - | - | - | - | - | - | - | - | - | |
| LLaMA2-13B | - | - | √ | - | - | - | - | √ | - | - | - | |
| LLaMA2-7B | - | - | √ | - | - | - | - | √ | - | - | - | |
| LLaMA2-70B | - | √ | √ | - | - | - | - | - | - | - | - | |
| LLaMA3-70B | - | √ | - | - | - | - | - | - | - | - | - | |
| LLaMA3.1-8B | - | - | √ | - | - | - | - | - | - | - | - | |
| LLaMA3.1-70B | - | - | √ | - | - | √ | - | √ | - | - | - |
Notes:
- 1 The w8a16 quantization mode is supported only by MindIE.
- 2 Both KVCache and FA3 quantization are categorized under c8, as both quantize the K and V caches in the LLM. Only MindIE supports c8 quantization modes, including w8a8c8, w4a8c8, and w4a4c8.
- 3 For better results, use the decompression feature of Atlas inference products after compression. Only MindIE supports sparse quantization modes, including w8a8s and w16a16s.
- 4 Only MindIE supports best practices that use the PDMIX quantization scheme.
- 5 For DeepSeek-V4-Flash-0731 and DeepSeek-V4-Pro-0813, the w4a4c8 column is a W4A4C8 mixed-precision scheme: MXFP4 for routed experts, MXFP8 for attention and shared experts, and the official FP8 reused for KV. Use
--quant_type w4a4c8during quantization. The verified scenarios are vLLM-Ascend + Ascend_950 and vLLM-Ascend + Atlas_350. For details, see the DeepSeek quantization guide.
Supported MLLMs¶
Calibration modality: For the calibration modalities supported by each model (text-only / image-text / audio-video, and so on) and the homogeneity constraints, refer to the corresponding example README. For example:
- Qwen3-Omni — Calibration Modality Support: Text-only and any combination of image/audio/video; the same task must be homogeneous.
- Qwen3.5 — Calibration Modality Support: Text-only and text+image (Qwen3.5 is listed in the LLM table, but its quantization follows the VLM path).
| Model Type | Model Name | Dependency Library | w8a8 | w8a8c8/w8a8f8 | w8a8s (Sparse Quantization)1 | w4a8 | w4a8c8 | w4a4c8 | w4a4f4 |
|---|---|---|---|---|---|---|---|---|---|
| Qwen3-VL series | Qwen3-VL-4B-Instruct | transformers==4.57.1 | √ | - | - | - | - | - | - |
| Qwen3-VL-8B-Instruct | transformers==4.57.1 | - | - | √ | - | - | - | - | |
| Qwen3-VL-32B-Instruct | transformers==4.57.1 | √ | - | - | - | - | - | - | |
| Qwen3-VL-Embedding&Reranker series | Qwen3-VL-Embedding-2B | transformers==4.57.1 | √ | - | - | - | - | - | - |
| Qwen3-VL-Embedding-8B | transformers==4.57.1 | √ | - | - | - | - | - | - | |
| Qwen3-VL-Reranker-2B | transformers==4.57.1 | √ | - | - | - | - | - | - | |
| Qwen3-VL-Reranker-8B | transformers==4.57.1 | √ | - | - | - | - | - | - | |
| Qwen3-VL-MoE series | Qwen3-VL-235B-A22B | transformers==4.57.1, flax | √ (one-click quantization) | - | - | - | - | - | - |
| Qwen3-Omni series | Qwen3-Omni-30B-A3B-Thinking | transformers==4.57.3 | √ | - | - | - | - | - | - |
| Qwen3-Omni-30B-A3B-Instruct | transformers==4.57.3 | √ | - | - | - | - | - | - | |
| Gemma4 series | gemma-4-31B-it | transformers==5.5.3 | √ (one-click quantization, supported only by vLLM Ascend; see also the README: MXFP8/MXFP4 hybrid, one-click quantization w4a4) | - | - | - | - | - | - |
| Qwen2.5-VL series | Qwen2.5-VL-7B | transformers==4.49.0, qwen_vl_utils | √ | - | - | - | - | - | - |
| Qwen2.5-VL-72B | transformers==4.49.0, qwen_vl_utils | √ | - | - | - | - | - | - | |
| Qwen2.5-Omni series | Qwen2.5-Omni-7B | transformers==4.57.3 | √ | - | - | - | - | - | - |
| Qwen2-VL series | Qwen2-VL-7B | transformers==4.46.0, qwen_vl_utils | √ | - | - | - | - | - | - |
| Qwen2-VL-72B | transformers==4.46.0, qwen_vl_utils | √ | - | - | - | - | - | - | |
| Qwen-VL series | Qwen-VL | transformers-stream-generator | √ | - | - | - | - | - | - |
| InternVL2 series | InternVL2-8B | transformers==4.46.0, timm, fastchat | √ | - | - | - | - | - | - |
| InternVL2-40B | transformers==4.46.0, timm, fastchat | √ | - | - | - | - | - | - | |
| LLaVA series | LLaVA-1.5-7B | transformers==4.37.2 | √ | - | - | - | - | - | - |
| GLM-4.1V series | GLM-4.1V-9B-Thinking | transformers==4.53.0 | - | - | √ | - | - | - | - |
| GLM-4.6V | GLM-4.6V | transformers==5.0.0rc0 | √ | - | - | - | - | - | - |
| Kimi-K2.5 | Kimi-K2.5 | transformers==4.57.6, compressed-tensors==0.13.0 | - | - | - | √ | - | - | - |
| Kimi-K2.6 | Kimi-K2.6 | transformers==4.57.6, compressed-tensors==0.13.0 | - | - | - | √ | - | √ | - |
| Kimi-K3 | Kimi-K3 | transformers==4.57.6, compressed-tensors==0.13.0, fla-core | - | - | - | √ (one-click quantization)3 | √ (one-click quantization) | - | - |
| InternVL3_5 series | InternVL3_5-38B | transformers==4.57.6 | √ | - | - | - | - | - | - |
| InternVL3_5-MoE series | InternVL3_5-241B-A28B | transformers==4.57.6 | √ | - | - | - | - | - | - |
| Gemma4-MoE series | gemma-4-26B-A4B-it | transformers==5.5.3 | √ (see also the README: MXFP8/MXFP4 hybrid, one-click quantization w4a4) | - | - | - | - | - | - |
| MiniMax-M3 series | MiniMax-M3-preview | transformers==5.12.0 | √ (one-click quantization) | - | - | - | - | - | - |
| Step-3.7-Flash series | Step-3.7-Flash | transformers>=4.57.1,<5.0.0 | √ (one-click quantization, supported only by vLLM Ascend) | - | - | - | - | - | |
| Multimodal Generative Models | SD3-Medium | diffusers | √ | - | - | - | - | - | - |
| Open-Sora-Plan v1.2 | huggingface_hub==0.25.2 | √ | - | - | - | - | - | - | |
| FLUX.1-dev | - | √ | √ (one-click quantization) | - | - | - | - | - | |
| HunyuanVideo | - | √ | √ (one-click quantization) | - | - | - | - | - | |
| Wan2.1 | - | √ (one-click quantization) | - | - | - | - | - | - | |
| Wan2.2 | - | - | √ (one-click quantization) | - | - | - | - | √ (one-click quantization) | |
| Qwen-Image-Edit | - | - | √ (one-click quantization) | - | - | - | - | √ (one-click quantization) |
Notes:
- 1 For better results, use the decompression feature of Atlas inference products after compression. Only MindIE supports sparse quantization modes.
- 2 FLUX.1-dev, HunyuanVideo, Wan2.2, and Qwen-Image-Edit-2509 support MXFP quantization.
- 3 The w4a8 of Kimi-K3 contains two practices, INT and MXFP (which share
quant_type=w4a8). MXFP requires additionally specifying--tags vLLM-Ascend Ascend_950. For details, see the Kimi-K3 quantization case. - 4 The w8a8c8 of Qwen3.6-35B-A3B is a mixed-precision scheme: MXFP8 for linear layers (W8A8), MXFP4 for routed experts (W4A4), and FA3 quantization for attention (Q/K MXFP8 per-block dynamic + V MXFP8 per-channel static), with KV Cache quantization included. It is currently adapted only for vLLM-Ascend + Ascend_950, and
--tags vLLM-Ascend Ascend_950must be specified during quantization. For details, see the Qwen3.5 quantization guide.