qwen2-5-3b-instruct-Q4_K_M-4Bit

Accuracy vs. Size Tradeoff

Every variant is benchmarked against the FP16 baseline. Choose the right balance of accuracy and footprint for your workflow.

Usage Code Snippets

Which variant for your hardware?

Choose the right quantization path for your device and performance target.

About This Model

qwen2.5-3b-instruct-q4_k_m.gguf is a Q4_K_M quantized GGUF version of Alibaba's Qwen2.5-3B-Instruct language model. Built on the Qwen2 architecture with 3.09 billion parameters, this model is optimized for instruction-following, reasoning, coding assistance, content generation, summarization, and multilingual conversations. The Q4_K_M quantization is widely considered the best quality-to-size tradeoff among GGUF formats, preserving most of the original model's capabilities while significantly reducing memory requirements. The model supports up to 32K native context length and multilingual understanding across more than 29 languages. It is compatible with llama.cpp, Ollama, LM Studio, Jan, GPT4All, Open WebUI, and LocalAI, making it a popular choice for local deployment on consumer hardware.