Machine Learning

The Complete Guide to AI Model Quantization in 2026

Ethan Rourke ·
The Complete Guide to AI Model Quantization in 2026

The Complete Guide to AI Model Quantization in 2026

Running a 72 billion parameter model requires 144GB of GPU memory in full precision. Most practitioners do not have that hardware. LLM quantization compresses model weights to lower precision (8-bit, 4-bit, or even 2-bit), reducing memory requirements by 2-8x with minimal quality loss. Understanding quantization is essential for anyone deploying open-source models locally or optimizing inference costs.

This LLM quantization guide covers every major method in 2026, with benchmarks showing the real quality trade-offs at each precision level.

What Quantization Does

  • Reduces memory usage. A 72B model in BF16 (16-bit) needs ~144GB. In 4-bit quantization, it needs ~40GB and fits on a single A6000 or two consumer GPUs.
  • Increases inference speed. Lower precision means less data to move from memory to compute units. Memory bandwidth is the primary bottleneck for LLM inference, so quantization directly improves throughput.
  • Reduces costs. Smaller memory footprint means fewer GPUs needed per model instance, which directly reduces hardware and cloud costs.
  • Introduces quality loss. Quantization is lossy compression. The model’s weights are approximated at lower precision, which introduces errors. The art of quantization is minimizing these errors.

Quantization Methods Compared

GPTQ (GPU-Optimized Post-Training Quantization)

GPTQ quantizes weights using calibration data to minimize output error. Produces 4-bit and 3-bit models optimized for GPU inference. Best for: NVIDIA GPU deployment via vLLM, TensorRT-LLM, or ExLlama. Quality: 4-bit GPTQ typically loses 0.5-1.5% on standard benchmarks compared to full precision.

AWQ (Activation-Aware Weight Quantization)

AWQ identifies important weights (based on activation patterns) and preserves them at higher precision while aggressively quantizing less important weights. Best for: When you need slightly better quality than GPTQ at the same bit width. Quality: Typically 0.3-1.0% better than GPTQ at the same precision.

GGUF (GPT-Generated Unified Format)

GGUF is the standard format for CPU and Apple Silicon inference via llama.cpp and Ollama. Supports a wide range of quantization levels (Q2_K through Q8_0). Best for: MacBooks, CPU inference, and consumer hardware without discrete GPUs. Quality: Comparable to GPTQ at the same bit width, optimized for CPU execution.

FP8 (8-bit Floating Point)

FP8 uses 8-bit floating point rather than integer quantization. It preserves more dynamic range than INT8 and produces near-lossless results. Best for: When you need maximum quality with moderate memory savings (2x reduction vs BF16). Quality: Typically less than 0.3% loss. For most applications, FP8 is functionally lossless.

“4-bit quantization is the sweet spot for most deployments. The quality loss is detectable on benchmarks but invisible in production applications for 95% of use cases.” — ML infrastructure engineer.

Benchmark Results: Qwen 3.5 72B at Different Precisions

BF16 (baseline, 144GB): MMLU 86.1, HumanEval 82.4%.

FP8 (72GB): MMLU 85.9, HumanEval 82.1%. Near-lossless.

4-bit GPTQ (40GB): MMLU 85.2, HumanEval 80.8%. Minor quality loss.

4-bit AWQ (40GB): MMLU 85.4, HumanEval 81.0%. Slightly better than GPTQ.

4-bit GGUF Q4_K_M (40GB): MMLU 85.1, HumanEval 80.5%. Good for CPU/Mac deployment.

3-bit GPTQ (30GB): MMLU 83.4, HumanEval 77.2%. Noticeable quality loss.

2-bit (20GB): MMLU 78.6, HumanEval 68.1%. Significant degradation. Only use when hardware constraints allow no other option.

Choosing the Right Quantization

  1. Maximum quality, 2x memory savings: FP8. Use this when you have enough VRAM for half-precision and want zero quality compromise.
  2. Best quality-to-size ratio: 4-bit AWQ or GPTQ. The standard choice for most deployments. Fits large models on single GPUs.
  3. Mac or CPU inference: GGUF Q4_K_M or Q5_K_M. Optimized for llama.cpp and Ollama.
  4. Maximum compression: 3-bit. Acceptable for prototyping or applications where speed matters more than quality.
  5. Do not use 2-bit for any application where output quality matters.

Practical Tips

Always benchmark on your specific task. Aggregate benchmarks hide task-specific quality differences. A model that loses only 1 point on MMLU might lose 5 points on your domain-specific evaluation.

Use calibration data from your domain. GPTQ and AWQ use calibration data during quantization. Using data similar to your production inputs produces better quality than generic calibration sets.

Consider quantization-aware training. If you are fine-tuning a model and plan to deploy it quantized, train with quantization-aware techniques (QLoRA). This produces better quantized models than post-training quantization of a full-precision fine-tuned model.

Quantization is a fundamental skill for anyone deploying open-source models. Master it, and you can run models that would otherwise require enterprise-grade hardware on equipment that fits on your desk.