# Quantization

> Quantization: lowering the precision of the numbers that make up an AI model so it takes less memory and runs on more modest hardware.

- **Category**: AI
- **URL**: https://www.romaindelfosse.fr/en/glossary/quantization/

---

**Quantization** means lowering the precision of the numbers that make up an artificial intelligence model. The model gets lighter, fits in less memory and can run on an ordinary graphics card. It's the technique that made it possible to use large language models outside data centers.

## Technical definition

### Billions of numbers to store

A language model is a huge table of numbers: its weights. A 27-billion-parameter model contains 27 billion of them. Stored at high precision, they take up tens of gigabytes. Quantization replaces each one with an approximate value encoded on fewer bits.

### A size-versus-fidelity trade-off

Fewer bits means a smaller file and answers slightly less faithful to those of the original model. So you pick a level: eight bits to stay as close as possible to the original, four to cut the size further at the cost of a moderate loss.

### An after-the-fact operation

Quantization doesn't require retraining the model. It is applied to a model that has already been released. That's why the same model circulates in many versions, produced by its publisher or by independent contributors.

## Current challenges

### Access to open models

Without quantization, running a recent model would require data-center hardware. With it, a laptop with a graphics card is enough for mid-sized models. It's a direct lever for controlling costs and protecting confidentiality, since data never leaves the machine.

### Quality you have to check yourself

Two files bearing the same model name can behave differently depending on their quantization level and the tool that produced them. Published comparisons rarely cover the version you actually downloaded. The only reliable measure is a test on your own tasks.

### File provenance

Quantized versions are often published by third parties. Before deploying one, you need to know who produced it, from which model, and whether it was modified along the way.


## Key Figures

- Quantization reduces the precision of a model&#39;s weights, for example from 32-bit floating-point numbers to 4-bit integers, which shrinks its size and can speed up inference *(llama.cpp — quantize tool documentation — accessed November 2026)*



## Frequently Asked Questions

### Does quantization degrade a model&#39;s quality?

Yes, a little, and increasingly so the more you compress. At 8 bits, the difference from the original is usually imperceptible. At 4 bits, the trade-off remains good for most uses. Below that, errors become visible. The llama.cpp documentation points out that this loss can be measured, notably through perplexity.

### What do Q4_K_M or Q8_0 mean in a file name?

They are quantization levels. The number indicates the bits per weight: 4 for Q4, 8 for Q8. The letters that follow designate the variant of the method. The lower the number, the lighter the file and the more noticeable the loss. Some authors add their own suffixes, which automatic download tools don't always recognize.

### How do you choose the right quantization level?

Start from the available GPU memory. The file has to fit in it entirely, with headroom for the context window. Then take the most precise level that meets that constraint. A small, lightly compressed model isn't necessarily better than a large, more heavily compressed one: compare them on your own use cases.



## Resources

- [quantize tool](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md?utm_source=romaindelfosse&amp;utm_medium=glossaire&amp;utm_campaign=quantification) — llama.cpp

- [GGUF](https://huggingface.co/docs/hub/gguf?utm_source=romaindelfosse&amp;utm_medium=glossaire&amp;utm_campaign=quantification) — Hugging Face

