AI

GGUF

GGUF: a single-file format that holds an AI model and everything needed to load it, designed for running models locally.

Boîte cylindrique en métal brossé au couvercle de verre entrouvert, remplie d’un amas dense de minuscules cubes de verre rosés et lumineux, posée sur un sol sombre réfléchissant sous un faisceau de lumière
GGUF — Image générée par IA (Qwen-Image 2.1, ComfyUI)

GGUF is a file format designed to distribute and run artificial intelligence models locally. A single file contains the model and everything needed to load it. It's the de facto format for language models you run yourself, outside online services.

Technical definition

One file, all included

Its specification sets three goals: single-file deployment, with no dependency on side files; an extensible format, to which information can be added without breaking existing models; and fast loading. GGUF succeeded several earlier formats from the same project, including GGML.

Readable metadata

Besides the model's numbers, the file carries structured metadata as key-value pairs: the architecture, the maximum context size, the chat template, the quantization level. A tool can therefore load a model without being handed its configuration separately.

Quantization's companion

The same model is usually offered as several GGUF files, one per quantization level. You pick the one that fits in your machine's memory. The file name indicates that level with a suffix.

Current challenges

Running a model at home

GGUF has made open models easier to access: where you used to have to assemble a technical environment, a file and a tool are now enough. For an organization, this opens up uses where data never leaves the workstation or the internal server.

What the file decides for you

The embedded metadata drives the model's behavior. An incomplete or altered chat template changes how the model handles instructions or tool calls. When a local model behaves oddly, the file is as serious a suspect as the model.

A supply chain to watch

Most GGUF files in circulation don't come from the model publishers. They are produced and republished by third parties, sometimes with modifications. Checking a file's provenance before deploying it is the same hygiene as for any software dependency.

Frequently Asked Questions

What is a GGUF file used for?

To run a language model on your own machine, with tools such as llama.cpp, Ollama, LM Studio or GPT4All. The file contains the model and its settings: you download it, load it, and it works. Models in this format are most often quantized, meaning slimmed down to fit on an ordinary graphics card.

What is the difference between GGUF and safetensors?

Both store a model's numbers. The Hugging Face documentation sums up the gap: safetensors contains only the tensors, whereas GGUF adds a standardized set of metadata. Safetensors is the format of original models, used for training. GGUF is an inference format, designed for fast loading.

Where can you find models in GGUF format?

Mainly on Hugging Face, which lets you filter repositories by this format and inspect a file's contents before downloading it. Many of these files are published by independent contributors rather than by the model's publisher: check the author, the original model and the quantization level.

Resources and Documentation