Quantization: how to fit a big model on your graphics card

What you'll see in this lesson

1. Quantizing: storing the numbers with less detail

   Quantizing means rounding the model's weights so they take less space: from 16 bits per number to 8, to 4 or even less.

2. The basic math: parameters × bits ÷ 8

   Multiply the parameters by the bits for each one and divide by 8: the result is the bytes the weights take up.

3. GGUF and Q4_K_M: how to read the file name

   GGUF is the format of llama.cpp, Ollama and LM Studio; the suffix in the name tells you how many bits it uses and how.

4. Dynamic quantizations: more bits where it matters

   Unsloth doesn't round everything the same: it keeps sensitive layers at more bits and squeezes the tough ones, to lose less quality.

5. FP8, NVFP4 and MXFP4: graphics card formats

   These are formats the graphics card computes natively; vLLM and SGLang use them, and many models now ship this way.

6. MLX: the Mac's own format

   On a Mac with an M chip, MLX makes use of unified memory and is often faster than GGUF; LM Studio uses it directly.

7. The KV cache: what the conversation uses

   Besides the weights, the model stores two vectors for every token it reads; with long contexts that's several gigs.

8. MoE: total for memory, active for speed

   In a mixture-of-experts model everything has to fit, but each token only reads a part: that's why it's much faster.

9. Summary

   Do the math before downloading: weights by bits, plus the conversation cache, plus a margin.