4-bit or 8-bit AI: how quantization helps a model fit on your computer

Smaller numbers, not a shorter summary

A quantized version can make a local AI model fit on a computer that cannot hold its higher-precision version. Quantization stores the model’s weights, the numbers learned during training, using fewer bits. It saves space by representing those values less precisely.

The goal, as Hugging Face explains, is to preserve useful accuracy while reducing memory requirements. This is not the same as removing a set of facts or summarizing a document. It changes the numerical representation used to produce answers.

Labels such as 4-bit and 8-bit describe that representation, not an intelligence score. Comparing different models solely by their bit depth tells you very little about which will answer your questions better.

Estimate the weights, then allow for the rest

Consider a simplified model with eight billion weights. At 16 bits each, those weights need about 16 decimal GB. At 8 bits they need about 8 GB, and at 4 bits about 4 GB. The calculation is weight count × bits ÷ 8. These are weight-storage estimates, not complete system requirements.

Actual formats include additional information and may leave some components at higher precision. Bitsandbytes documentation illustrates why a quantized model need not store every component in the same way. A 4-bit download therefore will not necessarily be exactly one quarter of a 16-bit download.

The application and the conversation need memory too. Ollama’s context documentation warns that increasing context length raises memory use. Context is the material available for the current request, rather than permanent recall. Our local AI versus cloud guide explains which resources your own machine must provide.

Compare versions on your actual work

Use the same base model and context length when comparing two quantizations. Start with a version your application supports on your hardware. A smaller model file may run faster or fit entirely on the GPU, but the software and quantization method affect the result. A fixed speed increase is not guaranteed.

Check quality as well as memory

Quantization is also different from fine-tuning. One changes how weights are represented, while the other adapts behavior through additional training.

• Choose an identified publisher and a format supported by your application. Check the base model, bit depth and estimated memory requirement.
• Run repeatable tasks with answers you can verify, such as extracting dates or summarizing a document you know. Fluent writing alone is not a quality test.
• Monitor memory with a conversation as long as your normal workload. In Ollama, ollama ps shows the CPU/GPU split and allocated context.
• If memory runs out, reduce context or try a smaller model. If answer quality deteriorates and memory allows it, compare a higher-precision version.