TL;DR: Quantization in AI is a way of shrinking a model by storing its numbers at lower precision, for example turning 16-bit or 32-bit weights into 8-bit or 4-bit integers. A quantized model needs much less memory, often runs faster and costs less to serve, which is why it’s the main trick that lets large language models run on laptops and phones. The trade-off is a possible loss of accuracy, which is usually small at 8 bits and more noticeable at 4 bits or below.
If you’ve downloaded a local AI model and seen labels like “Q4,” “INT8,” “GGUF” or “AWQ,” you’ve already met quantization. So what is quantization in AI, exactly? This guide explains it in plain English: what it does to a model, how much memory it saves, the main types and methods, and when the accuracy cost is worth it.
What is quantization in AI?
A neural network stores what it learned as weights, which are billions of numbers in a modern large language model. During training these are normally kept as floating-point numbers, such as 32-bit (FP32) or 16-bit (FP16 or BF16) values. IBM describes quantization as the process of converting those high-precision values into lower-precision formats to reduce memory and computation needs.
The Hugging Face quantization guide explains the core idea: an 8-bit integer (INT8) can represent only 256 distinct values, so quantization maps the much wider range of original values onto those 256 levels using a scale factor (and sometimes a zero point). At 4 bits, there are only 16 levels. Each weight becomes a rougher approximation of the original, but the model as a whole often still behaves almost the same.
A simple analogy: it’s like saving a photo as a smaller JPEG. You throw away detail most people won’t notice, and the file becomes far easier to store and share.
How much memory does quantization save?
The savings are easy to estimate, because memory for weights is roughly the number of parameters multiplied by the bytes per parameter. Here’s the math for a model with 7 billion parameters (weights only; real usage is higher because of activations and the cache used during inference):
| Precision | Bits per weight | Approx. weight memory (7B model) | Typical accuracy impact |
|---|---|---|---|
| FP32 | 32 | ~28 GB | Baseline |
| FP16 / BF16 | 16 | ~14 GB | Standard for serving; near-identical |
| INT8 / FP8 | 8 | ~7 GB | Usually small |
| INT4 | 4 | ~3.5 GB | Small to noticeable, depends on method |
| 2-3 bit | 2-3 | ~1.75-2.6 GB | Often significant |
That’s why a model that needs a data-center GPU at 16-bit precision can fit on a consumer graphics card or a laptop with unified memory once it’s quantized to 4 bits.
Why quantization matters
- Less memory: smaller models fit on cheaper hardware, including phones and laptops.
- Faster responses: generating text is often limited by how fast weights can be moved from memory, so smaller weights can mean more tokens per second. Hugging Face’s LLM optimization guide notes that speed gains depend on the hardware and the method.
- Lower cost: fewer or smaller GPUs per request reduce the cost of serving a model at scale.
- Lower energy use: integer math and less data movement generally use less power.
- Privacy and offline use: running a quantized model on your own device means prompts don’t have to leave it.
Types of quantization
Post-training quantization (PTQ)
PTQ takes a model that’s already trained and converts its weights afterward. It’s fast and doesn’t require retraining, which is why most downloadable quantized LLMs are made this way. Some PTQ methods use a small “calibration” dataset to choose scale factors that minimize error.
Quantization-aware training (QAT)
QAT simulates low precision during training or fine-tuning, so the model learns to cope with rounding errors. It usually preserves accuracy better at very low bit widths, but it costs much more compute.
Weight-only vs weight-and-activation
Many popular LLM formats quantize only the weights and keep activations (the intermediate values computed while the model runs) at 16-bit precision. Quantizing activations too can bring bigger speedups on hardware that supports fast low-precision math, but it’s harder to do without hurting quality.
Common quantization methods and formats
- GPTQ: a post-training method that quantizes weights layer by layer while compensating for errors.
- AWQ (Activation-aware Weight Quantization): protects the small share of weights that matter most, based on activation statistics.
- bitsandbytes: a library integrated with Hugging Face Transformers for loading models in 8-bit or 4-bit.
- GGUF: the file format used by llama.cpp and many local AI apps, with variants such as Q4 and Q8 that indicate bit width.
- FP8 and FP4: low-precision floating-point formats supported by newer data-center chips.
Does quantization hurt accuracy?
Sometimes, but less than you might expect. Research presented at ACL 2025, titled “Give Me BF16 or Give Me Death”?, evaluated accuracy-performance trade-offs across quantization formats and reported that 8-bit formats lost very little accuracy (roughly 1-3% at most for tuned INT8), while 4-bit weight-only quantization was competitive but more sensitive to the method used. Results vary by model, task and technique, so the safe approach is to test a quantized model on the work you actually need it for.
Rules of thumb that most practitioners follow:
- 8-bit is a low-risk default when you need to save memory.
- 4-bit is the sweet spot for running models locally, especially with GPTQ, AWQ or good GGUF variants.
- Below 4 bits, expect visible quality drops unless the model was trained for it.
- A larger model at 4 bits often beats a smaller model at 16 bits within the same memory budget.
Quantization vs other ways to shrink a model
| Technique | What it does |
|---|---|
| Quantization | Keeps every weight but stores it with fewer bits |
| Pruning | Removes weights or whole parts of the network that contribute little |
| Distillation | Trains a smaller “student” model to imitate a larger “teacher” |
These are often combined. A distilled small model can still be quantized to run on a phone.
The bottom line
Quantization is the reason powerful AI models can run on everyday devices. By storing numbers at lower precision, it cuts memory and cost dramatically, usually at a modest accuracy price. To understand the pieces it acts on, read our guides to transformer models and tokens in AI.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
