LLM Quantization for Inference: Methods, Memory, and Speed

LLM Quantization for Inference reduces the numerical precision of model weights and, in some methods, activations to lower memory consumption and improve execution efficiency. Converting model weights from 16 bits to 8 bits cuts their raw storage requirements in half. Four bit weights reduce that requirement by approximately 75%, before accounting for quantization metadata and runtime overhead.

The important distinction is that a smaller model is not automatically a faster model. FP8, INT8, INT4, GPTQ, and AWQ solve different deployment problems. Choosing between them requires understanding what gets compressed, how the hardware processes it, and how much numerical accuracy the application can afford to lose.

How LLM Quantization Works for Inference

A language model stores billions of learned parameters as numerical values. During inference, those values participate in matrix operations that transform input tokens into predictions.

Models commonly use BF16 or FP16 for these calculations, with each value occupying 16 bits. Quantization represents selected values using fewer bits, reducing the amount of data that must be stored or transferred.

The process is more involved than simply removing decimal places.

For integer quantization, a scaling factor maps a range of numerical values into the available integer levels. The values are rounded into that representation, and the scale is used when reconstructing their approximate values or performing compatible quantized operations.

An INT8 representation has 256 distinct bit patterns. A signed four bit integer representation has just 16.

Fewer representable values mean less storage, but they also increase the potential difference between the original and quantized numbers.

The engineering challenge is reducing that difference where it matters most to model predictions.

Quantization Scales and Numerical Precision

A single scaling factor applied to an entire tensor is simple, but different parts of that tensor can contain very different numerical ranges.

A small number of unusually large values can force the scale to cover a wide range. This leaves fewer useful quantization levels for the remaining values.

More granular scaling addresses that problem.

Per tensor quantization uses one scale for an entire tensor.

Per channel quantization uses separate scales for individual channels.

Groupwise quantization divides values into smaller groups and assigns scales within those groups.

Smaller groups provide greater control over quantization error, but they also require additional scale information and compatible execution kernels.

The appropriate granularity depends on the model, quantization algorithm, and runtime implementation.

Post Training Quantization

Post Training Quantization, or PTQ, converts a trained model into a lower precision representation without repeating its original training process.

This makes it particularly relevant to inference deployments where an existing model already meets the application's quality requirements.

Some PTQ approaches need representative calibration data to determine scales or identify sensitive values. Others derive the required information directly from model weights.

Calibration data should reflect the tasks and input distributions that the deployed model will encounter.

For a coding assistant, code generation examples provide more relevant evidence than a calibration collection containing only general conversational text.

The objective is to minimize the effect of compression on the outputs that matter to the application.

LLM Quantization and GPU Memory

The clearest immediate benefit of quantization is a smaller model weight footprint.

Consider a dense language model containing eight billion parameters.

At BF16 precision, each parameter requires two bytes. The raw weight storage is therefore approximately 16 GB.

At eight bit precision, the raw requirement falls to 8 GB.

At four bit precision, it falls to 4 GB.

These numbers are calculated from parameter count and bit width. They do not represent complete GPU memory requirements.

Memory Calculation for an 8B Model

BF16: 8 billion parameters × 2 bytes = 16 GB

8 bit weights: 8 billion parameters × 1 byte = 8 GB

4 bit weights: 8 billion parameters × 0.5 bytes = 4 GB

Actual memory consumption includes quantization scales, metadata, tensors retained at higher precision, temporary execution buffers, and the attention cache.

A four bit model therefore requires more memory than its theoretical 4 GB weight minimum.

This distinction matters when determining whether a model fits on a particular GPU.

A model that occupies 10 GB when loaded cannot safely be deployed on a GPU with exactly 10 GB of available memory. Inference still needs capacity for execution and generated context.

Weight Memory and KV Cache Memory

Quantizing model weights does not automatically quantize the KV cache.

The cache stores attention information associated with processed tokens. Its memory consumption increases with retained context length and active sequences for conventional full attention models.

A model with heavily compressed weights can still exhaust GPU memory when handling long prompts or many concurrent requests.

Weight quantization creates room for additional execution state, but the amount of usable capacity must be measured.

This distinction is essential in LLM Inference Optimization because reducing model storage and increasing the number of requests served are related objectives, not identical results.

FP8, INT8, INT4, and FP4 Quantization

Bit width determines how much information can be stored in each numerical value, but it does not fully describe how that value behaves.

Floating point and integer representations use different numerical structures.

FP8 Quantization

FP8 represents values using eight bit floating point formats.

Unlike an integer representation, floating point allocates bits to the exponent and significand, allowing values across different orders of magnitude to be represented.

FP8 is relevant to modern GPU inference because compatible hardware provides optimized execution for supported FP8 operations.

Depending on the quantization scheme, both weights and activations can use FP8.

This makes FP8 attractive when the objective includes faster matrix computation as well as reduced memory consumption.

Its performance benefit depends on the GPU architecture, scaling strategy, and available kernels.

INT8 Quantization

INT8 represents values using eight bit integers combined with quantization scales.

It supports different execution schemes.

In W8A16, weights use eight bits while activations remain at sixteen bit precision.

In W8A8, both weights and activations use eight bit representations.

W8A8 can take advantage of optimized integer matrix operations when the hardware and runtime support them.

The distinction matters because compressing only the weights does not provide the same computational opportunities as executing supported operations with quantized weights and activations.

INT4 Quantization

INT4 reduces weight storage to four bits per value.

A common approach is W4A16, where weights occupy four bits while activations use sixteen bit precision.

This is particularly useful when memory capacity prevents a model from fitting on the target hardware.

However, running a four bit model efficiently requires compatible kernels to unpack or process compressed weights.

If the runtime spends excessive time converting values into a form supported by the GPU, reduced storage does not translate into faster inference.

INT4 should therefore be evaluated first as a memory reduction strategy, with speed treated as a measured outcome.

FP4 Quantization

FP4 uses four bit floating point representations.

It is not interchangeable with INT4 because the numerical formats and supported operations differ.

Hardware implementations such as NVIDIA NVFP4 combine low precision values with scaling mechanisms designed for compatible accelerators.

FP4 offers substantial compression and can provide inference acceleration on supported hardware.

The tradeoff is greater sensitivity to numerical representation and a stronger dependence on the GPU architecture and execution backend.

Choosing an FP4 checkpoint without confirming hardware support is not a reliable deployment strategy.

LLM Quantization Methods

Precision defines the numerical representation. A quantization algorithm determines how a model is converted into that representation while managing the resulting error.

This distinction explains why two models with four bit weights can behave differently despite having similar storage requirements.

GPTQ

GPTQ is a Post Training Quantization method designed for compressing model weights.

It uses approximate second order information to guide quantization and compensate for errors introduced when weights are compressed.

The method is especially relevant to low precision weight quantization, including four bit configurations.

GPTQ requires an initial quantization process, but the resulting checkpoint can be reused for inference without repeating that process for every request.

Its deployment performance depends on the checkpoint layout and availability of optimized execution kernels.

AWQ

Activation Aware Weight Quantization, or AWQ, uses information from activations to identify weights whose quantization errors have a greater effect on model behavior.

It then applies scaling techniques to protect important weight information during compression.

AWQ is widely associated with four bit weight quantization.

Its main objective is reducing weight memory while maintaining useful model accuracy.

Like GPTQ, AWQ does not automatically determine inference speed. The execution backend must efficiently process the resulting representation.

SmoothQuant

SmoothQuant addresses a different difficulty: activation values can contain outliers that make integer quantization less accurate.

The technique redistributes numerical scaling between weights and activations, making both easier to quantize.

This supports W8A8 INT8 execution while controlling numerical error.

SmoothQuant is especially relevant when the deployment objective is efficient matrix computation using quantized weights and activations, rather than weight compression alone.

The choice between GPTQ, AWQ, and SmoothQuant should therefore begin with the intended execution scheme.

LLM Quantization Methods Compared

The most useful comparison is not simply which technique uses the fewest bits. The correct method must fit the available hardware, meet application quality requirements, and use an efficient inference backend.

The following table connects common quantization choices to their practical deployment goals.

These choices are not equivalent alternatives in every deployment.

GPTQ and AWQ describe quantization methods, while FP8 and INT4 describe numerical representations. W8A8 and W4A16 describe the precision assigned to weights and activations.

A model checkpoint may therefore combine a numerical scheme with a particular quantization algorithm.

GGUF adds another distinction. It is a model file format that can store quantized tensors, not an alternative numerical quantization algorithm.

When selecting a checkpoint, the quantization method, storage format, and inference runtime must be considered together.

Quantization Accuracy and Model Quality

Quantization changes numerical values, which means it can change model predictions.

The amount of change depends on the selected precision, scaling strategy, calibration process, and sensitivity of the model's computations.

Higher compression creates a more restrictive numerical representation.

That does not establish how much quality a particular application will lose.

A model can retain strong performance on broad language benchmarks while producing measurable errors in specialized tasks.

Evaluation Beyond General Benchmarks

For a production application, the most valuable evaluation data comes from the tasks the model is expected to perform.

A coding assistant should be evaluated using code generation and execution tests.

A document extraction system needs checks for field accuracy, missing values, and output structure.

A mathematical reasoning application requires problems that test calculation accuracy and the ability to follow multi step reasoning.

Comparing a quantized model against its BF16 baseline should use identical prompts, generation settings, and evaluation criteria.

Automated quality checks provide stronger evidence than reviewing a handful of generated responses.

Sensitive Layers and Mixed Precision

Quantizing every model component to the same precision is not always the best approach.

Some layers or operations are more sensitive to numerical changes.

Mixed precision strategies retain higher precision in selected components while compressing others.

This reduces the quality impact of quantization at the expense of some additional memory.

The relevant question is not whether the entire model uses four bits. It is whether the final representation achieves the required memory reduction without crossing the application's quality limits.

Quantization and Inference Speed

Reducing model weight storage decreases the amount of parameter data that must be held in memory.

It can also reduce memory traffic during decoding.

However, execution speed depends on more than the number of bytes stored.

The GPU must perform mathematical operations on the quantized representation, and the inference engine needs kernels that support those operations efficiently.

Memory Bandwidth and Decode Performance

During autoregressive decoding, model execution repeatedly accesses weights and attention information.

When memory bandwidth limits performance, reducing the amount of weight data transferred can improve token generation speed.

The benefit depends on how the quantized weights are processed.

An optimized kernel can perform matrix operations directly in a supported reduced precision format.

Another implementation may need additional conversion or unpacking operations.

Those operations introduce overhead that reduces or eliminates the expected speed improvement.

This explains why a four bit model can use less memory than an eight bit model without delivering higher tokens per second.

Prefill and Throughput Differences

Prefill processes the prompt using substantial parallel computation.

Decode generates subsequent output tokens and places different demands on memory bandwidth and GPU execution.

A quantization method that improves decode performance does not necessarily provide the same benefit during prefill.

Concurrency also changes the result.

With many requests active, the GPU processes larger amounts of work, affecting the balance between computing capacity and memory movement.

Performance testing should therefore distinguish initial prompt processing, subsequent generation speed, and aggregate request throughput.

A single tokens per second result does not describe every aspect of quantized inference.

Choosing the Right Quantization Method

The decision should begin with the constraint that needs to be removed.

Limited GPU Memory

When model weights do not fit within available memory, weight compression is the first priority.

A supported W4A16 configuration using GPTQ or AWQ is a relevant candidate.

The reduced memory requirement can make deployment possible on hardware that cannot hold the BF16 model.

The next step is measuring actual loaded memory and confirming enough capacity remains for runtime operations and the expected context.

High Throughput Requirements

For a service processing many concurrent requests, the priority extends beyond fitting model weights into memory.

Supported W8A8 execution offers opportunities for optimized matrix computation, while compatible FP8 implementations provide another route on modern accelerators.

The selection must account for the target GPU and inference engine.

Compare request throughput, token generation speed, and latency at the concurrency levels the service must sustain.

Strict Output Quality Requirements

When even small numerical differences can affect application results, more conservative precision choices deserve evaluation first.

An eight bit representation provides a useful starting point before testing more aggressive compression.

If a four bit model is required by memory constraints, representative calibration and task specific evaluation become essential.

The correct precision is the lowest one that satisfies both the deployment requirements and the application's quality criteria.

Hardware and Inference Engine Compatibility

A quantized checkpoint is useful only when the serving environment supports its numerical format and execution method.

GPU architecture determines which low precision operations are accelerated.

Inference engines determine which model formats, packing layouts, and quantized kernels are available.

For example, vLLM supports several quantization implementations, including GPTQ, AWQ, INT8, and FP8, but hardware compatibility differs between them.

NVIDIA TensorRT LLM also supports different quantization recipes depending on GPU generation.

A checkpoint designed for one runtime should not be assumed to perform equally well when loaded through another.

Selecting a Compatible Checkpoint

Before deploying a quantized model, verify its base model architecture, quantization configuration, supported runtime, and hardware requirements.

Check whether the checkpoint stores only quantized weights or includes additional metadata and scaling information required by the execution backend.

The selected runtime must recognize the checkpoint format and provide compatible kernels.

GGUF checkpoints, for example, are associated with execution environments such as llama.cpp, while other quantized formats are commonly used with GPU serving engines.

The specific loader matters because some frameworks convert stored quantized values back to higher precision during loading.

A compressed file on disk does not guarantee equally compressed execution in memory.

Evaluating Quantized LLM Performance

A useful comparison starts with the original model running on the same hardware and processing the same workload.

The baseline should record model quality, GPU memory consumption, latency, and throughput.

After quantization, repeat the measurements using the same prompts, output limits, sampling settings, and concurrency.

Memory usage establishes whether compression achieved its deployment objective.

Time to First Token measures responsiveness during prompt admission and processing.

Time per Output Token measures the speed of continued generation.

Request throughput shows how many requests the server completes under load.

Task accuracy establishes whether the compressed model still meets application requirements.

These measurements should be considered together.

A model that saves substantial memory but produces unacceptable outputs has not achieved a useful optimization.

Likewise, a model that preserves quality but increases latency beyond the service requirement is not necessarily a suitable replacement.

Comparing Total Serving Cost

Memory reduction creates several potential economic benefits.

A smaller model can fit on a less expensive accelerator, require fewer GPUs, or leave more capacity available for additional requests.

However, cost should be calculated using measured service performance rather than weight size alone.

For a dedicated inference deployment, compare infrastructure expense against completed requests or generated tokens over the same measurement period.

Include the GPU resources required by the quantized model, its supported concurrency, and the latency limits the service must satisfy.

The best result is the lowest sustainable cost for output that meets the application's quality requirements.

Final Thoughts

LLM quantization reduces the memory required to run large language models, but selecting the right precision is only the first part of the decision.

FP8 and INT8 provide options for efficient execution on supported hardware. GPTQ and AWQ enable substantial weight compression, while SmoothQuant addresses the challenge of quantizing both weights and activations.

The most aggressive compression is not automatically the best deployment choice. Numerical accuracy, kernel support, memory capacity, and actual inference performance determine whether the change provides value.

A successful quantization strategy makes the model less expensive to serve while preserving the output quality and response speed the application requires.

FAQ

Can I restore the original model weights after quantizing them?

Not exactly. Quantization discards numerical information when values are mapped into a lower precision representation. Converting those values back to BF16 or FP16 changes their storage format but does not recover the original precision. An unchanged copy of the original checkpoint is required for exact restoration.

Can I use LoRA adapters with a quantized model?

Yes, if the inference framework supports the selected quantization method and adapter combination. The adapter must be compatible with the underlying base model, and the runtime must support applying it to the quantized weights. Adapter merging and adapter execution are separate operations with different compatibility requirements.

Why is my GGUF model using more RAM than its file size?

The file size represents stored checkpoint data, not complete runtime memory. During loading, an inference framework may expand tensors into higher precision or allocate additional resources for model execution. Memory mapping also affects how file backed pages appear in system memory measurements. The runtime's actual tensor representation determines the result.

Does quantization work with Mixture of Experts models?

Yes. Quantization can be applied to the expert weights and other supported components of Mixture of Experts architectures. The routing mechanism and sensitive layers require appropriate numerical treatment. Evaluation must include workloads that activate different experts because a narrow test collection does not establish consistent behavior across the entire model.

Will quantization increase my model's maximum context length?

Not by itself. Quantization reduces selected memory requirements but does not change the model's architectural context limit. The freed memory can support additional KV cache capacity when the inference runtime permits it. Extending the context beyond the model's supported configuration requires separate architectural or positional encoding considerations.