tekBlueprint

Part 1 covered model architectures and the key metrics of production inference. This is Part 2 of 4 – it focuses on the two largest VRAM consumers: model weights and the KV cache. We’ll examine how quantization shrinks model weights and how KV cache techniques reclaim memory for higher batch sizes.

Part 1: Architecture and Metrics ← Part 2: Quantization and KV Cache → Part 3: Parallelism and Speed → Part 4: Frameworks and Operations

Quantization

What Is Quantization

Quantization reduces the numerical precision of a model’s weights and activations. A model trained in FP32 uses 32 bits per number. Quantization maps those values to fewer bits, typically FP16, BF16, FP8, INT8, INT4, or increasingly INT2. The tradeoff is always the same: fewer bits per parameter mean less memory and faster computation, at the cost of some representation fidelity.

Why does this matter? Large language models are bounded by three resources: VRAM capacity, memory bandwidth, and compute throughput. Quantization addresses all three simultaneously. Halving the bit width halves the memory footprint, doubles the effective bandwidth, and often allows the hardware to process more elements per cycle. For a 70B parameter model, moving from FP16 to INT4 can reduce VRAM needs by roughly 75%, turning a 140 GB requirement into roughly 35 GB.

Quantization operates at different granularities. Per-tensor quantization uses a single scale and zero-point for an entire tensor. Per-channel quantization assigns one scale per output channel, preserving more detail for outlier channels. Per-group quantization splits each channel into small groups, typically 32 or 64 elements, quantizing each group independently. Per-token quantization applies scales at attention-scoring granularity, often for KV cache compression. Finer granularity preserves accuracy better but adds overhead to track scale metadata.

FP8: The Data Center Standard

By 2026, FP8 has emerged as the default precision for data center LLM inference on H100, H200, and B200 GPUs. FP8 comes in two formats: E4M3 (4-bit exponent, 3-bit mantissa) for weights and activations, and E5M2 (5-bit exponent, 2-bit mantissa) for gradients during training. The E4M3 format provides sufficient dynamic range for weight distributions while fitting into the native FP8 Tensor Cores.

FP8 quantization typically requires minimal calibration compared to INT4. Most models can be converted to FP8 with simple per-tensor or per-channel scaling factors computed from a small calibration dataset. The accuracy loss is often negligible – typically 0.1-0.5% on standard benchmarks – making FP8 the sweet spot for production deployments where both performance and quality matter.

The Hopper architecture (H100/H200) supports FP8 with hardware-accurate scaling, enabling true FP8 computation without dequantization overhead. This makes FP8 particularly attractive for mixed-precision deployment: weights in FP8, KV cache in FP8, and activations computed in FP8 with selective upcasting to FP16 for numerically sensitive operations.

W4A4: Quantizing Both Weights and Activations

Until recently, production quantization followed the W4A16 pattern: 4-bit weights with activations remaining in FP16. This approach simplified calibration because only static weight distributions needed analysis. However, Hopper and later GPUs feature INT4 Tensor Cores that require both operands in INT4 to realize the full 4x compute speedup.

QuaRot has made W4A4 viable for production. It applies Hadamard rotation matrices to weight distributions before quantization, eliminating outliers and spreading quantization error evenly across channels. The result is INT4 weights and INT4 activations with accuracy degradation comparable to W4A16 PTQ, while unlocking the full compute throughput of INT4 Tensor Cores. For compute-bound deployments on Hopper hardware, W4A4 delivers measurably higher throughput than W4A16 at identical quality. Related work such as QServe (QoQ) targets a different configuration – W4A8KV4 – using rotation and smoothing for INT4 weights, INT8 activations, and INT4 KV cache simultaneously.

PTQ (Post-Training Quantization)

Post-training quantization converts a pre-trained model to lower precision without any gradient-based retraining. The process requires a small calibration dataset, typically 100 to 500 unlabeled examples similar to the model’s expected deployment domain. The calibration step runs the model in FP16 on these examples, collects activation statistics, and computes optimal scale factors for each quantization granularity. The calibration is usually complete within minutes on a single GPU.

GPTQ performs layer-wise quantization using second-order information. It approximates the Hessian matrix for each weight layer and uses Cholesky decomposition to factorize it, then quantizes columns of the weight matrix in an order that minimizes error propagation. The sequential quantization strategy processes one set of columns at a time, computing a restoration matrix that compensates for quantization error before moving to the next columns. GPTQ is fast, reliable, and delivers good accuracy for INT4 and INT3 targets, making it one of the most widely adopted PTQ methods for low-bit quantization.

AWQ (Activation-Aware Weight Quantization) improves on GPTQ by examining activation distributions during calibration. It identifies “important” channels – neurons with large activation magnitudes that contribute disproportionately to model accuracy – and preserves them at higher precision. Less important channels receive aggressive quantization. This selective approach consistently achieves better accuracy than GPTQ at the same bit width, typically closing a 1-2% gap on benchmarks like MMLU and HELM. AWQ works across INT4, INT8, and even FP8 quantization levels.

SmoothQuant takes a different angle. It addresses the fact that activations are harder to quantize than weights because activations have heavy-tailed distributions with extreme outliers. SmoothQuant applies a per-channel smoothing factor that scales down large pre-activations and scales up corresponding weight rows, shifting the quantization difficulty from activations entirely into weights. Since weights are static and easier to analyze, this redistribution enables aggressive activation quantization without accuracy collapse. SmoothQuant is particularly effective for INT8 deployment on hardware that quantizes both weights and activations.

PTQ methods share common strengths and weaknesses. The primary advantage is speed: no training infrastructure, no GPU cluster, no weeks of experimentation. A PTQ run might take minutes to hours. The primary disadvantage is accuracy ceiling: PTQ typically loses 1-8% accuracy depending on the target bit width, and there is a hard limit below which the model degrades unacceptably. For INT8 and FP8, the loss is often in the 0.5-2% range and may be negligible for production use. For INT4, losses climb to 3-8%, and for INT2, the degradation is frequently unacceptable without a QAT approach.

QAT (Quantization-Aware Training)

Quantization-aware training simulates low-precision arithmetic during the training process itself. Unlike PTQ, which is a post-hoc conversion step, QAT modifies the training loop so the model learns to operate under quantization noise from the beginning. The result is a model that is structurally adapted to survive aggressive precision reduction.

The core mechanism of QAT relies on two components. First, FakeQuant nodes are inserted into the computation graph at strategic points, typically before and after linear layer operations. During the forward pass, these nodes quantize values to the target precision and immediately dequantize back to FP32 or FP16, simulating the information loss that real integer arithmetic would produce. During the backward pass, no actual integer gradients exist, so QAT uses a Straight-Through Estimator (STE) to approximate gradients. The STE operator passes gradients directly through the quantization function, treating the step-like quantization boundary as if it were the identity function. This approximation is not mathematically rigorous but works in practice because it provides the optimizer with a signal proportional to the magnitude of values.

Over training epochs, the model’s weights and biases shift to account for the quantization noise. Neurons that are sensitive to precision loss adjust their magnitudes to more quantization-friendly values. Outlier activations distribute their influence across multiple channels rather than concentrating in heavy-tailed distributions. The model effectively learns robustness to integer arithmetic as a property of its parameters.

The primary advantage of QAT is accuracy. At INT4, a QAT model typically loses only 1-3% on standard benchmarks, compared to 3-8% for PTQ. At INT2, QAT may achieve 5-15% degradation where PTQ is often non-functional. For the lowest bit widths, QAT is not just an improvement – it is the only viable path.

The disadvantages are substantial. QAT requires full training infrastructure: GPU clusters, distributed training frameworks, data pipelines, and monitoring tooling. Training time is comparable to full fine-tuning, often days to weeks depending on model size. Recent developments have narrowed this gap somewhat. In June 2026, Google released Gemma 4 QAT checkpoints with 4-bit (Q4_0) and a mobile QAT format using selective 2-bit compression on token-generation layers, demonstrating that QAT can achieve production-quality accuracy at aggressive bit widths when applied by the original model authors with access to training data and compute.

GGUF and llama.cpp Quant Types

GGUF is a universal serialization format designed around the llama.cpp inference engine. It stores model weights along with quantization metadata, vocabulary, and architecture configuration in a single portable file. Because GGUF lives outside the GPU framework ecosystem, it has developed its own quantization types that prioritize CPU inference efficiency.

The most common GGUF quant types follow a naming convention. Q4_0 performs basic 4-bit quantization with per-group scaling. It is the baseline quantization level that balances memory usage and accuracy. Q4_K_M is part of the “K” family of quant types. The “K” quant types use mixed precision: most weights are quantized to 4 bits, but outlier channels – identified during quantization – are kept at higher precision (Q6 or Q8). The “M” suffix selects a medium granularity balance between memory and accuracy. Q5_K_M follows the same mixed-precision pattern but allocates 5 bits for the base precision, reclaiming accuracy at the cost of additional memory. Q8_0 uses straightforward 8-bit per-group quantization, delivering near-lossless accuracy with roughly half the memory of FP16.

The “K” quant types are the most significant innovation in GGUF quantization. By analyzing weight distributions and identifying channels with values that deviate significantly from the group mean, the quantizer assigns higher bit widths to outlier channels while compressing the majority of channels more aggressively. This approach recognizes that not all weights are equally important and that uniform quantization wastes bits on easily compressible regions while starving difficult ones.

GGUF quantized models are the default choice for CPU inference and edge deployment. The llama.cpp runtime uses efficient lookup-table-based matrix multiplication that avoids full dequantization to FP32, keeping arithmetic in integer and 8-bit domains for much of the computation. On consumer hardware with 16 GB of unified memory, a Q4_K_M quantized 13B model runs where the FP16 equivalent would not fit.

Quantization Format Comparison

The VRAM figures assume the model weights plus a moderate KV cache. Actual memory requirements increase with batch size and context length, since activation and KV cache memory is not always quantized. The speedup column reflects theoretical throughput gains from reduced memory bandwidth and higher compute utilization; real-world speedup depends on the hardware and whether the deployment is compute-bound or memory-bound. On memory-bound hardware, which is typical for large-batch LLM inference, the speedup approaches the reported values. On compute-bound hardware with very small batches, the speedup is small.

Choosing Quantization Level

The choice of quantization level depends on three constraints: acceptable accuracy tolerance, available hardware, and deployment latency requirements. The decision is rarely theoretical – it is driven by the specific model, the specific benchmark that matters for the use case, and the hardware that the deployment targets.

Start with FP16 or BF16 if the hardware can accommodate it and accuracy requirements are strict. BF16 is generally preferred over FP16 for its wider dynamic range, which reduces underflow issues during attention computation. This is the baseline for any production deployment where the GPU has sufficient VRAM.

Move to FP8 if you are on Hopper or Blackwell hardware and want near-lossless accuracy with 2x memory savings. FP8 is the sweet spot for data center deployments in 2026, offering the best balance of performance and quality on modern NVIDIA GPUs.

Use INT8 PTQ with AWQ or SmoothQuant if FP8 is not available on your hardware and accuracy loss in the 1-2% range is acceptable. INT8 remains a solid choice for older GPU architectures (A100, V100) where FP8 is not supported.

Use INT4 PTQ when VRAM is a hard constraint and a 3-8% accuracy loss is tolerable. This is common for edge deployment, consumer hardware, and scenarios where running a 30B model locally at INT4 is preferable to running a 7B model at FP16 remotely. Choose GPTQ for broader tooling compatibility or AWQ for a few percentage points of additional accuracy.

Reserve QAT for the highest-value models where INT4 accuracy must match or exceed PTQ, or where INT2 is required. QAT is a significant undertaking and is only justified when the model is large enough that the per-parameter savings outweigh the training cost. If the model provider releases QAT checkpoints – as Google did for Gemma 4 – use those rather than attempting QAT in-house.

One practical principle: quantize the smallest model that meets accuracy requirements at FP16, not the largest model pushed to the minimum viable bit width. A 13B model at INT8 with 2% loss is often more capable than a 30B model at INT2 with 12% loss, because accuracy at lower precision does not scale linearly with parameter count.

KV Cache Optimization

The KV cache is the single largest memory consumer during autoregressive inference. Understanding why, and understanding the strategies that reduce its footprint, is essential for building cost-effective serving systems.

The KV Cache Problem

Autoregressive language models generate text one token at a time. At each step, the model computes attention between the new query token and every previously generated or prompted token. Recomputing the key and value vectors from scratch at every step would make generation prohibitively expensive: processing a context of length L would require O(L^2) attention computations.

The KV cache eliminates this redundancy. After computing key and value vectors for a token during its first appearance, those vectors cache in GPU memory. Subsequent attention steps read the cached vectors instead of recomputing them. This reduces per-token compute from O(L^2) to O(L), enabling practical generation speeds.

The cache size grows proportionally to the context window, batch size, model depth, and hidden dimension. For a Llama 3 70B model with batch 32 and a 128K context window, the KV cache alone can exceed 100 GB. This single tensor structure often consumes more memory than the model weights themselves, and it constrains achievable batch sizes more directly than any other factor.

FlashAttention: IO-Awareness

Standard attention implementations materialize the full attention matrix in HBM, which consumes enormous amounts of memory and memory bandwidth. FlashAttention (and its subsequent v2/v3 iterations) is an IO-aware exact attention algorithm that fundamentally changed LLM scaling.

Instead of reading and writing the entire attention matrix to HBM, FlashAttention tiles the computation. It loads tiles of Q, K, and V from HBM into the GPU’s on-chip SRAM, computes attention block by block, and writes only the output tiles back to HBM. The KV cache itself remains in HBM; FlashAttention avoids the memory wall by never materializing the full N×N attention matrix. This reduces HBM traffic to approximately O(n + s) instead of O(n × s) for batch size n and sequence length s, allowing context windows to scale to hundreds of thousands of tokens without collapsing throughput.

Chunked Prefill

Before discussing cache management strategies, it’s worth mentioning Chunked Prefill – a technique that has become standard in modern serving frameworks like vLLM and SGLang.

When a request with a very long prompt (e.g., 100K tokens) arrives, processing it in a single prefill pass would monopolize the GPU for seconds, blocking all other requests and causing massive TTFT spikes for concurrent decode requests. Chunked Prefill solves this by splitting the prefill phase into smaller chunks (e.g., 4K or 8K tokens at a time) and interleaving them with decode steps from other requests.

The scheduler processes one chunk of the long prompt, then switches to processing decode tokens from other active requests, then returns to the next chunk. This keeps the GPU continuously fed with work, maintains low TTFT for short requests, and prevents memory spikes that could cause OOM errors. Chunked Prefill is particularly important for mixed workloads where long-context and short-context requests arrive simultaneously.

PagedAttention (vLLM)

PagedAttention draws an analogy to operating system virtual memory. Instead of requiring a single contiguous block of GPU memory for each request’s KV cache, it allocates cache in non-contiguous blocks. Each block holds a fixed number of tokens – typically 16 or 32 – and vLLM allocates blocks on demand as the sequence length grows.

This approach eliminates internal fragmentation. With contiguous allocation, every request reserves VRAM for its maximum possible context length, even if it generates only a few tokens. Under contiguous allocation, a deployment with requests ranging from 200-token to 128K-token responses wastes enormous amounts of reserved-but-unused memory. PagedAttention only allocates blocks that the request actually uses.

A secondary benefit is continuous batching. New requests can enter the generation loop at any time without waiting for existing requests to finish, because their KV cache blocks interleave with existing blocks in GPU memory. PagedAttention’s block table maps each request’s logical token positions to physical GPU blocks, maintaining this interleaving transparently.

Automatic Prefix Caching (APC) is built into vLLM’s scheduler. When multiple requests share the same system prompt or common prefix, vLLM computes a hash of each block and checks if that hash already exists in memory. If it does, vLLM reuses the cached block instead of recomputing it. This works well for exact prefix matches (e.g., identical system prompts) but is less effective for partial overlaps or when prefixes diverge early in the sequence.

The memory savings from PagedAttention are substantial in real deployments. Compared to contiguous allocation, vLLM achieves roughly 20 percent better VRAM utilization across typical API workloads with mixed sequence lengths. With APC enabled, savings can reach 30-40% for workloads with high prefix reuse.

RadixAttention (SGLang)

RadixAttention takes prefix caching further by building a radix tree – specifically a trie – over KV cache blocks. Each node in the tree represents a prefix of tokens, and its child edges represent continuations. When a new request arrives, SGLang checks the radix tree for existing nodes that match the request’s prefix and reuses their cached KV blocks.

This structure is most effective in chat API workloads. Consider a system where thousands of requests arrive with the same 4K-token system prompt followed by user-specific messages. Under RadixAttention, the 4K-token system prompt caches once in the radix tree, and all downstream requests share those blocks. Only the user-specific suffix requires new KV computation and allocation.

The key advantage of RadixAttention over block-level prefix caching is finer-grained reuse. If two requests share a 4K prefix but then diverge, and later share another 1K common segment, RadixAttention can cache and reuse both segments independently. Block-level caching would only cache the initial 4K prefix and miss the later overlap.

The radix tree requires active maintenance. As requests complete, their unique suffix nodes become candidates for eviction. SGLang uses a dynamic pruning strategy that tracks node reference counts: when no active request needs a suffix node, that node’s blocks free for reallocation. A reference-counting eviction policy ensures that memory stays bounded even under heavy prefix churn.

SGLang typically achieves 30-50 percent memory savings over PagedAttention in workloads where prefix sharing is common. When prefixes are entirely unique – such as document summarization with different documents per request – RadixAttention’s advantage diminishes and approaches PagedAttention’s baseline.

GQA, MQA, and MLA (architecture level)

Multi-Query Attention (MQA), Grouped-Query Attention (GQA), and Multi-head Latent Attention (MLA) reduce KV cache size through architectural design rather than runtime management.

Standard multi-head attention maintains separate key and value heads for each query head. MQA collapses all query heads to share a single KV head, reducing the cache by a factor equal to the number of attention heads.

GQA strikes a middle ground. Instead of one shared KV head, it groups multiple query heads together, with each group sharing one KV head. Llama 3 405B, for example, uses 128 query heads grouped into 8 KV head groups, achieving a 16x reduction in KV cache size relative to standard multi-head attention. Llama 3 70B uses 64 query heads with 8 KV heads (8x reduction), and Llama 3 8B uses 32 query heads with 4 KV heads (8x reduction).

Multi-head Latent Attention (MLA) represents the most aggressive cache compression to date. Popularized by DeepSeek-V2 and V3, MLA applies a low-rank joint compression to the key and value vectors. Instead of storing full K and V matrices, MLA projects them into a much smaller latent space using a down-projection matrix. Only this compressed latent vector is stored in the KV cache. At attention time, an up-projection reconstructs the full K and V vectors on the fly.

The compression ratio of MLA is dramatic: for DeepSeek-V3, the KV cache is reduced to roughly 5-10% of the standard multi-head attention size, which is even smaller than GQA-8. The quality loss is negligible because the up- and down-projection matrices are learned jointly with the rest of the model during training. MLA has become the standard attention architecture for new MoE models and long-context dense models, where KV cache pressure is the dominant constraint.

The trade-off between GQA and standard attention is nuanced. Some benchmarks show negligible quality loss when moving from full multi-head attention to GQA with reasonable group sizes. Other benchmarks show measurable degradation on specific reasoning or needle-in-a-haystack tasks. In practice, the KV cache savings of GQA make it the dominant choice for models above 13B parameters, because the reduced cache enables higher batch sizes that improve throughput more than any quality regression degrades it.

MQA, with only one KV head, achieves even greater reduction – up to 128x for models with 128 attention heads. However, the quality loss under MQA is more pronounced than under GQA, and most modern models reserve MQA for specific embedding or retrieval sub-networks rather than the main transformer stack.

KV Cache Quantization

Beyond reducing the number of cached vectors (GQA/MLA) or managing their allocation (PagedAttention), modern serving stacks also quantize the KV cache itself. In long-context, large-batch deployments, the KV cache can consume 60-80% of total VRAM, making its quantization more impactful than weight quantization alone.

The most common approach stores KV cache in FP8 (using the E4M3 format) with per-head or per-token scaling factors computed dynamically during generation. Because the distribution of KV values is relatively stable across layers and heads, FP8 KV cache incurs negligible quality loss – typically under 0.2% on standard benchmarks – while halving the cache memory footprint.

For even more aggressive compression, INT8 and INT4 KV cache formats are available in TensorRT-LLM and experimental vLLM builds. These require more sophisticated scaling strategies (e.g., per-token-per-head or smoothed quantization) to avoid attention score distortion, but can reduce cache memory by 4-8x relative to FP16.

The combination is powerful: a DeepSeek-V3 model served with MLA (architecture-level 5-10x cache compression), FP8 weights (2x), and FP8 KV cache (2x) achieves a memory footprint comparable to a dense 70B model in FP16, while delivering frontier-quality results.

KV Cache Comparison Table

These techniques are not mutually exclusive. A production deployment might use MLA or GQA at the architecture level to reduce cache size by 5-16x, quantize the cache to FP8 for another 2x reduction, use PagedAttention or RadixAttention to manage the remaining cache with minimal fragmentation, employ Chunked Prefill to handle long-context requests gracefully, and use TP to distribute the computation across GPUs. The combination pushes batch size limits further than any single technique.

One emerging approach for extreme workloads is KV cache offloading to CPU memory or NVMe SSDs. For requests with very long contexts that exceed GPU memory, frameworks can store inactive cache blocks on system RAM or fast NVMe storage and page them back on demand via RDMA. Systems like FlexGen and InfiniStore demonstrate that million-token contexts are achievable on a single GPU with aggressive offloading, though at the cost of higher per-token latency. This strategy introduces latency but enables serving requests that would otherwise be rejected. It remains a fallback rather than a primary strategy, reserved for corner cases where the context window and batch size jointly exceed GPU capacity.

Glossary

References

1. Quantization – Post-Training (PTQ)
   1. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323
   2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978
   3. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438
2. Quantization – W4A4 and Rotation-Based
   1. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. arXiv:2404.00456
   2. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving (QoQ algorithm). arXiv:2405.04532
3. Quantization-Aware Training (QAT)
   1. Gemma 4 Technical Report (QAT 4-bit and 2-bit checkpoints). arXiv:2607.02770
4. Attention and KV Cache
   1. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135
   2. Fast Transformer Decoding: One Write-Head is All You Need (MQA). arXiv:1911.02150
   3. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245
   4. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (MLA). arXiv:2405.04434
   5. DeepSeek-V3 Technical Report (MLA + MoE + MTP). arXiv:2412.19437
5. KV Cache Management
   1. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). arXiv:2309.06180
   2. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention). arXiv:2312.07104

Previous: Part 1 – Architecture and Metrics

Next: Part 3 – Parallelism and Speed covers multi-token prediction and distributed inference strategies.