tekBlueprint

Part 3 covered parallelism and speed techniques. This is the final part (4 of 4) of the Production LLM Inference series. It covers inference frameworks, production operations, and a practical decision framework for choosing the right stack.

Part 1: Architecture and Metrics → Part 2: Quantization and KV Cache → Part 3: Parallelism and Speed ← Part 4: Frameworks and Operations

Inference Frameworks

vLLM

vLLM is a general-purpose LLM serving engine built around PagedAttention, a technique that treats the key-value cache as virtual memory with non-contiguous physical allocation, eliminating fragmentation without requiring manual cache management. Its implementation of continuous batching allows the engine to interleave prefill and decode phases across requests, keeping the GPU fed with work throughout each iteration. Automatic Prefix Caching (APC) is built into the scheduler: vLLM tracks shared prefixes across requests via block hashing and reuses previously computed KV cache blocks when a new request overlaps with cached content.

The framework supports multiple quantization formats out of the box, including AWQ, GPTQ, INT8 weight-only, FP8 with mixed KV cache precision, and GGUF for CPU offloading. These formats load directly from checkpoint files without requiring a separate quantization pipeline, which simplifies deployment of memory-constrained workloads. Tensor parallelism and pipeline parallelism are implemented with NVIDIA’s NCCL backend, and vLLM includes native support for mixture-of-experts models with expert-level parallelism for large MoE topologies.

Multi-token prediction is supported for models that expose draft tokens natively, such as Gemma 4 with its draft heads, as well as Medusa and EAGLE-style feature-level speculation. The vLLM engine routes draft tokens through the same PagedAttention infrastructure, so MTP inference does not require a separate cache layout.

The server exposes an OpenAI-compatible API, including streaming, chat completions, and token-level logprobs, with built-in observability that surfaces per-request metrics such as time-to-first-token, inter-token latency, and cache hit rates. Recent versions support disaggregated serving, where prefill and decode can be assigned to separate GPU pools for optimal utilization. This makes vLLM a practical default choice for general-purpose serving, research experimentation, and mixed workloads that combine different model sizes and request patterns on the same cluster. Its main limitations are subtle: because vLLM is hardware-agnostic, it does not exploit NVIDIA-specific kernel fusions to the same degree as TensorRT-LLM, which can leave single-digit percentage throughput gains unrealized on premium GPU hardware.

TensorRT-LLM

TensorRT-LLM is NVIDIA’s official inference stack for LLMs, built on the TensorRT compiler with purpose-written CUDA kernels for attention, layernorm, and activation fusion that eliminate inter-layer memory traffic. The build process compiles a model configuration into an engine file that bakes in quantization levels, parallelism strategy, and kernel selection, producing highly optimized binaries for a specific hardware target. Weight streaming and layer fusion reduce activation footprint during decode, enabling TensorRT-LLM to push throughput closer to the theoretical memory bandwidth limit of A100, H100, and B200 GPUs.

Quantization is the framework’s strongest differentiator. FP8 precision is supported natively with hardware-accurate scaling factors for both weights and KV cache, and INT8 and INT4 are available through quantization-aware training pipelines that minimize accuracy regression. The QAT flow uses NVIDIA’s Custom Allreduce and per-channel scaling to keep quantization error within acceptable bounds for language modeling. Research approaches like QuaRot (rotated hidden states for outlier-free 4-bit inference) and QServe’s QoQ (W4A8KV4) demonstrate the full potential of ultra-low-precision Tensor Core utilization, though these are not integrated into the TRT-LLM stack. For production deployments where absolute throughput matters, TensorRT-LLM consistently benchmarks at or near the ceiling for NVIDIA hardware.

Parallelism is extensive: tensor parallelism (typically up to 8-way per node, scalable beyond via multi-node configurations), pipeline parallelism with inter-stage buffering, context parallelism for long-sequence workloads, and auto-parallel that selects an optimal split strategy based on model size and available GPUs. Mixture-of-experts models benefit from expert parallelism and cross-node communication optimizations built into the TensorRT-LLM runtime. The framework scales seamlessly from single-GPU deployments to multi-node clusters with dozens of GPUs, including disaggregated configurations.

The trade-off is operational complexity. Building a TensorRT-LLM engine requires GPU access during compilation, a working CUDA toolkit, and careful tuning of plugin configurations. The learning curve is steeper than vLLM’s, and the framework runs exclusively on NVIDIA hardware with no support for AMD, Intel, or CPU backends. Organizations locked into NVIDIA infrastructure and willing to invest in build pipelines find TensorRT-LLM’s throughput gains justify the overhead.

SGLang

SGLang differentiates itself through RadixAttention, a prefix caching system structured as a trie rather than a sequence-of-block cache. Unlike vLLM’s block-level prefix match, RadixAttention indexes individual tokens in a tree, enabling finer-grained reuse when requests share partial prefixes across different positions. For API workloads where system prompts are reused across hundreds of user requests, RadixAttention can save a significant fraction of prefill compute by reusing cache nodes for common prefix paths.

Structured generation is built into the framework at the sampling level. SGLang supports JSON schema constraints, regular expressions, and formal grammar constraints without post-generation filtering, because constrained decoding is integrated directly into the sampling step via grammar-based vocabulary masking. This eliminates the round-trip of generate, validate, and retry, making SGLang practical for downstream systems that require deterministic output formats.

The underlying attention kernels are provided by FlashInfer, which offers optimized implementations for both prefill-phase and decode-phase attention with variable sequence length support. SGLang’s OpenAI-compatible API makes it easy to drop into existing service architectures, and the framework handles concurrent requests with continuous batching similar to vLLM. SGLang also includes robust support for speculative decoding and multi-token prediction through its integration with FlashInfer’s draft model infrastructure, including Medusa and EAGLE-style drafting.

Quantization support has expanded significantly in recent versions: SGLang now supports AWQ, GPTQ, FP8 (including FP8 KV cache), and GGUF formats, making it competitive with vLLM for memory-constrained deployments. The project continues to evolve rapidly, with new optimizations for long-context workloads and multi-modal models.

llama.cpp

llama.cpp takes a CPU-first approach that requires no GPU, no CUDA toolkit, and no proprietary runtime. The project implements the full attention, layernorm, and MLP stack in C++ optimized for SIMD instruction sets, with AVX2 and AVX512 paths that run efficiently on consumer-grade processors. GPU offload is available through Vulkan, Metal (macOS), and CUDA backends, but the primary design goal is making LLM inference work on hardware that cannot afford a dedicated accelerator.

The GGUF file format is designed specifically for quantized model storage, with per-layer quantization types that mix precision across weight groups. Supported formats range from Q4_K_M, which compresses 4-bit weights with a per-group 16-bit scale and outlier 8-bit channels, up to Q8_0 and fully dequantized FP16 variants. The K-mixed quantization strategy keeps attention and output weights at higher precision while compressing intermediate MLP weights more aggressively, trading a small amount of accuracy for large memory savings.

The llama-server binary provides a full OpenAI-compatible API out of the box, including /v1/chat/completions and /v1/completions endpoints with streaming support. This makes llama.cpp suitable not just for local experimentation, but for lightweight production deployments on edge devices or CPU-only servers. Throughput on CPU is lower than GPU-accelerated frameworks, typically measured in tokens per second rather than tens or hundreds of tokens per second. Continuous batching is supported, though it is less optimized than in GPU-accelerated frameworks, and there is no native quantization pipeline: GGUF files must be produced by the project’s external conversion tools. For edge deployment, consumer hardware testing, and non-NVIDIA GPU environments where other frameworks lack support, llama.cpp remains the most accessible option.

TGI (Text Generation Inference)

TGI is Hugging Face’s production inference server, built with custom Rust and CUDA kernels optimized for high-throughput serving. The framework was an early adopter of PagedAttention concepts and continues to evolve its memory management strategies. Tensor parallelism is supported, as is speculative decoding with separate draft models, and the server provides optional watermarking for output attribution. TGI integrates directly with the Hugging Face Hub, enabling model pull-and-run workflows that fit into containerized deployment pipelines.

The framework is NVIDIA-focused, with CUDA kernels and NCCL collective operations optimized for A100 and H100 hardware. TGI’s feature set overlaps with vLLM in many areas, including batching, tensor parallelism, and constrained decoding, and in recent benchmark comparisons vLLM has been reported to supersede TGI on raw throughput for several model sizes. Note: the TGI repository was archived by Hugging Face in March 2026 and is now in maintenance mode. HF recommends vLLM, SGLang, llama.cpp, or MLX for new deployments. TGI remains a practical choice for teams already embedded in the Hugging Face ecosystem, but its momentum has shifted as vLLM adopted many of its design choices with broader hardware support and more active community development.

Frameworks Comparison Table

Approximate 2026 figures on a single H100 for a 70B parameter model. “High” and “Low” are relative rankings within the column; exact numbers vary with batch size, sequence length, and precision.

Production Operations

Choosing the right model, quantization, and framework is only half the battle. Operating an LLM serving stack in production introduces a distinct class of challenges centered around infrastructure scaling, monitoring, and reliability.

Observability and Metrics

Standard web-server metrics like CPU usage and network I/O are insufficient for LLMs. Production observability requires tracking the four inference metrics (TTFT, ITL, Throughput, Cost) at a per-request level. Modern serving frameworks expose native Prometheus endpoints (/metrics) that export queue depths, active batch sizes, KV cache block utilization, and token generation rates.

For distributed tracing, tools like OpenLLMetry or LangSmith integrate with OpenTelemetry to trace a request as it moves through the API gateway, tokenizer, prefill phase, and decode loop. This is critical for diagnosing latency outliers: a spike in TTFT might be caused by queue waiting time, while an ITL spike might be caused by intermittent chunked prefill interference or memory swapping.

Autoscaling and Cold Starts

Autoscaling LLM workloads is notoriously difficult due to the cold start problem. Loading a 70B model (even quantized to 35 GB) from disk to VRAM takes minutes. If an autoscaler waits for CPU/GPU utilization to breach a threshold before spinning up new pods, the incoming traffic will time out long before the new capacity is ready.

Production deployments must use predictive or queue-based autoscaling. Instead of reacting to CPU usage, the Kubernetes Event-Driven Autoscaling (KEDA) or custom controllers monitor the vllm:num_requests_waiting metric. When the queue depth grows, new pods are triggered. To mitigate cold starts, teams maintain “warm pools” of idle pods with weights pre-loaded in CPU RAM, allowing VRAM transfer to happen in seconds rather than minutes.

Fault Tolerance

In distributed LLM serving, hardware failures are statistical certainties. In a deployment using TP=8 across 100 nodes, a single GPU fault or NCCL timeout crashes the entire model replica.

For TP-replicated models, graceful degradation is rarely possible; if one GPU drops, the TP group fails. Fault tolerance is typically handled at the routing layer: a load balancer detects the failed replica’s health endpoint failing, reroutes traffic to healthy replicas, and signals the orchestrator to replace the pod. For disaggregated serving, if a prefill node crashes mid-computation, the router must detect the dropped connection and retry the request on another prefill node, ensuring no user-facing failure.

Decision Framework

The optimal framework choice depends on hardware constraints, latency targets, workload patterns, and operational tolerance for complexity. The following scenarios cover common deployment profiles.

Edge / Consumer Hardware

For deployment on machines without a GPU, use a dense 7-13B model with GGUF Q4_K_M quantization served by llama.cpp. The mixed-precision format keeps memory under 6 GB for a 7B model, and the CPU SIMD path provides interactive throughput on modern processors with at least 16 threads. There is no setup dependency beyond the binary, no quantization pipeline, and no CUDA toolkit. The OpenAI-compatible API allows integration with existing tooling. Accept that throughput will be measured in single-digit tokens per second, and avoid workloads that require sub-second time-to-first-token.

Small Cluster (2-4 GPUs) – Low Latency

When running on 2-4 GPUs with a latency-first requirement, deploy a dense 13-70B model quantized to FP8 (on Hopper hardware) or INT8 via AWQ (on older hardware) and serve it through vLLM with tensor parallelism across available GPUs. FP8 or INT8 reduces VRAM usage to roughly half of FP16 while maintaining 95-99 percent of full-precision accuracy on standard benchmarks. vLLM’s continuous batching and Chunked Prefill keep TTFT low under moderate concurrency, and the OpenAI-compatible API minimizes integration effort.

Small Cluster – Chat with Shared Prompts

For chat or agent workloads where system prompts are shared across thousands of requests, use SGLang to leverage RadixAttention’s trie-based prefix cache. The tree structure reuses cache nodes for common prefix paths more granularly than block-level caching, reducing prefill compute for repeated system messages. Combine this with SGLang’s built-in structured generation to enforce JSON schemas on API responses without post-processing. The expanded quantization support (AWQ, GPTQ, FP8) makes this configuration viable even for memory-constrained deployments.

Large Cluster (8+ GPUs) – Maximum Throughput

For high-throughput production on 8+ NVIDIA GPUs, pair a MoE model with MLA architecture, FP8 quantization for both weights and KV cache (or INT4 QAT if FP8 is not available), and serve it through TensorRT-LLM using 3D parallelism (tensor, pipeline, and expert parallel). The FP8 format keeps accuracy within acceptable bounds while cutting memory to half of FP16 precision, and scales seamlessly across multi-node deployments. TensorRT-LLM’s fused kernels and compiled engine graphs achieve the highest measured throughput on H100 hardware, and expert parallelism distributes MoE routing across nodes without excessive inter-node communication overhead.

Large Cluster – MTP Accelerated

If the workload generates long outputs with low entropy (code, documentation, structured data) and the chosen model supports multi-token prediction, deploy a model with native MTP heads (e.g., Gemma 4 31B) through vLLM. MTP produces 2-3x decode acceleration for small batch sizes (BS=1-4) because the draft heads autoregressively predict multiple tokens per forward pass, reducing the number of full-model iterations. vLLM’s PagedAttention infrastructure handles the expanded cache requirements transparently. This configuration is optimal when decode speed for individual requests, not aggregate throughput, is the binding constraint. For high-throughput workloads with large batches, disable MTP to avoid compute contention.

Budget-Constrained Serving

When GPU budget is tight, reduce model size to 7-13B and apply INT4 AWQ quantization through vLLM, augmented with speculative decoding using a smaller drafter model. AWQ INT4 can reduce VRAM to roughly one-quarter of FP16 usage with typically 1-3 percent accuracy regression on standard benchmarks. The smaller drafter, typically a 1-3B model, speculates tokens that the base model verifies, adding decode speed at the cost of drafter VRAM overhead. This stack maximizes capability per GPU dollar, particularly for latency-sensitive workloads with small batch sizes.

Multi-Model / Multi-Tenant Serving

For environments that must serve multiple models or isolate tenants on the same cluster, run vLLM with continuous batching and per-model tensor parallelism. vLLM supports running multiple models within a single process, with the scheduler interleaving requests by model, and dynamic batching groups requests by precision to maximize GPU utilization. OpenAI-compatible routing and request-level observability make it straightforward to implement tenant isolation at the API level.

Disaggregated Cluster Serving

For very large deployments (100+ GPUs) with mixed workloads of long-context and short-context requests, implement disaggregated serving with separate prefill and decode pools. Assign high-memory-bandwidth GPUs (H100/H200) to the decode pool and compute-optimized GPUs to the prefill pool. Connect them via high-speed InfiniBand or RoCE for KV cache transfer. This configuration maximizes overall utilization and minimizes ITL variance, though it requires careful load balancing and monitoring of the inter-pool network.

TCO and Economics: CapEx vs. OpEx

The deployment choice is ultimately dictated by Total Cost of Ownership (TCO). Buying H100 or B200 GPUs outright (CapEx) requires keeping utilization above 80% 24/7; otherwise, the amortized cost per token becomes uncompetitive. If traffic is spiky–high during the day, idle at night–renting capacity via OpEx cloud providers (AWS Bedrock, GCP Vertex, Together AI) is often cheaper despite higher per-token pricing, because you stop paying when traffic drops.

Furthermore, power consumption (TDP) is a dominant operational cost. An H100 SXM draws 700W; a B200 draws over 1000W. Over a 3-year lifespan, the cost of electricity and datacenter cooling can exceed the initial hardware purchase price. This economic reality heavily favors architectures that minimize FLOPs per token–making MoE models and aggressive quantization (W4A4/FP8) not just technical optimizations, but financial necessities at scale.

Decision Matrix Table

Summary

LLM inference optimization has matured into a layered stack where each technique addresses a distinct bottleneck, and the right combination depends on the specific constraints of each deployment. The table below maps common problems to their primary solutions and associated trade-offs.

There is no single configuration that dominates across all dimensions. A dense model with FP16 precision served by TensorRT-LLM will achieve the lowest latency but consumes the most memory per request and requires careful multi-node configuration for scaling. A MoE model with MLA architecture, FP8 quantization, and 3D parallelism delivers the highest throughput at cluster scale but introduces routing overhead and requires careful architecture selection. A smaller dense model with speculative decoding and prefix caching achieves a practical middle ground for mixed workloads.

The recommended starting point for most teams is a dense model with post-training quantization (FP8 on Hopper hardware, INT8/INT4 on older GPUs) served by vLLM. This stack deploys quickly, supports OpenAI-compatible routing, and provides enough observability to identify bottlenecks in the target workload. Once those bottlenecks are measured, iterate toward more specialized optimizations: add speculative decoding if decode latency at small batch sizes is the constraint, switch to QAT and TensorRT-LLM if throughput per dollar is the constraint, move to MTP models if long-output generation with low entropy is the primary use case, or implement disaggregated serving if you operate at cluster scale with heterogeneous workloads. The optimization path is incremental, not mandatory.

Glossary

References

1. Architecture and Attention Mechanisms
   1. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. arXiv:2305.13245
   2. Fast Transformer Decoding: One Write-Head is All You Need (MQA). arXiv:1911.02150
   3. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (MLA). arXiv:2405.04434
   4. DeepSeek-V3 Technical Report (MoE with auxiliary-loss-free balancing). arXiv:2412.19437
2. Mixture of Experts (MoE) and Routing
   1. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961
   2. Mixture-of-Experts with Expert Choice Routing. arXiv:2202.09368
   3. Mixtral of Experts (MoE architecture in practice). arXiv:2401.04088
3. Multi-Token Prediction (MTP) and Speculative Decoding
   1. Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192
   2. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774
   3. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv:2401.15077
   4. Gemma 4 Technical Report (Native MTP heads and QAT checkpoints). arXiv:2607.02770
4. Quantization Techniques
   1. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323
   2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978
   3. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438
   4. QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs. arXiv:2404.00456
   5. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving (QoQ algorithm). arXiv:2405.04532
5. Parallelism and Disaggregated Serving
   1. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053
   2. Ring Attention with Blockwise Transformers for Near-Infinite Context (Context Parallelism). arXiv:2310.01889
   3. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving. arXiv:2401.09670
   4. Splitwise: Efficient Generative LLM Inference Using Phase Splitting. arXiv:2311.18677
6. KV Cache Management
   1. Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM). arXiv:2309.06180
   2. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention). arXiv:2312.07104
7. Inference Frameworks and Hardware
   1. TensorRT-LLM Developer Guide. NVIDIA Docs
   2. llama.cpp & GGUF Format Specification. GitHub
   3. Text Generation Inference (TGI) Documentation. HF Docs
   4. Hopper Architecture In-Depth Whitepaper (FP8 Tensor Cores, 3.35 TB/s HBM3). NVIDIA Resources
   5. Gemma 4 MTP Documentation. Google AI
   6. Documentation. docs.vllm.ai