tekBlueprint

Running large language models in production requires more than loading a checkpoint and calling forward(). The gap between research benchmarks and real-world serving is defined by latency targets, memory budgets, and cost constraints that research papers rarely address.

This is Part 1 of 4 in the Production LLM Inference series. This article covers model architectures and the key metrics that define inference. The subsequent parts cover quantization and KV cache, parallelism and speed, and frameworks and operations.

Why LLM Inference Is Hard

LLM inference has two fundamentally different phases, and they stress the hardware in opposite ways. The prefill phase processes the entire prompt in one forward pass. It is compute-bound, dominated by matrix multiplications, and scales with the number of tokens in the input. The decode phase generates one token at a time. It is memory-bound, limited by how fast weights can be read from HBM (VRAM), and scales with the generation length.

The autoregressive bottleneck is the core constraint. Each generated token requires a complete forward pass through every layer of the model. For a 70B-parameter model, reading all those weights from HBM takes significantly longer than the actual computation on modern GPUs. The memory bandwidth becomes the wall you hit, not FLOP count. This is why a GPU with more FLOPs but the same memory bandwidth often shows no decode speedup over its predecessor.

The problem is worse for batch inference. Multiple sequences in the same batch have different lengths and finish generating at different times. You either pad them (wasting compute) or use continuous batching, a paradigm pioneered by the Orca system (OSDI 2022) which introduced iteration-level scheduling. This allows the engine to dynamically insert and eject requests from the batch at any token iteration, maximizing GPU utilization but introducing its own complexity in memory management and kernel scheduling.

By 2026, the hardware landscape has shifted. H100 delivers 2.0 TB/s (PCIe) to 3.35 TB/s (SXM5), while H200 reaches 4.8 TB/s peak memory bandwidth. Model sizes are outpacing on-chip memory more than bandwidth. An 8x H100 80GB cluster holds 640 GB of HBM – enough for a quantized 70B model with KV cache, and workable for a dense 405B model in 4-bit quantization, though with only ~330 GB of headroom left for KV cache and activations. MoE models complicate the picture further: all experts must fit in memory even though only a fraction activate per token.

Production inference is measured against four metrics that often conflict:

• TTFT – Time to First Token. The latency from request submission to the first generated byte. TTFT is dominated by the prefill phase and the first decode step.
• ITL (Inter-Token Latency) – historically also referred to as TPOT (Time Per Output Token). The sustained decode latency between consecutive tokens after the first token. ITL determines how fast long outputs are produced.
• Throughput – Total tokens processed per second across all concurrent requests. Throughput favors batching and compute-bound operation.
• Cost per million tokens – Directly derived from hardware usage, batch size, and decode efficiency. Lower ITL means lower cost, but only if you are keeping GPUs busy.

These metrics form a tension triangle. Optimizing TTFT typically means running prefill on dedicated hardware, which reduces throughput. Optimizing throughput means maximizing batch size, which can increase ITL as GPU memory bandwidth is shared across more sequences. Reducing cost per token requires high utilization, which is hard to maintain with variable traffic patterns.

VRAM pressure is the silent constraint behind all of this. The model weights are the base cost. The KV cache grows linearly with sequence length and batch size. For a batch of 32 sequences at 8K context, the KV cache can consume as much memory as the model weights themselves. You cannot increase batch size to improve throughput without eventually hitting the VRAM wall.

Model Architectures

The choice between dense, mixture-of-experts, or multimodal models is the first and most consequential decision for production inference. It determines your hardware sizing, your optimization strategy, and your operational complexity.

Dense Models

Dense Transformer models activate every weight on every forward pass. Given an input sequence, the tokens pass through stacked layers of multi-head attention and feed-forward networks. All parameters contribute to every output token. This is the architecture of Llama 3.1, Mistral Large 2, Qwen3 dense variants (up to 32B), and the majority of models in production today.

The primary advantage of dense models is predictability. The compute required per token is constant and known in advance: three matrix multiplications per FFN layer (gate, up, and down projections via SwiGLU) and four for the attention layer (Q, K, V projections and the output projection), totaling seven per transformer layer. Memory access follows a regular pattern: stream weights from HBM, compute on Tensor Cores, write output, repeat. This regularity is exactly what GPU compiler stacks and kernel fusing strategies are optimized for.

Fine-tuning dense models is straightforward. LoRA, QLoRA, full fine-tuning – all techniques transfer directly from research to production. The parameter space is uniform, so adapter methods like LoRA produce consistent quality regardless of which layers are adapted. There are no routing decisions to break, no load balancing to worry about.

The disadvantage is linear scaling. Doubling the parameter count doubles compute per token and doubles VRAM usage. There is no way around this arithmetic. A 70B dense model requires roughly twice the resources of a 34B model. If you need more capacity, you buy more hardware.

Dense models excel at single-node deployment. A 7B model fits on a single consumer GPU after quantization. A 70B model fits on a single 8x-A100 node after aggressive quantization. You avoid the entire class of multi-node networking problems before you even start thinking about them.

Vision-Language Models (VLMs)

By 2026, pure text LLMs are increasingly replaced by VLMs (Vision-Language Models) in production. VLMs integrate a visual encoder (like a Vision Transformer, ViT) with an LLM backbone. The visual encoder processes images into sequences of “image patch tokens,” which are then projected into the LLM’s embedding space and concatenated with text tokens.

For inference, VLMs introduce a specific challenge: image patches drastically inflate the prefill sequence length. A single high-resolution image can easily translate to 1,000 to 4,000 tokens. This amplifies the compute-bound nature of the prefill phase, spiking TTFT and rapidly consuming KV cache memory. When serving VLMs, aggressive chunked prefill and context parallelism become mandatory even for moderate batch sizes.

Mixture of Experts (MoE)

MoE models use conditional computation. Each token is routed to a subset of “expert” networks, and only those experts process the token. The classic formulation splits the FFN layer into N parallel experts and a router that selects top-k experts per token. For a model with 16 experts and top-2 routing, each token activates only 1/8 of the FFN parameters.

The key insight from MoE is separation of two quantities: model capacity (total parameters) and per-token compute (active parameters). Mixtral 8x7B has 46.7B total parameters but only 12.9B active per token. This means the model achieves performance comparable to a 30B dense model while using compute equivalent to a 13B dense model per forward pass. The trade-off is that all 46.7B parameters must be stored in memory.

Real-world architectures vary significantly in their MoE design. DeepSeek V3 uses 256 experts with top-8 routing and one shared expert per token (671B total, 37B active). DeepSeek V4 retains the 256 expert count but replaces the Sigmoid gating function with Sqrt(Softplus), removes the fixed top-K constraint, and activates approximately 16 experts on average (V4-Pro: 1.6T total / 49B active; V4-Flash: 284B / 13B active). Mixtral 8x7B uses 8 experts with top-2 routing. Qwen3 MoE scales to 235B total / 22B active across 128 experts with no shared expert. Gemma 4 introduced a 26B MoE variant with 4B active, while Gemma 3 remains purely dense (1B–27B).

The advantages of MoE for inference are specific. Throughput per watt improves because each token requires fewer FLOPs. If you are compute-bound, a MoE model outperforms a dense model of equivalent quality. More importantly, MoE models achieve dense-quality reasoning at a fraction of the per-token compute cost. For API providers processing millions of tokens, this translates directly into lower hardware costs.

The disadvantages are more numerous. Routing failures occur when the router sends all tokens in a batch to the same expert. This creates a hotspot that serializes computation and destroys throughput. Expert imbalance – where some experts are consistently preferred – leaves unused capacity sitting in VRAM. Both problems are worse at small batch sizes and with short sequences that provide insufficient context for the router.

Communication overhead is the hidden cost of MoE. In single-node deployment, routing tokens between experts on the same GPU requires data movement within HBM. In multi-node deployment, tokens may need to be sent across NVLink or PCIe to reach their assigned experts. The all-to-all communication pattern for expert routing adds latency that scales with batch size and expert count.

There is a hard VRAM constraint. All experts must fit in memory simultaneously, even though only a fraction are active. A MoE model with 8 experts of 7B parameters each requires memory for all 56B parameters – not just the 14B activated per token. This means a MoE model can consume 3-4x more VRAM than a dense model of equivalent active parameter count.

MoE Routing in Detail

The router is a small linear layer that computes logits over the expert set for each token. For top-k routing, the router selects the k experts with the highest logit scores and assigns weights proportional to the softmax of those logits. The token’s FFN output is the weighted sum of the selected experts’ outputs.

Top-2 routing assigns each token to 2 experts. This is a balance between coverage and compute. With only top-1, a single router error drops the token to a misassigned expert with no fallback. With top-4, compute nearly doubles with diminishing returns on quality. Top-2 is the most common production choice.

The router weights determine everything. A well-trained router distributes tokens evenly across experts, maximizing parallelism. A poorly trained router concentrates traffic, creating bottlenecks. Router training uses an auxiliary load-balancing loss that penalizes unequal expert utilization. The loss is computed as the dot product between token frequency and expert routing probabilities:

L_aux = alpha * N * sum(f_i * P_i)

Where f_i is the fraction of tokens routed to expert i, P_i is the average softmax probability the router assigns to expert i across the batch (ideally P_i ~ 1/N), N is the total number of experts, and alpha is a scaling coefficient. This loss is minimized when token distribution matches uniform routing probability, pushing the router toward balanced expert utilization.

Routing failures manifest in three ways. Capacity overflow occurs when more tokens are routed to an expert than the expert’s token capacity allows. The standard handling drops the overflow tokens or redistributes them, both degrading quality. Expert collision happens when all tokens in a batch select overlapping expert sets, reducing the effective parallelism. Dead experts are experts that receive zero tokens across hundreds of forward passes, wasting their allocated memory.

Expert Choice Routing has emerged as an elegant alternative that eliminates these failure modes entirely. Instead of tokens choosing their top-k experts, each expert selects its top-N tokens from the batch. This inversion guarantees perfect load balancing – every expert receives exactly N tokens, with no overflow, no collision, and no dead experts. The auxiliary load-balancing loss becomes unnecessary. Originally introduced by Google research, Expert Choice (alongside similar mechanisms like the auxiliary-loss-free balancing used in DeepSeek-V2 and V3) has since become a standard for new MoE architectures, though it requires specialized kernel implementations to realize its theoretical benefits in production.

Production MoE serving adds routing at each layer, not just at selected layers. For a 60-layer model with MoE at every layer, that is 60 routing decisions per token. The router itself is small – typically a single linear projection – but the cumulative latency adds up, especially when tokens require cross-device communication.

Dense vs MoE Comparison

When to Choose Dense vs MoE

Choose a dense model when your workload prioritizes low TTFT and predictable latency. Dense models have fewer moving parts, no routing overhead, and simpler deployment. If you are running fine-tuned variants for different clients, dense models with LoRA are dramatically easier to manage. For single-node deployment or edge inference, dense is the only practical choice.

Choose a MoE model when throughput and cost efficiency are the primary drivers. If you are processing millions of tokens per day for an API or internal service, the compute savings compound rapidly. MoE models handle large context windows more efficiently per token. They also scale better to multi-GPU clusters where expert parallelism maps naturally across devices.

A practical rule: if your peak batch size stays below 16, dense models typically perform better. At batch sizes above 32, MoE begins to show throughput advantages. The crossover point depends on expert count, routing strategy, and hardware topology.

Glossary

References

1. Core Transformer Architecture
   1. Attention Is All You Need (The Transformer architecture). arXiv:1706.03762
2. Mixture of Experts and Routing
   1. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (Original MoE formulation). arXiv:1701.06538
   2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961
   3. Mixture-of-Experts with Expert Choice Routing. arXiv:2202.09368
   4. Mixtral of Experts (MoE architecture in practice). arXiv:2401.04088
   5. DeepSeek-V3 Technical Report (MoE with auxiliary-loss-free balancing, 256 experts, top-8 routing). arXiv:2412.19437
   6. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence (V4 preview, Sqrt(Softplus) gating, 1M context). arXiv:2606.19348
3. Inference Serving and Continuous Batching
   1. Orca: A Distributed Serving System for Transformer-Based Generative Models (Iteration-Level / Continuous Batching). USENIX OSDI 2022
4. Model Fine-Tuning
   1. LoRA: Low-Rank Adaptation of Large Language Models (Low-Rank Adaptation). arXiv:2106.09685
   2. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314
5. Models and Architecture References
   1. Introducing Llama 3.1 (8B, 70B, 405B dense models). Meta AI Blog
   2. Mixtral 8x7B Model Card (46.7B total, 12.9B active, top-2 MoE). Mistral Docs
   3. Mistral Large 3 (675B total, 41B active, granular MoE). Mistral Docs
   4. Qwen3 Technical Report. arXiv:2505.09388
   5. Gemma 3 Technical Report (dense only, 1B–27B, multimodal). arXiv:2503.19786
   6. Gemma 4 Technical Report (dense and MoE architectures). arXiv:2607.02770
6. Hardware
   1. NVIDIA Hopper Architecture In-Depth (H100, FP8 Tensor Cores, 3.35 TB/s HBM3). NVIDIA Resources
   2. NVIDIA H200 Tensor Core GPU Datasheet (141 GB HBM3e, 4.8 TB/s). NVIDIA SC23 Datasheet

Next: Part 2 — Quantization and KV Cache covers how to reduce model memory footprint and cache pressure.