The Definitive Guide to Large Language Models (LLMs) in 2026

1. Transformer Architecture

The transformer, introduced in the landmark 2017 paper "Attention Is All You Need" by Vaswani et al., replaced recurrent and convolutional sequence-to-sequence models with a purely attention-based architecture. Every modern LLM - from GPT-4o to Llama 4 - is a descendant of this design. Understanding its internals is non-negotiable for anyone working seriously with these models.

Self-Attention: The Core Mechanism

Self-attention allows every token in a sequence to attend to every other token, computing a weighted sum of value vectors based on query-key compatibility. The canonical scaled dot-product attention formula is:

Where Q (queries), K (keys), and V (values) are linear projections of the input embeddings, and d_k is the dimensionality of the key vectors. The scaling factor 1/√dk prevents the dot products from growing too large in magnitude, which would push the softmax into regions with vanishingly small gradients.

Here's a minimal PyTorch implementation of scaled dot-product attention:

import torch
import torch.nn.functional as F

def scaled_dot_product_attention(Q, K, V, mask=None):
    """Scaled dot-product attention."""
    d_k = Q.size(-1)
    scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))
    weights = F.softmax(scores, dim=-1)
    return torch.matmul(weights, V), weights

The attention matrix has shape (seq_len, seq_len), giving it O(n²) memory and compute complexity - the fundamental bottleneck that drives most optimization research in the field.

Multi-Head Attention

Rather than computing a single attention function, multi-head attention runs h parallel attention heads, each with its own learned projection matrices. This allows the model to jointly attend to information from different representation subspaces at different positions:

class MultiHeadAttention(torch.nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.n_heads = n_heads
        self.d_k = d_model // n_heads
        self.W_q = torch.nn.Linear(d_model, d_model)
        self.W_k = torch.nn.Linear(d_model, d_model)
        self.W_v = torch.nn.Linear(d_model, d_model)
        self.W_o = torch.nn.Linear(d_model, d_model)

    def forward(self, x, mask=None):
        B, L, _ = x.shape
        Q = self.W_q(x).view(B, L, self.n_heads, self.d_k).transpose(1, 2)
        K = self.W_k(x).view(B, L, self.n_heads, self.d_k).transpose(1, 2)
        V = self.W_v(x).view(B, L, self.n_heads, self.d_k).transpose(1, 2)
        attn_out, _ = scaled_dot_product_attention(Q, K, V, mask)
        attn_out = attn_out.transpose(1, 2).contiguous().view(B, L, -1)
        return self.W_o(attn_out)

Modern models like Llama 3 use Grouped Query Attention (GQA), where multiple query heads share a single key-value head. This dramatically reduces the KV cache size (often by 4-8×) with minimal quality loss, making long-context inference practical.

KV Cache

During autoregressive generation, the model produces one token at a time. Without caching, it would need to recompute the key and value projections for all previous tokens at every step - O(n²) per token. The KV cache stores previously computed K and V tensors so that each new step only computes the projections for the latest token and appends them to the cache:

# Simplified KV cache during generation
cached_k, cached_v = None, None

for step in range(max_new_tokens):
    if cached_k is not None:
        # Only compute K, V for the new token
        new_k = W_k(new_token_embedding)
        new_v = W_v(new_token_embedding)
        cached_k = torch.cat([cached_k, new_k], dim=-2)
        cached_v = torch.cat([cached_v, new_v], dim=-2)
    else:
        cached_k = W_k(prompt_embeddings)
        cached_v = W_v(prompt_embeddings)
    # Attention uses full cached_k, cached_v but only new Q
    output = attention(Q_new, cached_k, cached_v)

The KV cache is the single largest consumer of GPU memory during inference. For a 70B parameter model with 128K context, the KV cache alone can exceed 40 GB in FP16. This is why techniques like GQA, quantized KV caches, and PagedAttention (discussed in Section 6) are critical.

Positional Encoding: RoPE and ALiBi

Transformers have no inherent notion of token order - attention is permutation-equivariant. Positional encodings inject sequence position information. Two dominant approaches have emerged:

Rotary Position Embeddings (RoPE), used by Llama, Qwen, Mistral, and most open-source models, encode position by rotating the query and key vectors in 2D subspaces. For a position m and dimension pair (2i, 2i+1):

def apply_rope(x, positions, dim):
    """Apply Rotary Position Embeddings."""
    freqs = 1.0 / (10000 ** (torch.arange(0, dim, 2).float() / dim))
    angles = positions.unsqueeze(-1) * freqs.unsqueeze(0)
    cos_vals = torch.cos(angles)
    sin_vals = torch.sin(angles)
    x1, x2 = x[..., ::2], x[..., 1::2]
    return torch.stack([x1 * cos_vals - x2 * sin_vals,
                        x1 * sin_vals + x2 * cos_vals], dim=-1).flatten(-2)

RoPE's key advantage is that the dot product between rotated queries and keys depends only on their relative position, enabling natural length extrapolation. Techniques like YaRN and NTK-aware scaling extend RoPE to context lengths far beyond training (e.g., training at 8K, inferring at 128K).

ALiBi (Attention with Linear Biases), used by models like BLOOM and MPT, takes a simpler approach: it adds a linear penalty to attention scores based on the distance between query and key positions. No learned parameters, no modifications to Q/K - just a static bias added to the attention matrix. ALiBi generalizes well to longer sequences but has largely been superseded by RoPE variants in 2025-2026 models.

Feed-Forward Layers

Each transformer block contains a position-wise feed-forward network (FFN) applied independently to each token. Modern LLMs use the SwiGLU activation (a gated variant of Swish), which consistently outperforms ReLU and GELU:

class SwiGLU_FFN(torch.nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.w1 = torch.nn.Linear(d_model, d_ff, bias=False)
        self.w2 = torch.nn.Linear(d_ff, d_model, bias=False)
        self.w3 = torch.nn.Linear(d_model, d_ff, bias=False)

    def forward(self, x):
        return self.w2(F.silu(self.w1(x)) * self.w3(x))

The FFN typically has a hidden dimension of 8/3 × d_model (rounded to a multiple of 256 for hardware efficiency). In a 70B model, the FFN parameters account for roughly two-thirds of total model weights.

Layer Normalization

Modern LLMs universally use RMSNorm (Root Mean Square Layer Normalization) instead of the original LayerNorm. RMSNorm drops the mean-centering step, keeping only the variance normalization, which is both faster and empirically equivalent:

class RMSNorm(torch.nn.Module):
    def __init__(self, dim, eps=1e-6):
        super().__init__()
        self.weight = torch.nn.Parameter(torch.ones(dim))
        self.eps = eps

    def forward(self, x):
        norm = torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
        return x * norm * self.weight

Placement matters: all modern architectures use Pre-Norm (normalize before attention and FFN) rather than Post-Norm, which improves training stability and allows deeper networks without careful learning rate tuning.

2. Model Families & Landscape (2026)

The LLM ecosystem in 2026 is a rich landscape of proprietary frontier models and increasingly capable open-weight alternatives. The gap between closed and open models has narrowed dramatically - open models now match or exceed GPT-4-class performance on many benchmarks. Here's the current state of play:

Proprietary Frontier Models

GPT-4o & GPT-4.5 (OpenAI) - GPT-4o unified text, vision, and audio into a single natively multimodal model. GPT-4.5, released in early 2026, pushed the frontier on creative writing, nuanced instruction following, and reduced hallucination rates. Both models feature 128K context windows and remain the default choice for many enterprise applications.

Claude 3.5 Sonnet & Claude 4 (Anthropic) - Claude 3.5 Sonnet became the go-to model for coding tasks in 2025, with exceptional instruction following and a 200K context window. Claude 4, launched Q1 2026, introduced improved agentic capabilities, stronger reasoning, and a 500K context window with near-perfect recall. Anthropic's Constitutional AI training continues to produce models with notably lower harmful output rates.

Gemini 2.0 (Google DeepMind) - Google's Gemini 2.0 family features native multimodality (text, image, video, audio, code) with a 2M token context window in the Pro variant. Gemini 2.0 Flash offers an exceptional quality-to-cost ratio and has become the default for high-volume applications where latency matters.

Open-Weight Models

Llama 3.1 / 3.2 / 4 (Meta) - Meta's Llama family democratized large-scale LLMs. Llama 3.1 405B was the first open model to genuinely compete with GPT-4. Llama 3.2 added multimodal vision capabilities. Llama 4 (2026) introduced a Mixture-of-Experts architecture, delivering frontier performance with dramatically lower inference costs.

Mistral Large & Medium (Mistral AI) - The French lab continues to punch above its weight. Mistral Large 2 rivals GPT-4o on reasoning benchmarks, while Mistral Medium offers an excellent balance of capability and efficiency for production workloads. Their models are popular in European enterprises due to EU data sovereignty compliance.

DeepSeek V3 (DeepSeek) - DeepSeek V3 stunned the community with its training efficiency - achieving GPT-4-class performance at a fraction of the compute budget using innovative MoE routing and FP8 mixed-precision training. The fully open model (including training code) has become a reference implementation for efficient large-scale training.

Qwen 2.5 (Alibaba) - Alibaba's Qwen 2.5 series spans 0.5B to 72B parameters with strong multilingual performance, particularly in CJK languages. The 72B variant is competitive with Llama 3.1 70B across most benchmarks and offers Apache 2.0 licensing.

Phi-4 (Microsoft) - Microsoft's "small but mighty" Phi series proves that data quality trumps raw scale. Phi-4 (14B parameters) matches or exceeds many 70B models on reasoning and coding benchmarks, making it ideal for edge deployment and resource-constrained environments.

Command R+ (Cohere) - Purpose-built for enterprise RAG workloads, Command R+ excels at grounded generation with citation, multilingual retrieval, and tool use. It's the go-to choice for companies building production RAG systems.

Model Comparison Table

Key trend: Mixture-of-Experts (MoE) has become the dominant architecture for frontier models. By activating only a fraction of total parameters per token (e.g., DeepSeek V3 activates 37B of 671B), MoE models achieve superior quality at dramatically lower inference cost. The tradeoff is higher total memory footprint - you need enough VRAM to hold all expert weights even though only a subset is used per forward pass.

3. Tokenization

Tokenization is the process of converting raw text into the integer sequences that models actually process. It's deceptively important - your tokenizer choice directly affects model quality, inference cost, multilingual performance, and even the types of tasks your model can handle well.

Byte Pair Encoding (BPE)

BPE is the dominant tokenization algorithm. It starts with individual bytes (or characters) and iteratively merges the most frequent adjacent pair into a new token. After training, you get a vocabulary of subword units that efficiently represent common patterns while gracefully handling rare words by decomposing them into smaller pieces.

The algorithm:

1. Start with a vocabulary of individual characters/bytes
2. Count all adjacent pairs in the training corpus
3. Merge the most frequent pair into a new token
4. Repeat until the desired vocabulary size is reached (typically 32K-128K tokens)

GPT-4 uses a BPE tokenizer with ~100K tokens. Llama models use SentencePiece BPE with 32K-128K tokens. Larger vocabularies improve compression (fewer tokens per text) but increase embedding table size.

SentencePiece

SentencePiece (Google) is a language-agnostic tokenizer that operates directly on raw Unicode text without pre-tokenization rules. It supports both BPE and Unigram algorithms. Most open-source LLMs (Llama, Mistral, Qwen) use SentencePiece because it handles multilingual text cleanly without language-specific preprocessing.

tiktoken

tiktoken is OpenAI's fast BPE tokenizer implementation, written in Rust with Python bindings. It's the standard tool for counting tokens and estimating API costs for OpenAI models:

import tiktoken

# Load the tokenizer for GPT-4o
enc = tiktoken.encoding_for_model("gpt-4o")

text = "Large Language Models are transforming software engineering."
tokens = enc.encode(text)

print(f"Text: {text}")
print(f"Token count: {len(tokens)}")
print(f"Tokens: {tokens}")
print(f"Decoded tokens: {[enc.decode([t]) for t in tokens]}")

# Output:
# Text: Large Language Models are transforming software engineering.
# Token count: 7
# Tokens: [35353, 11688, 27972, 553, 46648, 6891, 17tried.]
# Decoded tokens: ['Large', ' Language', ' Models', ' are', ' transforming',
#                   ' software', ' engineering.']

Token Counting for Cost Estimation

Since API providers charge per token, accurate counting is essential for budgeting:

import tiktoken

def estimate_cost(text, model="gpt-4o",
                  input_price_per_m=2.50, output_price_per_m=10.00):
    """Estimate API cost for a given text input."""
    enc = tiktoken.encoding_for_model(model)
    input_tokens = len(enc.encode(text))
    input_cost = (input_tokens / 1_000_000) * input_price_per_m
    # Estimate output as ~1.5x input for conversational use
    est_output_tokens = int(input_tokens * 1.5)
    output_cost = (est_output_tokens / 1_000_000) * output_price_per_m
    return {
        "input_tokens": input_tokens,
        "est_output_tokens": est_output_tokens,
        "input_cost": f"${input_cost:.6f}",
        "output_cost": f"${output_cost:.6f}",
        "total_est_cost": f"${input_cost + output_cost:.6f}"
    }

# Example: estimate cost for a code review prompt
with open("my_code.py") as f:
    code = f.read()
prompt = f"Review this code for bugs and security issues:\n\n{code}"
print(estimate_cost(prompt))

How Tokenization Affects Performance

Fertility rate (tokens per word) varies dramatically across languages. English averages ~1.3 tokens/word with GPT-4's tokenizer, but languages like Japanese, Thai, or Burmese can exceed 3-4 tokens/word. This means non-English users pay more per word and consume context window faster. Models like Qwen 2.5 and Gemma 2 address this with larger, more multilingual vocabularies.

Code tokenization is another pain point. Whitespace-heavy languages (Python) and verbose languages (Java) tokenize less efficiently than terse ones (Rust, Go). A 100-line Python file might consume 500 tokens while the equivalent Rust code uses 350. This matters when you're stuffing code into context windows for AI-assisted development.

4. Fine-Tuning

Fine-tuning adapts a pre-trained LLM to your specific domain, format, or task. The landscape of fine-tuning techniques has matured significantly, with parameter-efficient methods making it possible to customize 70B+ models on a single consumer GPU.

Full Fine-Tuning

Full fine-tuning updates all model parameters. It offers the highest potential quality but requires enormous compute: fine-tuning a 70B model needs 4-8× A100 80GB GPUs, costs thousands of dollars, and risks catastrophic forgetting if not done carefully. In 2026, full fine-tuning is reserved for organizations training specialized foundation models or when maximum quality justifies the cost.

LoRA (Low-Rank Adaptation)

LoRA freezes the pre-trained weights and injects small trainable low-rank matrices into each attention layer. Instead of updating a weight matrix W ∈ ℝ^d×d, LoRA learns two small matrices A ∈ ℝ^d×r and B ∈ ℝ^r×d where r ≪ d (typically 8-64). The effective weight becomes W + BA, adding only 0.1-1% trainable parameters.

Benefits: 10-100× less memory, training on a single GPU, easy adapter swapping (serve one base model with multiple LoRA adapters for different tasks), and no inference latency overhead when merged.

QLoRA (Quantized LoRA)

QLoRA combines 4-bit quantization of the base model with LoRA adapters trained in higher precision. This allows fine-tuning a 70B model on a single 48GB GPU (A6000 or A40) - a game-changer for accessibility. The key innovations are NF4 (Normal Float 4-bit) quantization and double quantization of quantization constants.

Here's a complete QLoRA fine-tuning example using Hugging Face PEFT and TRL:

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# 1. Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# 2. Load model and tokenizer
model_id = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

# 3. Prepare model for QLoRA training
model = prepare_model_for_kbit_training(model)

# 4. Configure LoRA adapters
lora_config = LoraConfig(
    r=16,                          # Rank - higher = more capacity
    lora_alpha=32,                 # Scaling factor (alpha/r = scaling)
    target_modules=[               # Which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Output: trainable params: 41,943,040 || all params: 8,072,204,288 || 0.52%

# 5. Load and format dataset
dataset = load_dataset("your-org/your-dataset", split="train")

def format_chat(example):
    """Format into Llama 3 chat template."""
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": example["instruction"]},
        {"role": "assistant", "content": example["response"]},
    ]
    return {"text": tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=False
    )}

dataset = dataset.map(format_chat)

# 6. Training arguments
training_args = TrainingArguments(
    output_dir="./qlora-output",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # Effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    optim="paged_adamw_8bit",       # Memory-efficient optimizer
    gradient_checkpointing=True,    # Trade compute for memory
    max_grad_norm=0.3,
)

# 7. Train with SFTTrainer
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
    dataset_text_field="text",
    max_seq_length=2048,
    packing=True,                   # Pack short examples together
)
trainer.train()

# 8. Save the adapter (not the full model)
trainer.save_model("./qlora-adapter")

# 9. Merge adapter into base model for deployment
from peft import AutoPeftModelForCausalLM
merged = AutoPeftModelForCausalLM.from_pretrained(
    "./qlora-adapter", device_map="auto", torch_dtype=torch.bfloat16
)
merged = merged.merge_and_unload()
merged.save_pretrained("./merged-model")

Fine-Tuning vs. RAG: When to Use Each

The pragmatic answer: Use RAG first. It's faster to set up, easier to debug, and handles most knowledge-grounding use cases. Fine-tune when you need the model to adopt a specific output format, tone, or reasoning pattern that prompting alone can't achieve. The best production systems often combine both - a fine-tuned model that's also RAG-augmented.

5. Quantization

Quantization reduces model precision from FP16/BF16 (16-bit) to lower bit widths (8-bit, 4-bit, even 2-bit), dramatically shrinking memory footprint and improving inference throughput. It's the single most impactful technique for making large models practical on real hardware.

Quantization Formats

GGUF (GPT-Generated Unified Format) - The standard format for CPU and hybrid CPU/GPU inference via llama.cpp and Ollama. GGUF files are self-contained (model weights + metadata + tokenizer) and support a rich set of quantization schemes (Q2_K through Q8_0). GGUF is the format you'll encounter most often when running models locally.

GPTQ (GPT Quantization) - A post-training quantization method that uses calibration data to minimize quantization error layer by layer. GPTQ produces GPU-optimized 4-bit and 8-bit models with excellent quality retention. It requires a one-time quantization step with a calibration dataset (~128 samples) but produces models that run efficiently with libraries like AutoGPTQ and ExLlamaV2.

AWQ (Activation-Aware Weight Quantization) - AWQ identifies the most "salient" weight channels (those that process large activation magnitudes) and preserves their precision while aggressively quantizing less important channels. This produces 4-bit models that often outperform GPTQ at the same bit width. AWQ is increasingly the default choice for GPU-based 4-bit inference.

bitsandbytes - Hugging Face's integration for on-the-fly quantization. Load any model in 4-bit (NF4) or 8-bit (INT8) with a single config flag. No pre-quantization step needed. Primarily used for QLoRA training and quick experimentation rather than production serving.

# Load a model in 4-bit with bitsandbytes - one line change
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-70B-Instruct",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    device_map="auto",
)

Quantization Quality Comparison

Using a 70B parameter model as reference (Llama 3.1 70B):

The sweet spot: Q4_K_M is the most popular quantization level for local inference. It offers a 3.5× size reduction with only 3-4% quality degradation - imperceptible for most conversational and coding tasks. Q5_K_M is preferred when you have the VRAM headroom and want near-lossless quality. Q8_0 is effectively lossless but only halves the memory requirement.

When quantization hurts: Tasks requiring precise numerical reasoning, complex multi-step logic, or nuanced creative writing show the most degradation at lower bit widths. If your use case involves these, benchmark carefully before deploying below Q5_K_M.

6. Inference Optimization

Serving LLMs at scale is an engineering challenge distinct from training. The goal: maximize throughput (tokens/second across all users) while minimizing latency (time-to-first-token and inter-token latency) and cost ($/million tokens). Here are the key techniques powering production LLM serving in 2026.

KV Cache Management

As discussed in Section 1, the KV cache stores previously computed key-value pairs to avoid redundant computation during autoregressive generation. The challenge is that KV cache memory grows linearly with sequence length and batch size. For a 70B model serving 32 concurrent users at 4K context each, the KV cache alone consumes ~20 GB.

Optimization strategies include: quantizing the KV cache to INT8 or FP8 (halving its memory with negligible quality loss), sliding window attention (Mistral's approach - only cache the last N tokens), and PagedAttention (see below).

PagedAttention (vLLM)

PagedAttention, introduced by the vLLM project, applies operating system virtual memory concepts to KV cache management. Instead of pre-allocating contiguous memory for each sequence's maximum possible length, PagedAttention allocates memory in fixed-size "pages" (blocks) on demand. This eliminates memory fragmentation and waste, improving GPU memory utilization from ~50% to ~95%.

The result: vLLM can serve 2-4× more concurrent requests than naive implementations on the same hardware.

Continuous Batching

Traditional static batching waits for all sequences in a batch to finish before starting new ones. Continuous batching (also called "iteration-level scheduling") immediately fills empty batch slots as sequences complete. This keeps the GPU saturated and dramatically improves throughput for workloads with variable output lengths.

Speculative Decoding

Speculative decoding uses a small "draft" model to generate candidate tokens quickly, then verifies them in parallel with the large "target" model. Since verification is a single forward pass (not autoregressive), it's much faster than generating each token individually. If the draft model's predictions match the target model's, you get multiple tokens for the cost of one forward pass.

Typical speedup: 2-3× for code generation and structured output, where the draft model can predict many tokens correctly. Less effective for creative or highly unpredictable text.

Flash Attention 2

Flash Attention 2 (Tri Dao, 2023) is a memory-efficient attention implementation that avoids materializing the full O(n²) attention matrix. By tiling the computation and keeping intermediate results in GPU SRAM (fast on-chip memory) rather than HBM (slow off-chip memory), Flash Attention 2 achieves 2-4× speedup over standard attention with significantly lower memory usage.

Flash Attention 2 is now the default in virtually all serving frameworks. Flash Attention 3 (2025) adds FP8 support and further optimizations for Hopper GPUs (H100/H200).

vLLM Serving Example

vLLM is the most popular open-source LLM serving framework, combining PagedAttention, continuous batching, and optimized CUDA kernels:

# Install: pip install vllm

# === Option 1: Python API ===
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    tensor_parallel_size=1,        # Number of GPUs
    gpu_memory_utilization=0.90,   # Use 90% of GPU memory
    max_model_len=8192,
    enable_prefix_caching=True,    # Cache common prefixes (system prompts)
)

params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=1024,
    stop=["<|eot_id|>"],
)

prompts = [
    "Explain the difference between TCP and UDP in one paragraph.",
    "Write a Python function to find the longest palindromic substring.",
]
outputs = llm.generate(prompts, params)

for output in outputs:
    print(f"Prompt: {output.prompt[:50]}...")
    print(f"Output: {output.outputs[0].text}\n")

# === Option 2: OpenAI-compatible API server ===
# Start the server:
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --enable-prefix-caching \
    --api-key your-secret-key

# Query it with any OpenAI-compatible client:
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer your-secret-key" \
    -d '{
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "What is PagedAttention?"}
        ],
        "temperature": 0.7,
        "max_tokens": 512
    }'

# === Option 3: Use with OpenAI Python SDK ===
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="your-secret-key")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain continuous batching."},
    ],
    temperature=0.7,
    max_tokens=512,
)
print(response.choices[0].message.content)

vLLM's OpenAI-compatible API means you can swap between self-hosted and cloud models by changing a single base URL - no code changes required.

7. Deployment Options

Choosing the right serving stack depends on your scale, hardware, latency requirements, and operational complexity budget. Here's a practical breakdown of the major options:

vLLM

Best for: Production GPU serving at scale. vLLM delivers the highest throughput for GPU-based serving thanks to PagedAttention and continuous batching. It supports tensor parallelism across multiple GPUs, quantized models (AWQ, GPTQ, bitsandbytes), and an OpenAI-compatible API out of the box. Use vLLM when you're serving models on dedicated GPU infrastructure and need maximum requests/second.

Hugging Face Text Generation Inference (TGI)

Best for: Hugging Face ecosystem integration and managed deployment. TGI is a Rust-based serving framework with built-in support for Flash Attention, continuous batching, quantization, and token streaming. It's the engine behind Hugging Face's Inference Endpoints. Choose TGI when you want tight integration with the Hugging Face Hub or are deploying via their managed infrastructure.

# Deploy TGI with Docker
docker run --gpus all --shm-size 1g -p 8080:80 \
    -v /data:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id meta-llama/Llama-3.1-8B-Instruct \
    --quantize awq \
    --max-input-tokens 4096 \
    --max-total-tokens 8192 \
    --max-batch-prefill-tokens 4096

Ollama

Best for: Local development and experimentation. Ollama wraps llama.cpp in a user-friendly CLI with a model registry, automatic GGUF downloading, and a REST API. It's the fastest path from zero to running a model locally. Not designed for production multi-user serving, but excellent for development, testing prompts, and running models on laptops.

# Install and run a model in seconds
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b

# Or use the API
curl http://localhost:11434/api/chat -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
}'

llama.cpp

Best for: CPU inference, edge devices, and maximum hardware compatibility. llama.cpp is a pure C/C++ implementation that runs GGUF models on CPUs, Apple Silicon (Metal), NVIDIA GPUs (CUDA), AMD GPUs (ROCm), and even Vulkan-capable devices. It's the foundation that Ollama, LM Studio, and many other tools build on. Use llama.cpp directly when you need fine-grained control over inference parameters or are deploying to unusual hardware.

# Build and run llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make -j LLAMA_CUDA=1

# Run with the built-in server
./llama-server \
    -m models/llama-3.1-8b-instruct-q4_k_m.gguf \
    --host 0.0.0.0 --port 8080 \
    -c 8192 -ngl 99 --flash-attn

TensorRT-LLM (NVIDIA)

Best for: Maximum single-stream latency on NVIDIA GPUs. TensorRT-LLM compiles models into optimized CUDA kernels with operator fusion, FP8/INT8 quantization, and hardware-specific tuning. It delivers the lowest latency per request but requires NVIDIA GPUs and a more complex build/deployment pipeline. Use it when you're running on NVIDIA hardware and every millisecond of latency matters (real-time applications, interactive agents).

Deployment Decision Matrix

8. Evaluation & Benchmarks

Benchmarks are essential for comparing models, but no single benchmark tells the whole story. Understanding what each measures - and what it doesn't - is critical for making informed model selection decisions.

MMLU (Massive Multitask Language Understanding)

MMLU tests knowledge across 57 academic subjects (STEM, humanities, social sciences, professional domains) using multiple-choice questions. It's the most widely cited general knowledge benchmark. Scores above 85% indicate strong general knowledge; frontier models now exceed 90%. Limitation: Multiple-choice format doesn't test generation quality, and the test set has known labeling errors (~4% of questions).

HumanEval & MBPP

HumanEval (164 problems) and MBPP (974 problems) test code generation by asking models to write Python functions that pass unit tests. The metric is pass@k - the probability that at least one of k generated solutions passes all tests. HumanEval+ and EvalPlus extend these with additional test cases to catch false positives. Limitation: Only tests Python, only tests function-level generation, and problems are relatively simple compared to real-world software engineering.

MT-Bench

MT-Bench evaluates multi-turn conversational ability across 8 categories (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities). An LLM judge (typically GPT-4) scores responses on a 1-10 scale. It's the best available proxy for "how good does this model feel in conversation." Limitation: LLM-as-judge introduces bias toward the judge model's preferences and writing style.

Chatbot Arena ELO

LMSYS Chatbot Arena uses blind head-to-head comparisons where humans choose which of two anonymous model responses they prefer. ELO ratings are computed from thousands of these pairwise comparisons. This is widely considered the most reliable overall quality ranking because it reflects real user preferences on real prompts. Limitation: Biased toward English, conversational use cases, and the demographics of Arena users.

Evaluating for Your Use Case

Public benchmarks are a starting point, not a destination. For production model selection:

1. Build a custom eval set - Collect 100-500 representative examples from your actual use case. Include edge cases and failure modes you care about.
2. Define clear metrics - Accuracy, format compliance, latency, cost per request. Weight them according to your priorities.
3. Use LLM-as-judge for subjective quality - Have a strong model (Claude 4, GPT-4o) score outputs on rubrics you define. Validate against human judgments on a subset.
4. A/B test in production - Benchmarks predict production performance imperfectly. Run candidate models on real traffic and measure user-facing metrics (task completion rate, user satisfaction, error rate).

# Simple LLM-as-judge evaluation framework
from openai import OpenAI

client = OpenAI()

def evaluate_response(question, response, rubric):
    """Use GPT-4o as a judge to score a model response."""
    judge_prompt = f"""Score the following response on a scale of 1-10.

Rubric: {rubric}

Question: {question}
Response: {response}

Provide your score as a JSON object: {{"score": N, "reasoning": "..."}}"""

    result = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": judge_prompt}],
        response_format={"type": "json_object"},
    )
    return result.choices[0].message.content

# Evaluate across your test set
rubric = "Accuracy, completeness, and clarity for a technical audience."
for example in test_set:
    score = evaluate_response(example["question"], example["response"], rubric)
    print(score)

9. Cost Analysis

API pricing varies dramatically across providers and models. Understanding the cost structure is essential for budgeting and architecture decisions. All prices below are per million tokens as of April 2026.

API Pricing Comparison

Note: Prices change frequently. Check provider pricing pages for current rates. Cached/batch pricing can reduce costs by 50% or more.

Cost Optimization Strategies

1. Prompt caching: Both Anthropic and OpenAI offer prompt caching that reduces input costs by 50-90% for repeated prefixes (system prompts, few-shot examples). If your system prompt is 2K tokens and you make 1M requests/month, caching saves thousands of dollars.

2. Model routing: Use a cheap, fast model (GPT-4o mini, Gemini Flash, Haiku) for simple queries and route complex ones to a frontier model. A well-tuned router can handle 70-80% of traffic with the cheap model, cutting average cost by 5-10×.

3. Batch API: OpenAI's Batch API offers 50% discount for non-real-time workloads (processing completes within 24 hours). Ideal for evaluation, data processing, and content generation pipelines.

4. Self-hosting breakeven: Self-hosting becomes cost-effective at roughly 50M+ tokens/day for a 70B model. Below that, API pricing is usually cheaper when you factor in GPU rental, ops overhead, and engineering time. The math changes if you need data sovereignty or have existing GPU infrastructure.

5. Output token optimization: Output tokens cost 2-5× more than input tokens. Instruct models to be concise, use structured output (JSON) to avoid verbose prose, and set appropriate max_tokens limits.