Open-Source Serving & Local Inference Lab

Deploying frontier open-source large language models requires navigating a trade-off surface spanning raw memory throughput, dynamic batching scheduling, multi-turn prefix reuse, and runtime dependency weight.

In early transformer serving architectures, model weights and Key-Value (KV) activation tensors were statically allocated in contiguous physical GPU memory. Because output token sequences have nondeterministic lengths bounded only by maximum generation limits ($L_{\text{max}}$), runtimes were forced to pre-allocate memory buffers sized to the worst-case context window. This resulted in internal fragmentation rates exceeding $60\%$ to $80\%$, severely throttling batch concurrency. Modern serving engines address this bottleneck through distinct memory virtualization abstractions.

Pioneered by researchers at UC Berkeley, vLLM re-engineers transformer memory management by adapting the classical operating system virtual memory page table to GPU DRAM.

The Virtual Page Table for KV Cache

Instead of requiring contiguous physical memory allocations, PagedAttention partitions the dynamic KV cache of each sequence into fixed-size physical blocks (typically $B = 16$ or $32$ tokens). A logical-to-physical block table maps incoming token representations into arbitrary non-contiguous physical DRAM pages:

During the attention kernel execution, the PagedAttention CUDA kernel fetches keys and values directly from non-contiguous physical blocks using the block table lookup, completely eliminating external fragmentation and driving GPU memory utilization above $96\%$.

Chunked Prefill & Continuous Batching

Traditional serving architectures suffer from severe tail latency when a large prompt prefill request arrives while multiple decode requests are running. The massive matrix multiplication of the prefill phase starves the memory bus, spiking Inter-Token Latency (ITL).

vLLM introduces Chunked Prefill (interleaved chunking), breaking long prompt sequences into fixed-size token slices (e.g. $512$ tokens). Prefill slices are batched concurrently with decode phase iterations, maintaining a balanced ratio between arithmetic intensity and memory bandwidth across every engine step.

# Production vLLM Deployment with Chunked Prefill & Tensor Parallelism
python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-72B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.95 \
    --max-model-len 32768 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --kv-cache-dtype fp8 \
    --port 8000

Developed at LMSYS and UC Berkeley, SGLang (Structured Generation Language) is engineered specifically for complex, multi-turn agentic workflows, few-shot prompting, and deterministic JSON extraction.

Radix Tree KV Cache Management

While vLLM utilizes hash-based lookups for prefix matching, SGLang structures the entire KV cache across all concurrent and historical requests as a unified Radix Tree (Trie). Each node in the tree represents a sequence of tokens with its corresponding precomputed KV tensors residing in GPU memory:

• Shared System Prompts: Multiple agent sessions sharing a 2,000-token system prompt hit the root node of the Radix Tree, executing with $0\text{ ms}$ prefill compute.
• Multi-Turn Conversation: When a user sends turn $N$, turns $1 \dots N-1$ already reside along the ancestor path in the tree, requiring prefill computation only for the delta tokens.
• Tree Search & MCTS: Parallel speculative rollouts, Monte Carlo Tree Search, and reflection loops branch off existing tree nodes without duplicating KV cache memory.
• LRU Eviction with Node Pruning: When GPU VRAM reaches threshold capacity, SGLang prunes leaf nodes using Least Recently Used (LRU) tracking, maintaining high-frequency prefixes indefinitely.

# Launching High-Throughput SGLang Server with RadixAttention
python -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V3 \
    --tp 8 \
    --trust-remote-code \
    --enable-radix-cache \
    --grammar-backend xgrammar \
    --port 30000

Authored by Georgi Gerganov, llama.cpp is a dependency-free, bare-metal C/C++ inference runtime designed to maximize hardware execution across consumer CPUs, Apple Silicon Metal, and NVIDIA/AMD GPUs.

The GGUF Binary Container Anatomy

Older quantization formats (such as GGML and raw PyTorch bin files) separated model weights from hyperparameter metadata and tokenizer configurations, creating frequent versioning incompatibilities. GGUF (GPT-Generated Unified Format) resolves this by packing everything into a single, self-describing binary:

• Magic Header: 4-byte ASCII signature GGUF followed by file version integer (v3).
• Metadata KV Store: Hierarchical key-value pairs specifying architecture name (e.g. llama, qwen2), context length, embedding dimensions, attention head count, and tokenizer merges.
• Tensor Information Index: Offset tables, data types, and byte lengths for every weight tensor in the network.
• Binary Tensor Data: 32-byte aligned tensor blocks optimized for memory mapping via the operating system's mmap() system call.

Zero-Copy Memory Mapping (`mmap`)

By aligning tensor byte boundaries, llama.cpp avoids reading gigabytes of weights into process heap memory on startup. Instead, mmap() maps the file directly into virtual address space. The OS kernel loads weight pages on demand directly from NVMe storage into DRAM, enabling sub-second cold starts and allowing multiple CLI processes to share the same physical memory pages.

Integer Quantization Kernels: K-Quants vs I-Quants

Ollama wraps the high-performance C++ compute primitives of llama.cpp into a developer-ergonomic daemon written in Go, providing container-like image management and local REST API serving.

Architecture of the Ollama Daemon

The Ollama daemon runs as a background service, dynamically managing GPU memory allocation and model swapping. When an inference request targets a model not currently in memory, Ollama inspects available VRAM, unloads idle weights, and offloads transformer layers onto the GPU via dynamic library bindings (libllama.so / libllama.dylib).

Declarative Modelfiles

Similar to Dockerfiles, Ollama allows developers to define model behavior, context lengths, stop sequences, and system instructions in a declarative file:

# Modelfile: Production Coding Specialist
FROM qwen2.5-coder:32b-instruct-q4_K_M

# Set execution parameters
PARAMETER temperature 0.2
PARAMETER top_p 0.95
PARAMETER num_ctx 32768
PARAMETER stop "<|im_end|>"
PARAMETER num_gpu 999

# Institutional System Directive
SYSTEM """
You are an expert systems engineer specializing in low-latency C++, CUDA kernels, and distributed AI serving.
Always provide mathematically rigorous derivations and complete, compile-ready code implementations.
"""

Quantization compresses high-precision floating point representations (FP16/BF16, 16 bits per parameter) down to lower bit-widths (8-bit, 4-bit, or 3-bit), reducing memory bus bandwidth requirements and unlocking local inference on consumer hardware.

Activation-Aware Weight Quantization (AWQ)

Uniform weight quantization treats all parameters equally, causing severe perplexity degradation at 4 bits. Researchers discovered that less than 1% of weights are responsible for model accuracy—specifically, weights corresponding to channels with large activation magnitudes.

AWQ protects these critical channels by finding an per-channel scaling factor $s$ that minimizes the output reconstruction error:

By scaling the salient weights up before rounding and dividing by $s$ during matrix multiplication, AWQ preserves critical activation channels without requiring mixed-precision hardware support.

GPTQ: Second-Order Hessian Compensation

GPTQ adapts the classical Optimal Brain Surgeon framework to layer-wise transformer quantization. For a weight matrix $W$ and input activations $X$, the rounding error of quantized column $w_q$ is compensated by updating the remaining unquantized weights:

By inverting the activation Hessian matrix using Cholesky decomposition with lazy updates, GPTQ achieves 4-bit quantization with minimal loss in perplexity in just hours of single-GPU calibration.

Accurately size model weights, KV cache requirements, and memory bandwidth throughput across consumer and datacenter GPUs before downloading weights.

Select your operational workload to receive architectural ratings and production-grade CLI start commands tailored for immediate deployment.