Local Inference Deep Dive: Choosing Between Ollama, vLLM, and llama.cpp for Workstations and Servers

Deploying open-weights language models (such as Llama 3.3 70B, DeepSeek-R1, or Qwen 2.5 72B) on local hardware requires navigating a complex trade-off space between latency, memory efficiency, concurrent throughput, and operational complexity. Three systems dominate the current ecosystem:

• Ollama: An ergonomic, Go-based daemon wrapping llama.cpp that packages weights, system prompts, and Modelfiles into Docker-like images for rapid developer deployment.
• llama.cpp: Georgi Gerganov’s pure C/C++ inference engine, hand-optimized with zero external runtime dependencies, providing world-class CPU, Metal, and mixed CPU-GPU execution.
• vLLM: UC Berkeley’s high-throughput Python/C++ serving engine designed for enterprise data centers, implementing PagedAttention and continuous iteration-level batching.

1. The Memory Wall & Roofline Model of LLM Inference

To evaluate these engines, one must understand why token generation is fundamentally constrained by memory bandwidth rather than raw compute FLOPS. The autoregressive generation phase decodes one token at a time:

Consider an FP16 70B parameter model consuming approximately 140 GB of VRAM. An NVIDIA H100 GPU provides 3,350 GB/s of HBM3 memory bandwidth. For a single concurrent user (batch size = 1), generating each new token requires transferring all 140 GB of weights from high-bandwidth memory into GPU SRAM registers:

Regardless of whether the GPU has 1,000 TFLOPS or 10,000 TFLOPS of compute capability, the compute cores spend over 90% of their clock cycles idle, waiting for weight bytes to arrive across the memory bus. This physical reality drives the two foundational optimizations of modern inference engines: weight quantization (reducing model footprint) and concurrent batching (amortizing weight reads across multiple users).

2. Quantization Formats: GGUF vs. AWQ vs. GPTQ

Quantization compresses floating-point weights (FP16 or BF16, 16 bits per parameter) down to 8-bit, 4-bit, or even 2-bit representations. However, the runtime engine determines which format is viable:

GGUF (GPT-Generated Unified Format)

Developed specifically for llama.cpp and adopted by Ollama, GGUF stores both model metadata, tokenizer vocabulary, and quantized tensor arrays in a single, contiguous binary file. It excels at:

• Layer Offloading: Distributing layers between CPU RAM and GPU VRAM (e.g., placing 28 layers on an 8GB GPU and the remaining 12 layers on system RAM).
• K-Quants: Employing non-uniform quantization blocks (e.g., Q4_K_M, where attention matrices receive 6-bit precision while feed-forward weights receive 4-bit precision to maintain reasoning accuracy).

AWQ (Activation-aware Weight Quantization)

AWQ observes that not all weights are equally important; protecting the top 1% of salient weights based on activation magnitudes eliminates perplexity degradation in 4-bit models. AWQ is specifically optimized for NVIDIA Tensor Cores. Because it requires zero CPU layer splitting, engines like vLLM and SGLang use AWQ to achieve maximum memory bandwidth utilization on server-class GPUs.

3. Memory Management: PagedAttention & KV Cache Fragmentation

Beyond model weights, the Key-Value (KV) cache stores historical attention state to avoid recalculating past tokens at each step. In naive serving systems, KV cache allocation causes catastrophic memory waste:

vLLM’s PagedAttention resolves this by borrowing the virtual memory paging architecture from operating systems. It divides the KV cache into fixed-size physical memory blocks (typically 16 tokens per block). Non-contiguous physical blocks are mapped dynamically through a page table as the sequence generates tokens, reducing memory waste to less than 4% and increasing serving capacity by 2x to 4x on the same hardware.

4. Batching Dynamics: Iteration-Level Continuous Batching

The single most dramatic architectural difference between developer tools (Ollama) and production engines (vLLM) lies in request scheduling:

5. Hardware Acceleration: Apple Silicon Metal vs. NVIDIA CUDA

Choosing an inference engine is largely dictated by your underlying silicon architecture:

Apple Silicon (M1/M2/M3/M4 Max & Ultra)

Mac workstations feature a Unified Memory Architecture (UMA) where CPU, GPU, and Neural Engine share up to 128 GB or 192 GB of memory with bandwidths exceeding 800 GB/s. llama.cpp and Ollama compile directly to Apple Metal Performance Shaders (MPS), enabling engineers to load massive 70B parameter models entirely into local VRAM at a fraction of server GPU costs.

NVIDIA CUDA & Data Center Clusters

NVIDIA architectures leverage specialized hardware features: FP8 tensor cores, asynchronous copy instructions, NVLink multi-GPU fabrics, and FlashAttention-3 kernels. vLLM takes full advantage of these primitives via custom CUDA/C++ kernels, delivering up to 10x higher aggregate tokens-per-second than CPU-based engines under multi-tenant enterprise loads.

6. Comprehensive Architectural Matrix

7. Production Deployment Recipes

Recipe A: Deploying High-Throughput vLLM on NVIDIA RTX / Data Center

# Run vLLM with AWQ 4-bit quantization and PagedAttention prefix caching
docker run --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model casperhansen/deepseek-coder-33b-instruct-awq \
    --quantization awq \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching

Recipe B: Running llama.cpp Native Server on Apple Silicon

# Clone and build with Apple Metal acceleration
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && cmake -B build -DGGML_METAL=ON && cmake --build build --config Release

# Start OpenAI-compatible server on port 8080
./build/bin/llama-server \
    -m ./models/qwen2.5-coder-32b-instruct-q4_k_m.gguf \
    -c 32768 \
    -ngl 99 \
    --port 8080 \
    --host 0.0.0.0