In interactive applications—such as voice agents, IDE autocomplete, and real-time trading assistants—Time to First Token (TTFT) is the primary user experience bottleneck. Routing every user prompt back to a single centralized hyperscaler region introduces an unavoidable $100\text{ ms}$ to $250\text{ ms}$ network RTT penalty before the model generates its first token.
Disaggregated Prefill and Decode (Split-Architecture)
Modern inference clusters decouple the two distinct phases of LLM generation:
- Phase 1: Prefill (Compute-Bound): Ingests the large prompt (e.g., 8,000 tokens of system prompts, code files, and conversation history) in a single parallel GEMM matrix multiplication. Executed on high-compute GPU nodes at edge PoPs.
- Phase 2: Decode (Memory-Bandwidth Bound): Generates subsequent output tokens sequentially ($1\text{ token per step}$), dominated by HBM memory bandwidth.
Anycast BGP Steering & Session Stickiness
Using Anycast BGP routing, global clients announce to a single IP address that routes to the nearest Internet Exchange Point (IXP). Once a session is established, consistent hashing maps subsequent requests to the specific GPU worker holding that conversation's KV-cache, avoiding expensive cache eviction and prompt re-computation.
# Consistent Hash Ring for Edge KV-Cache Stickiness in Python
import hashlib
import bisect
class KVCacheHashRing:
def __init__(self, nodes: list, replicas: int = 100):
self.replicas = replicas
self.ring = dict()
self.sorted_keys = []
for node in nodes:
for i in range(replicas):
h = int(hashlib.md5(f"{node}:{i}".encode()).hexdigest(), 16)
self.ring[h] = node
self.sorted_keys.append(h)
self.sorted_keys.sort()
def get_node(self, session_id: str) -> str:
h = int(hashlib.md5(session_id.encode()).hexdigest(), 16)
idx = bisect.bisect_right(self.sorted_keys, h)
if idx == len(self.sorted_keys):
idx = 0
return self.ring[self.sorted_keys[idx]]