SGLang

SGLang (Structured Generation Language) is an open-source runtime for serving large language models at high throughput and low latency. Introduced through LMSYS in January 2024, it pairs a general serving engine with a structured-output programming language, and its RadixAttention cache design has made it one of the two dominant open-source inference engines alongside vLLM.^[1][2]

History

SGLang was developed in UC Berkeley's Sky Computing Lab and released via LMSYS — the team behind Chatbot Arena — in January 2024. Its original paper reported up to 6.4× higher throughput than Guided Generation, vLLM and LMQL on workloads with heavy prefix sharing, and roughly 1.7× lower time-to-first-token during a month-long production deployment on Vicuna-33B.^[3]

By 2026 SGLang had become a mainstream production engine, adopted by major AI clouds and hardware vendors, and the standard comparison point for vLLM in enterprise architecture decisions.^[1][2]

Key Concepts and Technology

The central problem SGLang addresses is the KV cache — the per-token memory of attention states that dominates inference cost. Agents and chat applications resend the same system prompt, tool definitions and growing conversation history on every call, so recomputing shared prefixes wastes GPU memory and time.^[2]

RadixAttention stores computed KV states in a radix tree keyed by token prefix: when a new request arrives, the engine reuses every matching branch and computes only the new suffix, evicting least-recently-used branches under memory pressure. It is paired with cache-aware scheduling, which runs requests sharing a prefix close together while the cache is still warm.^[2] In production, RadixAttention achieved cache hit rates of 52–74% on real chat traffic.^[3]

SGLang also ships paged attention, a DSL for constrained and structured generation, and OpenAI-compatible APIs. Independent benchmarks show throughput within about 5% of vLLM on fully unique prompts, but 20–40% lower time-to-first-token on prefix-heavy agent and RAG workloads — meaning the engine choice is driven by workload shape, not raw speed.^[2][4]

Applications and Impact

SGLang is used to serve open-weight models behind chat products, RAG pipelines, agent frameworks and structured-extraction services, often deployed on single-GPU boxes or multi-node clusters with vLLM-style orchestration. Its success reinforced prefix reuse as a standard feature — engines that lack it now treat it as table stakes — and made cache-hit rate a routine operational metric for AI teams.^[2][3]

>See Also

• Inference
• vLLM
• KV Cache
• Open-Weight Models
• Model Serving
• Malaysian AI Startups