Last month a friend of mine at a small startup messaged me asking why his chatbot was crawling under load. GPU dashboard showed 60 percent utilization, users staring at loading spinners, and he had already tried bumping the instance size twice. Turns out the model was fine. The inference engine was the problem, and he was running the default setup he copied from a tutorial two years ago.
That’s basically the whole story of vLLM vs SGLang right now. Both are open source, both serve Hugging Face models, both give you an OpenAI compatible API you can swap in with one line of config. And yet picking the wrong one can quietly cost you 20 to 30 percent of your GPU budget every month. Nobody notices until the AWS bill shows up.
I’ve spent the last few hours running both engines for different projects, and I’m only going to tell you what actually mattered in practice, not what the GitHub README claims.
What These Two Things Even Are
vLLM came out of UC Berkeley and it’s been the default choice since around 2023. Its whole pitch is something called PagedAttention. Basically it treats your GPU’s KV cache (the memory that stores attention keys and values for every token you’ve generated so far) like an operating system treats virtual memory. Instead of reserving one big continuous chunk of memory per request and wasting most of it, it splits memory into small pages and hands them out on demand. Less fragmentation, more requests fit on the same GPU.
SGLang is newer, it came out of a research group connected to Berkeley too actually, and its headline feature is RadixAttention. The idea is: a lot of real traffic reuses the same prefix. Think of a chatbot with a long system prompt, or a RAG app where fifty users ask questions about the same uploaded PDF, or an agent that keeps replaying earlier steps of a conversation. RadixAttention caches that shared prefix in a tree structure and reuses it across requests instead of recomputing the same attention states again and again.
So one optimizes for “serve as many independent requests as possible.” The other optimizes for “a lot of these requests actually look alike, don’t redo the work.” That single design difference explains almost every benchmark number you’ll see floating around this year.
The Throughput Numbers (And Why They Keep Flipping)
Here’s the thing nobody tells you upfront: the benchmark you read last week is probably already a bit stale. Both projects ship new releases every few weeks, and the ranking flips depending on model size, prompt shape, and GPU. I saw one comparison from earlier this year running SGLang 0.5.15 against vLLM 0.25.1 on a single RTX 4090 with Qwen2.5–7B, and the gap between them was noticeably different from what a 70B model on H100s would show.
Broadly, though, a pattern does hold up across most of the write ups I’ve read (and I read a lot of them for this article, not gonna lie, some of the vendor blogs are basically ads with numbers attached):
At smaller model sizes with prefix heavy traffic, SGLang tends to pull ahead by a decent margin, sometimes 25 to 30 percent more tokens per second. This makes sense once you get the RadixAttention idea, since prefill cost is a bigger chunk of the total work when models are small and outputs are short, so caching that prefill pays off more.
At larger model sizes, say 70B and up, the gap shrinks to something like 3 to 5 percent. Doesn’t really matter which one you pick at that point, honestly.
For genuinely one-shot, unique prompts with no repeated structure, vLLM sometimes edges ahead slightly, because there’s nothing to cache and its batching and memory management are just very mature at this point. One test I saw had vLLM about 1.1x faster than SGLang on pure single-shot completions.
For multi-turn conversations where context keeps growing turn over turn, several teams have reported SGLang holding steady around 30 tok/s under high concurrency while vLLM’s throughput dropped from 22 down to 16 as cache pressure built up. That’s the kind of gap that actually shows up in your customer support bot’s response times, not just in a spreadsheet.
I’ll admit I was skeptical of the RadixAttention hype the first time I read the paper, it sounded like one of those “novel technique, ten percent better on our cherry picked benchmark” stories you see constantly in ML research. It’s not that. In prefix heavy production traffic the difference is real and you’ll feel it.
Latency, Not Just Raw Throughput
Throughput is the number everyone quotes, but if you’re building anything interactive, latency is what your users actually feel. Two numbers matter here: TTFT (time to first token) and ITL (inter token latency, basically how smooth the streaming feels once it starts).
SGLang generally shows lower TTFT at the 95th percentile, somewhere in the 5 to 8 percent range compared to vLLM, across most concurrency levels people have tested. That sounds small until you remember it compounds. If every user in a chat app waits 8 percent longer before the first word appears, and you have thousands of concurrent sessions, that adds up to a genuinely different feeling product.
vLLM isn’t bad here either, to be fair. For templated prompts, batch summarization jobs, anything where every request basically looks the same, its latency is completely fine and honestly a bit more predictable release over release, because the project has been stable for longer and there are fewer edge cases to hit.
Structured Output and Agentic Workloads
This is where SGLang really built its reputation, and it’s honestly the reason I started using it in the first place. If your app needs JSON schema enforcement, function calling with strict formats, or constrained generation of any kind, SGLang’s frontend language was basically designed around this from day one. Both engines now support grammar backends like XGrammar and LLGuidance, so on paper the feature list looks similar. In practice SGLang’s implementation feels less like a bolt on and more like the core design.
For agent loops specifically, the pattern is usually: same system prompt, growing conversation history, occasional tool calls that get appended back into context. That’s exactly the prefix reuse case RadixAttention is built for. I switched a RAG project over from vLLM to SGLang back in April mostly for this reason and honestly the GPU bill dropped by something close to 25 percent within the first week, though I’ll be upfront that part of that was also us fixing some sloppy prompt construction on our end at the same time, so I can’t give SGLang 100 percent of the credit.
Deployment, Ecosystem, and the Boring Stuff That Actually Matters
vLLM wins this category without much argument. It has the broadest hardware support (AMD, Intel Gaudi, TPU, not just NVIDIA), the biggest community, and it’s the default engine behind Hugging Face’s own Inference Endpoints now. When Hugging Face put their own TGI project into maintenance mode back in December 2025, they pointed everyone toward vLLM or SGLang for new deployments, and vLLM became the default choice, SGLang the alternative. That single decision tells you a lot about where the ecosystem’s confidence sits for general purpose serving.
SGLang deployment is, and I’ll just say it plainly, a bit more fiddly. Nothing crazy, but you do need to configure the runtime a little more carefully to actually get the RadixAttention benefits, and the documentation, while improving fast, still has gaps compared to vLLM’s years of accumulated tutorials, Stack Overflow answers, and Discord troubleshooting history. If your team is small and doesn’t have anyone who’s used it before, budget an extra day or two for the learning curve.
Both support quantization formats like FP8, INT8, and AWQ with roughly similar performance, so that’s not really a deciding factor either way.
So Which One Actually Suits You
I’ll just give you the straight version instead of hedging forever.
If you’re running batch jobs, generating thousands of independent summaries or classifications where every prompt is different, or you need the widest possible hardware compatibility and a large support community to lean on, use vLLM. It’s the safer default, and for a huge chunk of production use cases the 3 to 5 percent throughput difference at scale genuinely doesn’t matter compared to how much smoother deployment is.
If you’re building a chatbot, a customer support agent, a RAG app where users repeatedly query the same documents, or anything with long system prompts and multi-turn history, seriously consider SGLang. The prefix caching story is not marketing fluff, it shows up in real GPU bills and real latency numbers.
And if you’re running a mixed workload, which honestly describes most real companies I’ve talked to, plenty of teams just run both behind a gateway and route by request type. That sounds like extra complexity but swapping between the two really is closer to a config change than a rewrite since they both expose the same OpenAI style API. Worth trying both on your own traffic pattern before committing, because the write ups (including this one) can only tell you what worked for someone else’s workload, not yours.
One thing that’s still unresolved as I’m writing this in August 2026: there’s ongoing chatter in both projects’ GitHub issues about long context performance under 100k+ token windows, and neither engine has fully nailed that yet for every model architecture. If your use case leans heavily on long documents, test that specifically before you commit either way, don’t just trust the marketing page.