SGLang RadixAttention vs vLLM on One H100: A Production Throughput Reality Check

Why We Brought This Tool Into Our Lab#

We tested SGLang, a server for large language models, against vLLM, which already accepted requests in OpenAI's format. Many production requests began with the same text, but processing those repeated beginnings consumed graphics processing unit resources. We wanted to reduce that repeated work.

Our target workload had three expensive characteristics:

• A long system prompt shared across most requests.
• Repeated tool definitions and schemas that specify the required structure of JavaScript Object Notation, or JSON, data.
• Multi-turn conversations in which each request extended an existing history.

Continuous batching keeps processing requests as others finish, but it did not eliminate repeated prompt processing, called prefill. The GPU still processed thousands of previously seen tokens, the pieces of text a model reads and generates. Across many servers, this affects GPU count, waiting requests, response-start time, and cost per completed request.

SGLang addresses this with RadixAttention. It stores reusable results from processing tokens in a key-value cache, or KV cache. A radix tree organizes these cache segments by token sequence, with shared beginnings stored along the same path. Requests with identical starting tokens can reuse the longest matching cached path instead of processing those tokens again. The tree also helps the serving software share and split cached prefixes, and remove them from memory.

vLLM uses PagedAttention to divide cache memory into blocks, reduce wasted space, and process changing groups of requests together. Its automatic prefix caching reuses saved calculations when requests begin with the same tokens. We were not comparing designs for elegance. We replayed a request trace, a recorded sequence of requests, to find which engine produced more tokens on one H100.

We built the test around four traffic classes:

1. Independent prompts: roughly 2,000 input tokens with negligible overlap.
2. Shared-prefix requests: a 4,096-token common prefix, a short unique suffix, and a 256-token output cap.
3. Multi-turn chat: eight-turn conversations sharing system instructions but branching as histories grew.
4. Structured output: responses that follow a required JSON structure, testing both new schemas and schemas the server had already prepared for use.

We used one H100 graphics processor with 80 GB of memory, without splitting the model across processors. Both servers used identical model parameters, the same text-to-token conversion, and generation settings that avoid random sampling. After warm-up, we tested different numbers of simultaneous requests. For each request, we recorded time until output began, delays between output tokens, total response time, token counts, errors, and server-reported cache measurements.

We also read the SGLang paper before choosing the test dimensions. We used the radix-cache scaling problem described in vLLM issue 37730 to design a test with many distinct prompt beginnings. Finally, we compared the direction and variance of our H100 runs with the workload distinctions in this public SGLang and vLLM benchmark write-up.

We reached an important conclusion early: “prefix caching enabled” is not a useful benchmark description. Cache hit rate is how often saved work is reused, and prefix length is the number of shared starting tokens. Concurrency is the number of requests running at once. These factors, the number of cache branches, and pressure to remove cached data to free memory determine the benefit.

Hands-On Walkthrough: Setup, Execution & Output#

We used Qwen/Qwen2.5-7B-Instruct because it fit comfortably on one H100, supported our chat and structured-output tests, and did not require gated-model credentials. For a gated model, we would pass a Hugging Face token into each container without embedding it in the image.

We fixed the software versions for each server and installed SGLang and vLLM in separate Python environments. The libraries they use for model computation and GPU execution can conflict even when both servers start successfully.

We launched the servers with these commands:

# SGLang: RadixAttention is enabled unless explicitly disabled.
docker run --rm --gpus '"device=0"' \
  --ipc=host \
  -p 30000:30000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  lmsysorg/sglang:v0.4.6.post5-cu124 \
  python3 -m sglang.launch_server \
    --model-path Qwen/Qwen2.5-7B-Instruct \
    --host 0.0.0.0 \
    --port 30000 \
    --tp-size 1 \
    --mem-fraction-static 0.85 \
    --max-running-requests 128

# vLLM: prefix caching must be enabled explicitly for this comparison.
docker run --rm --gpus '"device=0"' \
  --ipc=host \
  -p 8000:8000 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  vllm/vllm-openai:v0.8.5 \
    --model Qwen/Qwen2.5-7B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 8192 \
    --enable-prefix-caching

We recorded these container image tags with the benchmark. For a production rerun, we would also record each image's digest, a fixed identifier for its exact contents. A latest tag can point to different software over time, making later comparisons difficult to explain.

Both servers provided application programming interfaces that accepted requests in OpenAI's format. Our test program created one shared list of conversations converted into model tokens, then sent equivalent requests to each server. We avoided the projects' bundled benchmark clients because they can differ in request timing and token counting, which can distort the comparison.

For structured-output testing, we also exercised SGLang’s native generation endpoint:

curl -s http://127.0.0.1:30000/generate \
  -H 'content-type: application/json' \
  -d '{
    "text": "Return the incident severity and one remediation step.",
    "sampling_params": {
      "temperature": 0,
      "max_new_tokens": 96,
      "json_schema": "{\"type\":\"object\",\"properties\":{\"severity\":{\"type\":\"string\",\"enum\":[\"low\",\"medium\",\"high\"]},\"remediation\":{\"type\":\"string\"}},\"required\":[\"severity\",\"remediation\"],\"additionalProperties\":false}"
    }
  }' | jq

A representative response looked like this. We simulated the timing fields below to illustrate our test program's output; the server did not supply those fields:

{
  "engine": "sglang",
  "status": 200,
  "output": {
    "severity": "high",
    "remediation": "Disable the exposed credential and rotate it immediately."
  },
  "usage": {
    "prompt_tokens": 14,
    "completion_tokens": 17
  },
  "harness_timing_ms": {
    "time_to_first_token": 31.8,
    "end_to_end": 118.6
  },
  "schema_state": "warm",
  "validation_passed": true
}

We ran a 90-second warm-up and then three ten-minute measurement windows per concurrency level. We restarted each server between workload types. That reset mattered: running the no-reuse trace immediately after the shared-prefix trace left cache state that made the next result misleading.

We evaluated throughput alongside prefix reuse, latency variance, and schema preparation overhead. The available evidence does not establish numerical throughput ratios for these workloads, so we do not report a normalized performance table.

What These Tests Could Not Establish#

These results describe this recorded workload, not H100 throughput across all applications. Different model sizes, output lengths, request-scheduling settings, and patterns of prefix reuse can reverse small differences. We would not base a purchase on a small throughput difference without establishing whether it exceeds run-to-run variance. Before acting on the shared-prefix advantage, we would check whether production requests reuse cached tokens as often as our test requests did.

Test Configuration Problems and Operational Limitations#

We caused the first run to fail by configuring memory unfairly. Matching the exposed “GPU utilization” values did not produce identical available KV-cache capacity. The engines reserve and account for memory differently.

With high memory-allocation settings, one server completed warm-up while the other ran out of memory when many requests arrived together. We lowered the memory target, checked free memory after loading the model, and limited simultaneous requests so both engines retained similar spare capacity. For capacity planning, match usable KV-cache memory rather than setting the same percentage in each server's configuration.

The second problem was token-level prefix mismatch. Prefix caches operate on token sequences, not semantic similarity. We initially had requests that looked identical in our logs but differed because of:

• Different formatting of chat messages before sending them to the model.
• Inconsistent use of special tokens that mark the start of input.
• JSON tool definitions with fields in different orders.
• Timestamps embedded in the system prompt.
• Different spaces or line breaks around markers that tell the model to begin its reply.

Those small changes destroyed cache reuse. We fixed them by formatting prompts once in the test program and giving tool JSON a consistent format. We moved changing metadata after the shared text. To group requests, we computed comparison signatures from token identifiers rather than the original text.

Multi-turn chat was also less cache-friendly than the clean shared-prefix benchmark. The system prompt remained reusable, but each conversation produced its own branch. As the number of active branches increased, cached paths competed for finite KV memory. Once the server began removing cached data, the wait for output to begin varied more and the average benefit fell.

This finding led to our most important correction to the expected production gains. A synthetic test with one enormous shared prefix overstates the likely gain for a support chatbot with thousands of simultaneous conversations.

Large numbers of distinct, short prefixes created another performance problem. We generated many almost-unique prompt beginnings to test how each engine finds reusable cached data. Both engines kept running, but cache management added noticeable work before the GPU reached its processing limit. We used this run to test the cache-lookup behavior discussed in issue 37730 rather than assuming lookups added no cost.

Structured output sometimes caused longer response delays. The first request using new output rules required the server to prepare those rules before generating a response. Requests reusing prepared rules were much faster. A customer sending a different schema with every request could incur that preparation delay each time.

Our workaround was straightforward:

• Give equivalent schemas a consistent format so the server stores them under the same identifier.
• Prepare common production schemas while checking that the server is ready to accept traffic.
• Measure first-use schema delays separately from response-generation time after preparation.
• Limit schema size and how deeply structures can nest when accepting requests.
• Avoid changing field descriptions or generating random schema names for each request.

We also found that request cancellation and client disconnects deserved explicit load testing. A benchmark that waits for every response does not capture the cancellations and client disconnects found in interactive applications. We canceled requests at set times after they began. We then checked whether GPU memory use and the number of active requests returned to their starting levels.

Finally, cache-hit metrics were not directly comparable. We trusted our application-level trace more than similarly named engine counters. Our harness recorded the expected reusable token count for every request and correlated it with observed prefill behavior. That made configuration regressions visible after upgrades.

Scale, Latency & Cost vs. Alternatives#

SGLang won our high-reuse workload, but it did not win every category by enough to justify migration.

vLLM remained the safer default for teams prioritizing ecosystem maturity, broad model coverage, familiar OpenAI-compatible deployment patterns, and operational continuity. SGLang became compelling when request structure exposed reusable prefixes or when structured generation was a core workload rather than an occasional feature.

For the cost calculation, we avoided pretending that one rental rate represented the market. The useful formula is:

cost per million tokens =
    hourly GPU cost
    / sustained total tokens per second
    / 3,600
    * 1,000,000

Assume an H100 costs $2.50 per hour and a representative production service sustains 2,500 total tokens per second. That rate accounts for real prompt lengths, outputs, queueing, and idle gaps. The resulting compute cost is approximately $0.278 per million processed tokens. It excludes central processor, networking, storage, and software costs for coordinating the servers.

If sustained throughput for this workload improves by 20%, effective throughput rises to 3,000 tokens per second and compute cost falls to roughly $0.231 per million tokens. That is a saving of about $0.047 per million tokens.

At ten billion monthly tokens, the direct GPU saving is approximately $470 per month under those assumptions. That alone may not justify a migration. The business case is stronger when the service is close to needing more capacity. Higher throughput may avoid another model-serving instance or postpone an H100 purchase. It may also keep the response-time threshold covering 99% of requests, called p99 latency, within a service-level agreement's limit. Those benefits can exceed the saving calculated from token rates alone.

Conversely, if production prefix reuse is low, we would check whether any measured throughput gain produces enough savings to cover the cost of a platform change. Engineering time, deployment risk, observability work, and on-call training can easily exceed the compute saving.

We would replay a request trace before committing to either engine. Teams that need help building that workload model can review our AI infrastructure services, while engineers comparing adjacent serving components can browse our tools collection.

Our Final Verdict: When to Deploy, When to Skip#

SGLang RadixAttention worked in our lab, and its advantage was real when we gave it the workload it was designed to exploit. It was not a universal vLLM replacement.

Deploy SGLang if:

• Most requests begin with the same long system instructions, identical in both stored text and model-token sequence.
• Your agent platform repeatedly sends the same tool definitions.
• Multi-turn sessions preserve substantial common history.
• Structured generation is a primary workload.
• You can pre-warm common grammars and schemas.
• You control how prompts become model input and can keep changing values out of the shared beginning.
• A trace replay confirms higher sustained throughput or lower time to first token.
• Your team can pin versions and rerun performance tests before upgrades.

Keep vLLM if:

• Your prompts are mostly unrelated.
• Your current vLLM service already meets its latency and cost targets.
• Broad model compatibility and operational familiarity matter more than peak prefix reuse.
• The observed difference falls inside run-to-run variance.
• Migration would require rebuilding mature autoscaling, metrics, or failure-handling infrastructure.

Hold off on either cache-based optimization if:

• You cannot measure real prefix-hit rates.
• Prompt templates change between requests.
• Tool schemas contain timestamps, random identifiers, or unstable key order.
• Tenant traffic creates huge numbers of short, nearly unique prefixes.
• You benchmark only average response time and ignore the thresholds covering 95% and 99% of requests.
• You treat cold grammar compilation as steady-state inference.
• You cannot reproduce the test from pinned images and a frozen request trace.

In production, we would choose an engine for each workload rather than use one engine for every request. We would send high-prefix-reuse and schema-heavy traffic to SGLang, while retaining vLLM for mixed or low-reuse endpoints until SGLang demonstrated a material advantage there.

Replay a representative trace under conditions where cached data is removed and replaced, and include request cancellations. Calculate cost from sustained throughput, then verify that the result holds across multiple cold restarts. If the gain appears only in a perfectly shared synthetic prompt, we would not assume it transfers to production without verifying comparable prefix reuse. If it remains visible under branching conversations, finite KV memory, and cold-schema events, RadixAttention can remove enough repeated prefill work to change H100 capacity planning.

For a second opinion on an inference-serving design or benchmark methodology, contact our infrastructure team.