vLLM Keeps Generating After Cancel Unless the Client Closes the Connection: Tested on SGLang and Two Gateways

We cancelled streaming requests mid-answer three ways and counted the tokens the scheduler accepted afterwards, on vLLM 0.31.0, SGLang 0.5.21, LiteLLM 1.105.1 and Bifrost transports/v2.3.0, in one Kubernetes cluster in AWS.

On vLLM 0.31.0, closing the HTTP connection stopped generation at the abandon point in every arm we tested: 0.0 scheduler-accepted tokens after the close. Clients that kept reading, or held the socket unread, left the engine running. SGLang 0.5.21 stopped later, and an unread socket did not stop its retried requests. One cluster, six blocks per arm.

Summary

A user presses stop. Whether the GPU stops depends on what the client did with the connection. We measured that on current engine versions, in one Kubernetes cluster in AWS, with an in-process observer on the engine scheduler that records every request admission, every accepted token batch and every logical terminal. The measure throughout is scheduler-accepted tokens: the count the scheduler accepted for a request, not GPU kernel time and not energy.

Each request was run under three client conditions. P3g250 closes the transport 250 ms after the abandon point. P2 is a drained walk-away: the client no longer wants the answer but keeps reading to the end. P2nr is a bounded no-read: the client holds the socket unread for 5 s, then closes it. Three fault types were used: F0 normal completion, F3 abandonment after 8 visible content chunks, and F4 a 1.5 s attempt deadline followed by a retry.

On vLLM 0.31.0 with Qwen3-8B on one NVIDIA L4 (360 matched sets, 1,817 attempts), a transport close left 0.0 tokens accepted after the close. The drained walk-away accepted a mean 52.8 more tokens per abandoned request than the close (95% block-bootstrap CI 48.5 to 57.0), and the bounded no-read 44.3 more (40.9 to 47.7). Under the retry fault, non-winning attempts accepted 147.9 orphan tokens per task under the drained walk-away against 0.0 under the close, and completion within the deadline fell from 38.0% to 20.7% (drained) and 11.3% (no-read). The same direction held for gpt-oss-20b on L4 and A10G, at batch 64, on free-form prompts, under tensor parallel 2 and 4 and pipeline parallel 2, and from a client on another node in the same VPC.

LiteLLM 1.105.1 passed a client close through with no extra tokens. Bifrost transports/v2.3.0 closed the upstream about 3 tokens later on every abandoned set (+2.95, CI +2.90 to +3.00). A 1 s gateway timeout on LiteLLM stopped the upstream under every client condition; on Bifrost it did not when the client kept the connection, adding 109.5 to 124.9 orphan tokens per set. A 502 after engine admission doubled engine work on both gateways (61.95 orphan tokens per set). A 503 before admission added nothing.

SGLang 0.5.21 stopped later than vLLM after a close (2.0 tokens and 120 ms against 0.0 and 61 ms), and the bounded no-read did not stop retried requests (155.1 orphan tokens per set against 20.5 on vLLM). Under the retry fault, SGLang's close-to-terminal mean was 2,300 ms, which this study did not explain.

For the record, an earlier run on vLLM 0.11.0 was withdrawn on 15 October 2026 and is superseded by this report. Results are limited to one cluster, one host per arm, six fresh-engine blocks per arm, greedy decoding and synthetic fixtures. No cost figure is given.

Figure 1: Tokens the scheduler accepted after the abandon point under F3, mean per matched set, by client condition. The close is 0.0 in every arm tested; the two walk-away conditions accepted the rest of the answer. vLLM 0.31.0 unless the arm names SGLang.

What this study shows and does not show

Scope, environment and evaluation bounds:

• Versions: vLLM 0.31.0 (image vllm/vllm-openai@sha256:c1c9f6fd), SGLang 0.5.21 (image lmsysorg/sglang@sha256:2dbe4c7f), LiteLLM 1.105.1, Bifrost transports/v2.3.0. Other engines, versions and serving front ends were not tested.
• Environment: One Kubernetes cluster in AWS built for the study. NVIDIA L4 on g6.2xlarge, A10G on g5.2xlarge, four L4 on one g6.24xlarge host for the parallel arm. No production systems or user data.
• Measure: Scheduler-accepted tokens from an in-process observer calibrated against a stock engine. Not GPU kernel time, not energy, not a bill.
• Client behaviour: Clients abandoned after the eighth visible content chunk and timed out on a fixed 1.5 s attempt deadline. Real abandonment follows other distributions.
• Statistics: Six fresh-engine blocks per arm. Confidence intervals are 95% percentile block bootstrap with 10,000 resamples of the paired difference against the close condition. They support means, not tails. The study was planned, not pre-registered.
• Parallelism: Qwen3-8B only. gpt-oss-20b was not run under tensor or pipeline parallelism.
• Hosted providers: Dated observations from 7 October 2026 on one prompt shape, retained unchanged. Provider-reported usage, not an invoice.
• Spend: No cost figure is given. The study's accrued spend was not reconciled.

Key takeaways

• A transport close stopped vLLM 0.31.0 at the abandon point in every arm tested: 0.0 scheduler-accepted tokens after the close.
• Not closing accepted the rest of the answer: +52.8 tokens per abandoned Qwen3-8B request when the client drained, +44.3 when it held the socket unread for 5 s.
• Under retries, closing stale attempts raised completion within the deadline by 17 to 27 points on one L4, and up to 35 points at load 8 with short deadlines.
• LiteLLM 1.105.1 passed the close through cleanly. Bifrost transports/v2.3.0 added about 3 tokens per close, and its 1 s timeout did not stop the upstream unless the client closed.
• Tensor parallel ranks stopped together: 0 forward steps after close on every rank. Pipeline parallel ran one extra step per stage.
• SGLang 0.5.21 let 2.0 tokens through after a close, and an unread socket did not stop its retried requests.

Want the steps without the method? Our practical guide, how to stop your LLM generating tokens after a user cancels, turns these findings into a checklist. The full report follows.

Background

A streaming LLM response is an HTTP connection that stays open while the engine decodes. When the user cancels, the client can do three different things with that connection, and the engine only learns about one of them. If the client closes the transport, the server's write fails or its disconnect check fires, and the request can be aborted. If the client keeps reading to the end, nothing tells the engine that the answer is unwanted. If the client stops reading but leaves the socket open, the engine keeps writing into the socket buffer until the buffer fills or the request ends.

Each abandoned request that keeps decoding occupies a slot in the continuous batch ^[1] and a KV cache allocation that a live request could have used ^[2]. Under load, that is the mechanism behind retry storms: a client that times out and retries, without closing the first attempt, now has two requests in the batch, and the engine is slower for everyone ^{[3, 4, 5]}. Both engine projects have issue reports describing disconnected streams that decode to max_tokens ^{[6, 7]}, and vLLM has fixed one middleware path where disconnects were not propagated ^[8] and one gRPC front-end path in October 2026 ^[9]. The SGLang report was closed as fixed on 15 October 2026, during this study.

Two things made earlier measurements of this hard to trust. Client-side token counts miss everything the engine produces after the client looks away, and reasoning models produce hidden tokens before the first visible one. Engine Prometheus metrics are sampled, so they cannot place a terminal against a close to the millisecond. This study instead observes the scheduler itself, from inside the engine process, with a calibration run that bounds the observer's own effect on throughput and latency.

A note on language. We write "orphan tokens" for tokens the scheduler accepted for a request after the point at which no client wanted them: after the abandon point under F3, or for a non-winning attempt under F4. We write "tokens after close" for tokens accepted after the client's transport close specifically. Under P2 the client never closes, so that measure is not defined for it.

Method

Engines, images and models

Every engine_version.txt under the arms reads 0.31.0 for vLLM or 0.5.21 for SGLang. Image digests are pinned in the Kubernetes manifests in the data repository.

Hardware

The observer

The observer wraps the engine scheduler in process (vllm_observer_v3.SyncScheduler on vLLM; a tokenizer-manager observer on SGLang that converts cumulative token counts to per-step deltas). It journals four event types per request to a local file: admission, each accepted output batch with its token count, finish signal and logical terminal. Every event carries a monotonic timestamp on the engine host and a per-emitter sequence number, so gaps are detectable. On the parallel arm a rank observer (vllm_rank_observer_v3.Worker) also journals forward steps and free events on every rank.

{"sequence": 23124, "event_type": "scheduler_logical_terminal",
 "request_id": "chatcmpl-rsrch-v3a-F3-0060-P3g250-a1-87d41ac8",
 "utc_timestamp": "2026-10-15T04:48:10.058103+00:00",
 "clock_domain": "real-single-host",
 "data": {"status": "FINISHED_ABORTED", "cumulative_output_tokens": 8,
          "num_computed_tokens": 61, "gpu_completion_observed": false,
          "kv_release_observed": false, "scheduler_mode": "sync"}}

The journal records scheduler acceptance events. It does not observe GPU kernel completion or KV release (gpu_completion_observed and kv_release_observed are false by design).

Calibration gates

Before any measurement, the observer's effect on the engine was bounded with 12 fresh-engine blocks on one L4, alternating stock and observed engines in the order ABBA BAAB ABBA, 65 measured requests per block at concurrency 8, 256 tokens per request with ignore_eos. Each block produced 16,640 tokens.

Each observed journal had 16,783 events in a gap-free sequence, 66 admissions and 66 logical terminals, 0 unmatched. Six pairs support the mean ratio only. The SGLang observer had a two-block smoke run and a unit test; its overhead ratio was not measured.

Conditions and faults

Each matched set runs one fixture under all three conditions in seeded order within the same block, so condition contrasts are paired within fixture and within block. Conditions are never pooled.

Blocks and statistics

Every arm ran six fresh-engine blocks. The engine was restarted between blocks and the plan for each block was generated by a seeded planner (tools/plan_v3_*.py). Confidence intervals are 95% percentile block bootstrap of the paired difference against P3g250, 10,000 resamples, drawn over blocks. Six blocks support means and block-level intervals, not tail precision. Decoding was greedy. Fixtures are synthetic prompts stratified by input and output length, with a development and a validation split; their SHA-256 is recorded with each plan.

python3 tools/analyze_v3_section_a.py \
  evidence/runs/rsrch-section-a-20261015/section-a
# rewrites section-a-analysis.json in that folder and prints it

"coverage": {"journal_sequence_ok": 6, "blocks": 6,
             "attempts_200": 1817, "admitted": 1817, "terminal": 1817},
"complete_sets": 360, "incomplete_sets": 0

tools/fetch_v3_run.sh <engine-pod> /data/runs section-a \
  evidence/runs/rsrch-section-a-20261015
# prints MANIFEST-OK when every file's SHA-256 matches the in-pod manifest

Coverage was checked for every arm: every HTTP 200 attempt has a scheduler admission and a logical terminal, every journal sequence is gap-free, and every set is complete. The counts per arm are in the findings below. Every analysis JSON in the data repository was regenerated from the raw journals on 19 October 2026 and compared byte for byte with the stored copy.

Findings: vLLM 0.31.0 on one GPU

Qwen3-8B on one L4

Run 15 October 2026, 04:43 to 07:27 UTC, one L4, 6 blocks, 360 matched sets (F0 60, F3 150, F4 150), 1,817 HTTP 200 attempts, every attempt admitted and terminal, 6 of 6 journals gap-free.

Close to scheduler logical terminal under P3g250, same host clock: F3 mean 60.7 ms, p95 62.1 ms; F4 mean 94.0 ms, p95 465.6 ms. Under F3 P2nr, 117 of 150 requests reached their terminal before the 5 s close, so an unread socket did not stop generation for most short requests; the other 33 ended a median 6.0 ms after the close.

Figure 2: Share of F4 tasks that completed within the 1.5 s attempt deadline, by client condition. Closing the stale attempt helped on one L4 and at tensor parallel 4; at tensor parallel 2 the order reversed; SGLang 0.5.21 completed under 10% in every condition.

gpt-oss-20b on L4 and A10G

gpt-oss-20b is a reasoning model that produces hidden tokens before the first visible content chunk, so the abandon point arrives later in the request's life. L4 run 15 October, 09:16 to 11:35 UTC, 1,510 attempts, 380 sets. A10G replication 08:08 to 09:00 UTC, 488 attempts, 160 sets, each with an L4 twin by task id. A first L4 attempt was lost when the autoscaling group rebalanced its node; the run reported here is a full restart.

The 34.9 orphan tokens under P3g250 on L4 (97.8 at medium reasoning) are hidden reasoning produced before the first visible chunk. No cancellation path can recover them, because they exist before the user has anything to cancel. Under P2nr on this model the engine finished most requests before the 5 s close, so the unread hold accepted most of the remaining answer: 80.4 of the 138.4 tokens a full drain accepted on L4, and 141.6 of 150.1 on A10G. On A10G the 1.5 s attempt deadline rarely fired, so the F4 contrast rests on the L4 run.

Batch 64, free-form prompts and the F4 response surface

Three further Qwen3-8B arms on one L4, same day.

Batch 64. --max-num-seqs 64 with 48 background streams per task, 150 F3 sets, 450 attempts. Tokens after abandonment: P3g250 0.0, P2 53.7 (CI of difference 47.3 to 59.1), P2nr 40.3 (37.8 to 42.8). The batch-8 result (52.8 and 44.3) reproduced at batch 64 on this engine.

Free-form prompts. 60 natural prompts, each as an F3 set and an F0 reference set of three uninterrupted runs, 360 attempts, greedy decoding. Tokens after abandonment: P3g250 0.0, P2 174.7 (125.0 to 224.2), P2nr 80.6 (79.9 to 81.0). The three reference runs per prompt gave identical token counts, and the reference length minus the 8 chunks seen predicted the P2 figure exactly (median absolute error 0, Pearson 1.0). Under greedy decoding on this engine, the waste of a drained walk-away is the full remaining answer, and it is predictable from one uninterrupted run. With sampling the references would vary.

F4 response surface. Five attempt deadlines (0.5, 1.0, 1.5, 3.0, 6.0 s) by three filler loads (4, 8, 12 streams), 40 sets per cell, 3,730 attempts, 600 sets. Completion within the deadline:

Orphan tokens under P3g250 were 0.0 in all 15 cells. The completion gain from closing stale attempts peaked where retries fired often but the engine still had room to finish (load 8, short deadlines). It disappeared when retries never fired (load 4, deadline 1 s or more) and when the engine was saturated so that nothing finished in time (load 12). Spearman correlation between P2 timeout rate and gain over all 15 cells was 0.21; the gain is monotone in timeout rate only below saturation. Without the five load-12 cells it was 0.95, an exploratory figure that was not planned.

Figure 3: F4 response surface on one L4, Qwen3-8B: completion gain of the transport close in percentage points, 40 sets per cell. The gain peaks at load 8 with 0.5 s and 1.0 s deadlines and is zero at load 12, where no condition could complete.

Findings: gateways and a remote client

Gateways with retries on

Run 16 October 2026, 01:36 to 04:28 UTC. LiteLLM 1.105.1 and Bifrost transports/v2.3.0 in front of vLLM 0.31.0 with Qwen3-8B on one L4, with each gateway's retry policy enabled and a fault-injecting proxy between gateway and engine. 480 sets (80 per arm), 2,118 client attempts, 2,598 engine requests, every engine request with a scheduler terminal. The proxy journal records 480 pre-admission 503 injections and 480 admitted-then-502 injections.

Paired differences on the timeout arms: Bifrost orphan tokens P2 minus P3g250 +109.5 (87.6 to 130.1), P2nr minus P3g250 +124.9 (106.1 to 145.4); Bifrost completion -0.20 (-0.35 to -0.04) and -0.29 (-0.40 to -0.17). LiteLLM orphan tokens 0 (0 to 0) and +0.01 (0 to 0.04).

Client-side status on the timeout arms: LiteLLM returned HTTP 500 on 497 attempts and 200 on 99. On Bifrost, 463 attempts ended in a client header timeout (no response headers before the client deadline) and 99 returned 200.

Figure 4: Engine tokens accepted for non-winning F4 attempts through two gateways, 80 sets per cell. A 1 s gateway timeout on LiteLLM 1.105.1 stopped the upstream under every client condition; on Bifrost transports/v2.3.0 it did not unless the client closed.

Gateways with retries off

Same day, before 05:05 UTC, retries disabled on both gateways, three routes (direct to vLLM, LiteLLM, Bifrost), 570 sets (F0 30, F3 80, F4 80 per route), 2,633 client attempts, 2,633 engine requests, exactly one per attempt on every route. Each route by condition cell used the same fixture for the same set index, so the route contrast is paired.

Gateway minus direct, paired on fixture and condition (95% block bootstrap, n = 80 per cell):

A client close propagated through LiteLLM with no extra tokens. Bifrost closed the upstream about 3 tokens later on every abandoned set. An unread held connection under F4 added about 60 more orphan tokens per set through either gateway than direct: direct P2nr stopped when the engine's send buffer filled and the 5 s close fired, while both gateways appeared to keep reading upstream, so the gateway's buffer, not the client's, decided when generation stopped. No route changed F4 completion within the CI.

Remote client

Run 16 October 2026, 02:06 to 02:53 UTC. The engine pod and a client pod on different nodes in the same VPC, 150 F3 sets, each fixture run as local and remote by P3g250 and P2 in seeded order, 600 attempts. Each block measured the engine-to-client clock offset from 200 probes before and after; minimum RTT was 257 to 272 µs, offset drift 28 to 79 µs per block, and the conversion bound (larger half-RTT plus half the drift) was at most 170 µs. Token counts come from the engine journal and do not depend on the clock.

A close from another node stopped engine work the same way as a close on the engine host, with 0 tokens after close in all 150 sets. The close-to-terminal difference of -0.12 ms is inside the 170 µs clock bound, so this run does not show a difference. The delivery-lag difference is about one same-VPC round trip; its CI upper end exceeds the clock bound and its lower end does not, so the run supports only "remote adds less than about 0.3 ms". Cross-region, TLS, proxies and load balancers between client and engine were not tested.

Findings: Tensor and pipeline parallelism

Run 18 October 22:26 UTC to 19 October 03:32 UTC, Qwen3-8B on one g6.24xlarge host with four L4 GPUs, vLLM 0.31.0, four configurations: tensor parallel 1, 2 and 4 (TP=1, TP=2, TP=4) and pipeline parallel 2 (PP=2). All configurations share the same fixture draw, so configuration contrasts are within fixture. TP=1 on the same host is the baseline for this arm; it is not the Section A g6.2xlarge number, so compare conditions within a configuration.

One EngineDeadError traceback appears in the TP=4 block 3 engine log at 02:25:46 UTC. It is block teardown: the driver sends SIGTERM after the last task, the API server logs shutdown triggered first, and the output handler then fails on the closed engine. The last scheduler terminal in that block is 21 s earlier, so no task was affected.

Scheduler measures

Close to scheduler terminal under F3 P3g250: TP=1 60.9 ms, TP=2 32.4 ms, TP=4 18.3 ms, PP=2 61.6 ms.

Figure 5: Client close to scheduler logical terminal under F3 P3g250 on one g6.24xlarge host, Qwen3-8B on vLLM 0.31.0. The terminal arrived sooner as tensor parallel degree rose; pipeline parallel 2 matched the single GPU.

Rank measures

Two readings in this arm are inferred from means and were not checked against per-attempt timelines. First, under F3 the unread socket accepted as much as a full drain on TP=2 and TP=4 (55.8 tokens each) but less on TP=1 and PP=2; the reading that the socket send buffer, not the engine, bounds P2nr on TP=1 assumes a buffer fill point that was not measured. Second, under F4 the P2nr orphan tokens rose with tensor parallel degree (24, 54, 118) while P2 orphan tokens fell (155, 128, 73); the reading that faster decode lets more retries finish before the deadline comes from the token and completion means only.

Findings: SGLang 0.5.21

Run 16 October 2026, Qwen3-8B on one L4, SGLang 0.5.21, 280 sets (F0 30, F3 150, F4 100) by three conditions, 560 HTTP 200 attempts, 560 admitted, 560 with a scheduler terminal, 6 of 6 journals gap-free. The vLLM column is the Qwen3-8B on one L4 arm above, with the same design but different fixtures and seed, so compare it by direction, not by exact value.

Close to scheduler terminal under P3g250: F3 mean 120 ms (vLLM 61 ms), F4 mean 2,300 ms (vLLM 94 ms).

Two items are open. The F4 P3g250 close-to-terminal of 2,300 ms with only 0.58 orphan tokens suggests that closed requests stay in the SGLang waiting queue and receive a terminal only when the scheduler reaches them; this was not confirmed from the queue state. The F4 completion gap against vLLM may come from engine throughput under saturation or from fixture differences; median time to first token across all attempts was 229 ms on SGLang against 217 ms on vLLM, so admission latency alone does not explain it.

Figure 6: SGLang 0.5.21 beside vLLM 0.31.0, Qwen3-8B on one L4, by client condition. The two engines ran different fixture draws, so compare direction rather than exact values. SGLang let 2.0 tokens through after a close; the unread socket did not stop its retried requests.

Hosted providers

These are dated observations from 7 October 2026, 06:44 to 10:39 UTC, retained unchanged from the earlier study. They are provider-reported usage, not an invoice, and they describe one prompt shape on the routes and models named. They were not rerun.

Design: one prompt (count 1 to 512, one number per line, max_tokens 1024; a full Claude Haiku 4.5 answer is exactly 1,024 output tokens). Each pair is one full request read to the end and one cancelled request that closed the TCP stream after 8 or 64 visible chunks, in randomised order, 30 pairs per route per abandon point. Routes: OpenRouter anthropic/claude-haiku-4.5 pinned to the Anthropic, Google Vertex and Azure providers; Amazon Bedrock claude-haiku-4-5 and gpt-oss-20b through ConverseStream. Metered tokens come from the OpenRouter generation record (native_tokens_completion) ^[19] and from the CloudWatch AWS/Bedrock OutputTokenCount metric with one request per model per minute bucket. Delivered tokens are an exact count of the received text; the Bedrock gpt-oss-20b delivered count is an estimate.

On every route tested, hanging up stopped the meter short of the full answer, except for 2 of 180 OpenRouter cancels where the upstream stream had already finished before the cancel propagated. At 8 chunks, OpenRouter routes metered 4 to 16 tokens beyond what was delivered. At 64 chunks they metered 90 to 95 tokens fewer than delivered, on all three providers, which looks like OpenRouter accounting rather than provider behaviour; this is exploratory and makes no claim about invoices. The CloudWatch metric recorded 7 output tokens for every cancelled Bedrock Claude Haiku 4.5 stream whether the client had received 30 lines or 255, and cancelled calls recorded no Invocations datapoint; whether that metric reflects the account's charge was not verified against an invoice. Bedrock gpt-oss-20b streamed in a few large chunks, so the 64-chunk abandon point was never reached and the 8-chunk cancel landed late. The OpenRouter cancelled flag was true on every Anthropic and Azure cancel and false on every Vertex cancel, even though Vertex metering stopped early in the same way, so the flag is not a reliable signal of partial metering across providers.

Figure 7: Hosted routes, dated observations from 7 October 2026: output tokens metered by the provider against tokens delivered before the client hung up, one prompt shape, full answer 1,024 tokens. Provider-reported usage, not an invoice. Not rerun for this report.

Limitations

• Accepted tokens are scheduler acceptance events, not GPU kernel time and not a bill. The observer does not see GPU completion or KV release.
• One Kubernetes cluster in AWS, one engine host per arm, one model per arm, synthetic fixtures with greedy decoding. Real abandonment follows other distributions.
• Six blocks per arm support means and block-level confidence intervals, not tail precision. The study was planned, not pre-registered: the plan document was never frozen and the freeze step was not separately recorded.
• gpt-oss-20b was not run under tensor or pipeline parallelism, so the parallel findings are Qwen3-8B only.
• The SGLang observer overhead was not measured. It had a two-block smoke run and a unit test, with no stock-against-observed ratio like the vLLM calibration.
• No cost figure is given. The study's accrued spend was not reconciled against a billing record, so this report states no spend.
• The withdrawn earlier run was not rerun as a bridge, so this report makes no version-effect claim between vLLM releases.
• The hosted provider results are dated observations from 7 October 2026 on one prompt shape. They are provider-reported usage, not reconciled against an invoice, and were not rerun for this report.
• Two parallel-arm readings, the socket-buffer bound on P2nr under TP=1 and the F4 trade-off across tensor parallel degree, are inferred from means, not per-attempt timelines.
• The SGLang F4 close-to-terminal of 2,300 ms is unexplained; the queue behaviour it suggests was not confirmed.
• The TP=1 baseline on the g6.24xlarge host is not the Section A g6.2xlarge number; compare conditions within a configuration, not across hosts.
• The single EngineDeadError in the TP=4 block 3 log is block teardown, 21 s after the last terminal, and affected no task.
• The A10G F4 arm had almost no deadline expiries, so the F4 contrast for gpt-oss-20b rests on the L4 run alone.
• The response-surface result under saturation (load 12) shows no gain because no condition could complete; the correlation without those cells is exploratory.
• The remote client arm covers one same-VPC hop. Cross-region, TLS, proxies and load balancers were not tested, and its sub-millisecond latency differences sit at the 170 µs clock bound.
• The free-form prediction holds for greedy decoding; with sampling the reference lengths would vary.
• Teardown of the study's cloud resources is recorded in the infrastructure journal; a KMS key remains pending deletion until 26 October 2026 and could not be verified independently at the time of writing.

Sorami's view

This section is opinion. It is how we would act on these results if we ran LLM serving in production.

Make the close the cancel. In every arm we tested, vLLM 0.31.0 stopped at the abandon point once the transport closed, and the waste came from clients that kept the connection: reading on, or holding the socket unread. Both are client and gateway behaviours you control. Fix the stop button so it closes the HTTP connection, close every attempt you replace before you retry it, and treat a retry without a close as a bug.

Choose and configure gateways on this behaviour. In our runs LiteLLM 1.105.1 passed the close through and stopped the upstream on its own timeout; Bifrost transports/v2.3.0 added about 3 tokens per close and relied on the client close to stop its upstream. Either gateway kept reading upstream when the client stopped, so behind a gateway the unread-socket path is worse than direct. Test your own gateway with the harness rather than assuming.

On SGLang, add a deadline of your own. The 2.0 tokens after close is small; the 155 orphan tokens per set under the unread socket and the 2,300 ms terminal under retries are not. For reasoning models, count tokens at the engine, because hidden reasoning before the first visible chunk is invisible to the client and is produced before anything can be cancelled.

Request cancellation sits beside rate limits and timeouts in our AI production readiness review. The companion guide lists the client-side steps.

What to do now

• Close the HTTP connection on cancel; do not just stop reading.
• Close each timed-out attempt before sending its retry.
• Test the close through your gateway with the harness in the data repository.
• Log engine-side terminals with their status, and alert on aborted requests that accepted tokens after the close.
• Meter reasoning models from engine events, not from what the client saw.
• Give SGLang requests a server-side deadline under saturation.

To have Sorami check how your LLM serving stack handles cancelled and abandoned requests, see the AI production readiness review or contact us.

Frequently asked questions

Appendix: References

1. G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim and B.-G. Chun. Orca: a distributed serving system for transformer-based generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI '22), pages 521 to 538, 2022. usenix.org/conference/osdi22/presentation/yu.
2. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang and I. Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP '23), pages 611 to 626, 2023. doi:10.1145/3600006.3613165.
3. J. Dean and L. A. Barroso. The tail at scale. Communications of the ACM, 56(2):74 to 80, 2013. doi:10.1145/2408776.2408794.
4. M. Ulrich. Addressing cascading failures. Chapter 22 in B. Beyer, C. Jones, J. Petoff and N. R. Murphy (eds.), Site Reliability Engineering: How Google Runs Production Systems, 1st edition. O'Reilly Media, 2016. sre.google.
5. N. Bronson, A. Aghayev, A. Charapko and T. Zhu. Metastable failures in distributed systems. In Proceedings of the Workshop on Hot Topics in Operating Systems (HotOS '21), pages 221 to 227, 2021. doi:10.1145/3458336.3465286.
6. cduk (GitHub). [Usage]: How to purge zombie requests. vllm-project/vllm issue #4000, opened 11 April 2024. github.com/vllm-project/vllm/issues/4000.
7. wkgcass (GitHub). [Bug] Disconnected streaming client leaves a zombie request that decodes to max_tokens. sgl-project/sglang issue #36333, opened 25 August 2026. github.com/sgl-project/sglang/issues/36333.
8. tomeras91 (GitHub). Requests aren't aborted when client disconnects if user provided a middleware. vllm-project/vllm issue #10087, opened 6 November 2024, closed 17 December 2024. github.com/vllm-project/vllm/issues/10087.
9. alec-flowers (GitHub). [Rust Frontend][gRPC] Remove GenerateStream producer task so disconnects abort requests. vllm-project/vllm pull request #60922, merged 10 October 2026. github.com/vllm-project/vllm/pull/60922.
10. vLLM project. vLLM v0.31.0 release and source tree, including vllm/v1/core/sched/scheduler.py. github.com/vllm-project/vllm/releases/tag/v0.31.0.
11. vLLM project. vllm/vllm-openai container image on Docker Hub. Digest used in this study: sha256:c1c9f6fd5c109ba7f0546a59f5b2f15fb87f64c77782e90a27b648b42a8e67c3. hub.docker.com/r/vllm/vllm-openai.
12. SGLang project. SGLang v0.5.21 release. github.com/sgl-project/sglang/releases/tag/v0.5.21.
13. SGLang project. lmsysorg/sglang container image on Docker Hub. Digest used in this study: sha256:2dbe4c7f53230b09aaa8fa41891c13c51d9d2e253f486a70822aaf08783fff75. hub.docker.com/r/lmsysorg/sglang.
14. BerriAI. LiteLLM v1.105.1 release. github.com/BerriAI/litellm/releases/tag/v1.105.1.
15. Maxim AI. Bifrost HTTP transport release transports/v2.3.0. github.com/maximhq/bifrost/releases/tag/transports/v2.3.0.
16. Qwen team. Qwen3-8B model card and weights, revision b968826d9c46dd6066d109eabc6255188de91218. huggingface.co/Qwen/Qwen3-8B.
17. OpenAI. gpt-oss-20b model card and weights. huggingface.co/openai/gpt-oss-20b.
18. SGLang project. SGLang v0.5.21 source: SGLANG_REQUEST_STATE_WAIT_TIMEOUT = EnvInt(4) in python/sglang/srt/environ.py and its use in TokenizerManager._stream_one_response in python/sglang/srt/managers/tokenizer_manager.py. github.com/sgl-project/sglang/blob/v0.5.21/.../tokenizer_manager.py.
19. OpenRouter. API reference: streaming, section on stream cancellation. openrouter.ai/docs/api-reference/streaming.

About Sorami

Sorami is an Australian cyber security and cloud consultancy. We build and secure cloud environments, Kubernetes and AI agents, and the senior engineers who scope the work deliver it themselves. To discuss this research or a review of your own stack, use the form below. Only your email is required, and we reply within one business day.