kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #4209] [Bug][TE] TCP transport: first transfer after an idle gap fails with queue-full rejections; PD decode hangs until timeout #1057

New issue

Open

opened 2026-10-29 21:22:25 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:22:25 +00:00

Owner

Originally created by @CAICAIIs on GitHub (Sep 18, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4209

environment

mooncake-transfer-engine 0.3.13.post1 (PyPI wheel); also reproduced on the 0.3.14rc1 release wheel
vLLM 0.29.0 (built-in MooncakeConnector), 1 node, disaggregated prefill/decode (1P1D) on one host
2× NVIDIA H20 96GB (one GPU per instance), TCP transport (mooncake_protocol: "tcp", MC_FORCE_TCP=1; RDMA path not usable on this host)
Qwen/Qwen3-8B

🐛 Describe the bug

With the TCP transport, the first KV transfer between two engine instances succeeds, but a transfer submitted after a short idle gap (~5 s) fails: on the producer, every slice of the batch is rejected by the TCP lane queue (TCP lane queue-full rejection count: 1 → 2 → 4 → ... → 512+ within milliseconds), batch_transfer_sync_write returns -1, and the error is relayed to the consumer (pulling kv_caches for [...] failed: Mooncake transfer engine returned -1).

The consumer request then stays in WAITING_FOR_REMOTE_KVS (Deferred: 1) and the client hangs until its own timeout. A request sent ~90 s later succeeds, so deployments observe either 60-90 s TTFT spikes after every idle period or outright request failures, depending on client timeout.

How to reproduce

Start P and D with the built-in vLLM MooncakeConnector over TCP (single host; any router that fires the prefill and decode requests concurrently works, e.g. examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py):

# prefill (GPU 0)
vllm serve Qwen/Qwen3-8B --port 8111 --tensor-parallel-size 1 \
  --max-model-len 8192 --max-num-seqs 4 --max-num-batched-tokens 2048 \
  --gpu-memory-utilization 0.60 --no-enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_connector_extra_config":{"mooncake_protocol":"tcp"}}'
# decode (GPU 1)
vllm serve Qwen/Qwen3-8B --port 8112 --tensor-parallel-size 1 ... \
  --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"mooncake_protocol":"tcp"}}'

Send a completion request through the router → succeeds (P logs Sending to ... done, took 0.18 s).
Wait ~5 s. Send it again → hangs until client timeout. P side:

W tcp_transport_lane_impl.h:1330] TCP lane queue-full rejection count: 1
W tcp_transport_lane_impl.h:1330] TCP lane queue-full rejection count: 2
... doubling up to 512+ within one millisecond ...
E mooncake_connector.py:1877] pulling kv_caches for ['cmpl-...'] failed: Mooncake transfer engine returned -1

Send it again ~90 s after step 3 → succeeds again.

Consistently reproduced on two machines (H20 and A100), multiple runs. Also reproduced with MC_TCP_ADMISSION_TIMEOUT_MS=60000 (failure shape changes: the batch is not hard-rejected but still never transfers).

Analysis

After a transfer completes and the peer connection group goes idle, the group is retired — all lanes are closed and group->state is set to CLOSED (tcp_transport_lane_impl.h, retirement block). A submission that arrives shortly after finds the group via state->groups.find(key), sees group->state != GroupState::OPEN, and is hard-rejected with SHUTDOWN (enqueuePooledTransfer) instead of re-creating the group and its connections. The retired group is not removed from state->groups in the retirement path, so submissions keep failing until something re-creates the group — which matches the observed ~90 s self-heal.

Evidence summary:

First transfer after engine start always succeeds; the failure only appears after an idle gap
During the hang the four lane connections are ESTABLISHED with empty kernel queues → the stall is in TE user space (lane queue / admission), not the kernel
Failure depends on the admission timeout (default 1 s fails immediately; 60 s still hangs) but recovery time (~90 s) is unchanged — consistent with a stale CLOSED group rather than a transient queue backlog

Suggested fix: in enqueuePooledTransfer, drop a non-OPEN group and re-create it (or remove retired groups from state->groups atomically when retiring) instead of rejecting the work.

Workaround

Retry the request after ~90 s (self-healed), or keep the peer continuously busy (impractical). No configuration knob we found avoids it.

Originally created by @CAICAIIs on GitHub (Sep 18, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4209 ## environment - mooncake-transfer-engine **0.3.13.post1** (PyPI wheel); also reproduced on the **0.3.14rc1** release wheel - vLLM 0.29.0 (built-in `MooncakeConnector`), 1 node, disaggregated prefill/decode (1P1D) on one host - 2× NVIDIA H20 96GB (one GPU per instance), TCP transport (`mooncake_protocol: "tcp"`, `MC_FORCE_TCP=1`; RDMA path not usable on this host) - Qwen/Qwen3-8B ## 🐛 Describe the bug With the TCP transport, the **first KV transfer between two engine instances succeeds, but a transfer submitted after a short idle gap (~5 s) fails**: on the producer, every slice of the batch is rejected by the TCP lane queue (`TCP lane queue-full rejection count: 1 → 2 → 4 → ... → 512+` within milliseconds), `batch_transfer_sync_write` returns -1, and the error is relayed to the consumer (`pulling kv_caches for [...] failed: Mooncake transfer engine returned -1`). The consumer request then stays in `WAITING_FOR_REMOTE_KVS` (`Deferred: 1`) and the client hangs until its own timeout. A request sent **~90 s later succeeds**, so deployments observe either 60-90 s TTFT spikes after every idle period or outright request failures, depending on client timeout. ## How to reproduce 1. Start P and D with the built-in vLLM `MooncakeConnector` over TCP (single host; any router that fires the prefill and decode requests concurrently works, e.g. `examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py`): ```bash # prefill (GPU 0) vllm serve Qwen/Qwen3-8B --port 8111 --tensor-parallel-size 1 \ --max-model-len 8192 --max-num-seqs 4 --max-num-batched-tokens 2048 \ --gpu-memory-utilization 0.60 --no-enable-prefix-caching \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_connector_extra_config":{"mooncake_protocol":"tcp"}}' # decode (GPU 1) vllm serve Qwen/Qwen3-8B --port 8112 --tensor-parallel-size 1 ... \ --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"mooncake_protocol":"tcp"}}' ``` 2. Send a completion request through the router → **succeeds** (P logs `Sending to ... done, took 0.18 s`). 3. Wait ~5 s. Send it again → **hangs until client timeout**. P side: ``` W tcp_transport_lane_impl.h:1330] TCP lane queue-full rejection count: 1 W tcp_transport_lane_impl.h:1330] TCP lane queue-full rejection count: 2 ... doubling up to 512+ within one millisecond ... E mooncake_connector.py:1877] pulling kv_caches for ['cmpl-...'] failed: Mooncake transfer engine returned -1 ``` 4. Send it again ~90 s after step 3 → **succeeds again**. Consistently reproduced on two machines (H20 and A100), multiple runs. Also reproduced with `MC_TCP_ADMISSION_TIMEOUT_MS=60000` (failure shape changes: the batch is not hard-rejected but still never transfers). ## Analysis After a transfer completes and the peer connection group goes idle, the group is **retired** — all lanes are closed and `group->state` is set to `CLOSED` (`tcp_transport_lane_impl.h`, retirement block). A submission that arrives shortly after finds the group via `state->groups.find(key)`, sees `group->state != GroupState::OPEN`, and is **hard-rejected with `SHUTDOWN`** (`enqueuePooledTransfer`) instead of re-creating the group and its connections. The retired group is not removed from `state->groups` in the retirement path, so submissions keep failing until something re-creates the group — which matches the observed ~90 s self-heal. Evidence summary: - First transfer after engine start always succeeds; the failure only appears after an idle gap - During the hang the four lane connections are `ESTABLISHED` with empty kernel queues → the stall is in TE user space (lane queue / admission), not the kernel - Failure depends on the admission timeout (default 1 s fails immediately; 60 s still hangs) but recovery time (~90 s) is unchanged — consistent with a stale `CLOSED` group rather than a transient queue backlog Suggested fix: in `enqueuePooledTransfer`, drop a non-`OPEN` group and re-create it (or remove retired groups from `state->groups` atomically when retiring) instead of rejecting the work. ## Workaround Retry the request after ~90 s (self-healed), or keep the peer continuously busy (impractical). No configuration knob we found avoids it.

bsakaguchi commented

2026-10-29 21:22:26 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 18, 2026):

Thanks for opening this issue, @CAICAIIs!

Field	Value
Issue	#4209
GitHub user ID	`39020005`
Reporter	@CAICAIIs

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 18, 2026): Thanks for opening this issue, @CAICAIIs! | Field | Value | |-------|-------| | **Issue** | #4209 | | **GitHub user ID** | `39020005` | | **Reporter** | @CAICAIIs | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:22:26 +00:00

Author

Owner

@KaroUniform commented on GitHub (Oct 7, 2026):

We hit what looks like the same failure in production (SGLang PD disaggregation, DeepSeek-V4.1-Flash, mooncake-transfer-engine-cuda13 0.3.13, MC_FORCE_TCP=1). After 1–2.5 h under load every KV transfer to the decode instance failed with TCP lane queue-full rejection count: … and never recovered until the pods were restarted. Below is the root cause we found. It is already fixed on main, but not in any release.

Root cause (v0.3.13, v0.3.13.post1, v0.3.14rc1). In TcpTransport::enqueuePooledTransfer (tcp_transport_lane_impl.h), requestGroupPumpLocked() runs for every submission, including one that is hard-rejected. It sets pump_scheduled = true whenever the queue is non-empty and no pump is scheduled. But the pump is only posted on the non-rejected path:

if (rejected)
    failWorkItem(std::move(*rejected), rejection_reason, state->failure_counters);
else if (pump_epoch != 0)          // pump requested above is dropped on rejection
    postGroupPump(group, pump_epoch);

When a rejection happens right after a pump run has cleared pump_scheduled (queue still non-empty), the flag stays true with no pump posted. Every later requestGroupPumpLocked() returns 0, so the lanes go idle and the queue (1024 entries) never drains. From then on, every submission to that peer is either hard-rejected (QUEUE_FULL) or times out in pending admission after MC_TCP_ADMISSION_TIMEOUT_MS (QUEUE_TIMEOUT). A larger admission timeout only changes the failure shape: the work waits for queue room that never frees, which fits your 60 s observation. The group stays OPEN the whole time, so re-creating non-OPEN groups would not help this path. A single batch_transfer_sync_write that exceeds the 1024 + 1024 caps by a few hundred entries triggers it reliably. In our tests one thread is enough, and a batch of 2300 or more entries wedged the peer every time.

Evidence. On a wedged sender, gdb shows requestGroupPumpLocked with state=OPEN, pump_scheduled=1, a pump_epoch that stays frozen across probes, exactly 1024 queued items and all four lanes IDLE, and no postGroupPump/runGroupPump. A fresh engine in another process transfers to the same receiver fine.

Minimal repro (two processes, host memory, no GPU, default caps):

# python repro.py   (MC_FORCE_TCP=1)
import multiprocessing as mp, time, numpy as np
ITEM, BIG, SMALL = 64 * 1024, 6000, 16

def engine():
    from mooncake.engine import TransferEngine
    te = TransferEngine(); assert te.initialize("127.0.0.1", "P2PHANDSHAKE", "tcp", "") == 0
    return te

def receiver(conn):
    te = engine(); buf = np.zeros(ITEM * BIG, dtype=np.uint8)
    assert te.register_memory(buf.ctypes.data, buf.nbytes) == 0
    conn.send((f"127.0.0.1:{te.get_rpc_port()}", buf.ctypes.data)); conn.recv()

if __name__ == "__main__":
    mp.set_start_method("spawn")
    parent, child = mp.Pipe(); mp.Process(target=receiver, args=(child,), daemon=True).start()
    peer, dst = parent.recv(); te = engine()
    src = np.ones(ITEM * BIG, dtype=np.uint8); assert te.register_memory(src.ctypes.data, src.nbytes) == 0
    def send(n):
        t = time.time()
        r = te.batch_transfer_sync_write(peer, [src.ctypes.data + i * ITEM for i in range(n)],
                                         [dst + i * ITEM for i in range(n)], [ITEM] * n)
        return r, round(time.time() - t, 3)
    print("small before:", send(SMALL)); print("big:", send(BIG)); time.sleep(3)
    for _ in range(3): print("small after:", send(SMALL))
    parent.send("done")

0.3.13 (3/3 runs):

small before: (0, 0.002)
big: (-1, 0.013)
small after: (-1, 1.0)
small after: (-1, 1.0)
small after: (-1, 1.0)

0.3.13 with the one-line fix below (3/3 runs):

small before: (0, 0.003)
big: (-1, 0.014)
small after: (0, 0.001)
small after: (0, 0.001)
small after: (0, 0.001)

Fix. Post the pump regardless of the rejection (if (pump_epoch != 0) postGroupPump(...)). This is exactly commit 74d26b56 "[TE] Preserve queued TCP pump after rejection", which landed on main as part of #4063 together with a regression test. We run 0.3.13 rebuilt with that single line in production, plus MC_TCP_MAX_QUEUED_TRANSFERS_PER_PEER / MC_TCP_MAX_PENDING_ADMISSIONS_PER_PEER = 65535 (the defaults since #3710) to avoid the rejections themselves.

Request. Could this be shipped in a patch release (0.3.13.post2 or 0.3.14)? Every released wheel can permanently wedge a PD pair on TCP after a single oversized batch. cc @adenzhou1350 (author of 74d26b56) @alogfans

Caveat: this explains the queue-full storm and the "never transfers" part of the report. We did not reproduce your ~90 s self-recovery. In our case the wedge lasted until restart, and a peer-key change does not retire the stuck group. So something on the vLLM side, for example re-opening the segment, may be involved there.

@KaroUniform commented on GitHub (Oct 7, 2026): We hit what looks like the same failure in production (SGLang PD disaggregation, DeepSeek-V4.1-Flash, `mooncake-transfer-engine-cuda13` 0.3.13, `MC_FORCE_TCP=1`). After 1–2.5 h under load every KV transfer to the decode instance failed with `TCP lane queue-full rejection count: …` and never recovered until the pods were restarted. Below is the root cause we found. It is already fixed on `main`, but not in any release. **Root cause (v0.3.13, v0.3.13.post1, v0.3.14rc1).** In `TcpTransport::enqueuePooledTransfer` (`tcp_transport_lane_impl.h`), `requestGroupPumpLocked()` runs for every submission, including one that is hard-rejected. It sets `pump_scheduled = true` whenever the queue is non-empty and no pump is scheduled. But the pump is only posted on the non-rejected path: ```cpp if (rejected) failWorkItem(std::move(*rejected), rejection_reason, state->failure_counters); else if (pump_epoch != 0) // pump requested above is dropped on rejection postGroupPump(group, pump_epoch); ``` When a rejection happens right after a pump run has cleared `pump_scheduled` (queue still non-empty), the flag stays `true` with no pump posted. Every later `requestGroupPumpLocked()` returns 0, so the lanes go idle and the queue (1024 entries) never drains. From then on, every submission to that peer is either hard-rejected (`QUEUE_FULL`) or times out in pending admission after `MC_TCP_ADMISSION_TIMEOUT_MS` (`QUEUE_TIMEOUT`). A larger admission timeout only changes the failure shape: the work waits for queue room that never frees, which fits your 60 s observation. The group stays `OPEN` the whole time, so re-creating non-`OPEN` groups would not help this path. A single `batch_transfer_sync_write` that exceeds the 1024 + 1024 caps by a few hundred entries triggers it reliably. In our tests one thread is enough, and a batch of 2300 or more entries wedged the peer every time. **Evidence.** On a wedged sender, gdb shows `requestGroupPumpLocked` with `state=OPEN`, `pump_scheduled=1`, a `pump_epoch` that stays frozen across probes, exactly 1024 queued items and all four lanes `IDLE`, and no `postGroupPump`/`runGroupPump`. A fresh engine in another process transfers to the same receiver fine. **Minimal repro** (two processes, host memory, no GPU, default caps): ```python # python repro.py (MC_FORCE_TCP=1) import multiprocessing as mp, time, numpy as np ITEM, BIG, SMALL = 64 * 1024, 6000, 16 def engine(): from mooncake.engine import TransferEngine te = TransferEngine(); assert te.initialize("127.0.0.1", "P2PHANDSHAKE", "tcp", "") == 0 return te def receiver(conn): te = engine(); buf = np.zeros(ITEM * BIG, dtype=np.uint8) assert te.register_memory(buf.ctypes.data, buf.nbytes) == 0 conn.send((f"127.0.0.1:{te.get_rpc_port()}", buf.ctypes.data)); conn.recv() if __name__ == "__main__": mp.set_start_method("spawn") parent, child = mp.Pipe(); mp.Process(target=receiver, args=(child,), daemon=True).start() peer, dst = parent.recv(); te = engine() src = np.ones(ITEM * BIG, dtype=np.uint8); assert te.register_memory(src.ctypes.data, src.nbytes) == 0 def send(n): t = time.time() r = te.batch_transfer_sync_write(peer, [src.ctypes.data + i * ITEM for i in range(n)], [dst + i * ITEM for i in range(n)], [ITEM] * n) return r, round(time.time() - t, 3) print("small before:", send(SMALL)); print("big:", send(BIG)); time.sleep(3) for _ in range(3): print("small after:", send(SMALL)) parent.send("done") ``` 0.3.13 (3/3 runs): ``` small before: (0, 0.002) big: (-1, 0.013) small after: (-1, 1.0) small after: (-1, 1.0) small after: (-1, 1.0) ``` 0.3.13 with the one-line fix below (3/3 runs): ``` small before: (0, 0.003) big: (-1, 0.014) small after: (0, 0.001) small after: (0, 0.001) small after: (0, 0.001) ``` **Fix.** Post the pump regardless of the rejection (`if (pump_epoch != 0) postGroupPump(...)`). This is exactly commit 74d26b56 "[TE] Preserve queued TCP pump after rejection", which landed on `main` as part of #4063 together with a regression test. We run 0.3.13 rebuilt with that single line in production, plus `MC_TCP_MAX_QUEUED_TRANSFERS_PER_PEER` / `MC_TCP_MAX_PENDING_ADMISSIONS_PER_PEER` = 65535 (the defaults since #3710) to avoid the rejections themselves. **Request.** Could this be shipped in a patch release (0.3.13.post2 or 0.3.14)? Every released wheel can permanently wedge a PD pair on TCP after a single oversized batch. cc @adenzhou1350 (author of 74d26b56) @alogfans Caveat: this explains the `queue-full` storm and the "never transfers" part of the report. We did not reproduce your ~90 s self-recovery. In our case the wedge lasted until restart, and a peer-key change does not retire the stuck group. So something on the vLLM side, for example re-opening the segment, may be involved there.

bsakaguchi commented

2026-10-29 21:22:26 +00:00

Author

Owner

@soovv commented on GitHub (Oct 28, 2026):

Independent SGLang PD reproduction: released wheels are affected by the dropped-pump bug; the stale-CLOSED-group path does not apply

We hit this in a single-node prefill/decode deployment and independently reproduced the released-wheel failure. Posting the data because it changes the conclusion in the original report: the root cause in 0.3.13.post1 is the dropped pump epoch on rejection, not a retired group left in state->groups.

Environment

SGLang 0.5.22 (lmsysorg/sglang:latest), Qwen3.8-27B-NVFP4 + DFlash draft model
2P2D: 2 prefill + 2 decode pods, one RTX PRO 6000 per pod, all on one node / NUMA0
Transport: mooncake_tcp, MC_TCP_ENABLE_CONNECTION_POOL=true,
MC_TCP_MAX_QUEUED_TRANSFERS_PER_PEER=262144,
MC_TCP_MAX_PENDING_ADMISSIONS_PER_PEER=262144
EvalScope 1.14 custom_multi_turn: 4 turns, 2,048 output tokens per turn;
32K / 64K / 128K rendered context; 2 repetitions; no PD restart between points

Results

Success rate (mooncake-transfer-engine-cuda13 0.3.13.post1, two reps):

Workload	Success rate
32K context, c16	98.4% / 86.7%
64K context, c8	84.4% / 96.9%
128K context, c4	59.4% / 65.6%
128K context, c8	42.2% / 39.1%

Failures show the same shape as the report:

KVTransferError: Failed to get kvcache from prefill instance, it might be dead
TCP lane queue-full rejection count: 256
TCP lane queue-full rejection count: 512
Failed to send kv chunk ... to <decode>

The prefill session is blacklisted; the SGLang failed-session probe can clear the
blacklist (recovered via probe; un-blacklisted) but the next transfer fails again.

A build of main 179ffb2 (CUDA 13 wheel, same SGLang setup, default connection
pool and queue 262144, no restarts) completed all four points with zero failures:
128/128, 64/64, 32/32, 64/64.

Root cause: pump dropped on the rejection path

In 0.3.13.post1, enqueuePooledTransfer() requests a pump for every submission,
including one that is hard-rejected, but posts it only on the non-rejected path:

if (rejected)
    failWorkItem(std::move(*rejected), rejection_reason,
                 state->failure_counters);
else if (pump_epoch != 0)          // pump requested above is dropped
    postGroupPump(group, pump_epoch);

requestGroupPumpLocked() has already set pump_scheduled = true when the queue is
non-empty. If the pump is not posted, the flag stays true, every later
requestGroupPumpLocked() returns 0, and the queue never drains. This matches the
queue-full storm and the permanent blacklist. The fix is 74d26b56
("[TE] Preserve queued TCP pump after rejection"), merged in #4063, and it is
present in v0.3.14-pre:

if (pump_epoch != 0) postGroupPump(group, pump_epoch);

The stale-CLOSED-group explanation does not apply

The retired group is already removed from state->groups before it is marked
retiring, in both versions:

v0.3.13.post1: state->groups.erase(old_group_it); is immediately followed by
retiring_group->retiring = true; in the key-change block of
enqueuePooledTransfer().
main 179ffb2: same block, state->groups.erase(old_group_it); at
tcp_transport_lane_impl.h:600, retiring_group->retiring = true; at line 603.

retiring = true has no other assignment, and GroupState::CLOSED is only set in
runGroupRetirement() (and shutdown, which clears the maps). Therefore
runGroupRetirement() cannot observe a group that is still in state->groups; an
extra erase there is unreachable. We initially carried a 6-line patch that did this;
after the source audit we dropped it. It cannot be the reason the main build
passes — main itself contains 74d26b56.

Request: cut a patch release

PyPI still serves 0.3.13.post1 for mooncake-transfer-engine-cuda13 (checked
2026-10-28); v0.3.14-pre is GitHub-only. Every released TCP wheel can wedge a
PD pair after a single oversized batch. Could you cut a patch release
(0.3.13.post2 or 0.3.14) containing 74d26b56 and #4234?

The MC_TCP_ENABLE_CONNECTION_POOL=false + larger per-peer queues + periodic
restart combination is only a workaround; the pump fix removes the wedge.

We can provide the EvalScope JSONL/repro scripts and the full prefill/decode logs.

@soovv commented on GitHub (Oct 28, 2026): ### Independent SGLang PD reproduction: released wheels are affected by the dropped-pump bug; the stale-CLOSED-group path does not apply We hit this in a single-node prefill/decode deployment and independently reproduced the released-wheel failure. Posting the data because it changes the conclusion in the original report: the root cause in `0.3.13.post1` is the **dropped pump epoch on rejection**, not a retired group left in `state->groups`. #### Environment - SGLang 0.5.22 (`lmsysorg/sglang:latest`), Qwen3.8-27B-NVFP4 + DFlash draft model - 2P2D: 2 prefill + 2 decode pods, one RTX PRO 6000 per pod, all on one node / NUMA0 - Transport: `mooncake_tcp`, `MC_TCP_ENABLE_CONNECTION_POOL=true`, `MC_TCP_MAX_QUEUED_TRANSFERS_PER_PEER=262144`, `MC_TCP_MAX_PENDING_ADMISSIONS_PER_PEER=262144` - EvalScope 1.14 `custom_multi_turn`: 4 turns, 2,048 output tokens per turn; 32K / 64K / 128K rendered context; 2 repetitions; no PD restart between points #### Results Success rate (`mooncake-transfer-engine-cuda13 0.3.13.post1`, two reps): | Workload | Success rate | | --- | ---: | | 32K context, c16 | 98.4% / 86.7% | | 64K context, c8 | 84.4% / 96.9% | | 128K context, c4 | 59.4% / 65.6% | | 128K context, c8 | 42.2% / 39.1% | Failures show the same shape as the report: ``` KVTransferError: Failed to get kvcache from prefill instance, it might be dead TCP lane queue-full rejection count: 256 TCP lane queue-full rejection count: 512 Failed to send kv chunk ... to <decode> ``` The prefill session is blacklisted; the SGLang failed-session probe can clear the blacklist (`recovered via probe; un-blacklisted`) but the next transfer fails again. A build of `main` `179ffb2` (CUDA 13 wheel, same SGLang setup, default connection pool and queue 262144, no restarts) completed all four points with **zero failures**: 128/128, 64/64, 32/32, 64/64. #### Root cause: pump dropped on the rejection path In `0.3.13.post1`, `enqueuePooledTransfer()` requests a pump for every submission, including one that is hard-rejected, but posts it only on the non-rejected path: ```cpp if (rejected) failWorkItem(std::move(*rejected), rejection_reason, state->failure_counters); else if (pump_epoch != 0) // pump requested above is dropped postGroupPump(group, pump_epoch); ``` `requestGroupPumpLocked()` has already set `pump_scheduled = true` when the queue is non-empty. If the pump is not posted, the flag stays `true`, every later `requestGroupPumpLocked()` returns 0, and the queue never drains. This matches the queue-full storm and the permanent blacklist. The fix is `74d26b56` ("[TE] Preserve queued TCP pump after rejection"), merged in #4063, and it is present in `v0.3.14-pre`: ```cpp if (pump_epoch != 0) postGroupPump(group, pump_epoch); ``` #### The stale-CLOSED-group explanation does not apply The retired group is **already removed from `state->groups`** before it is marked retiring, in both versions: - `v0.3.13.post1`: `state->groups.erase(old_group_it);` is immediately followed by `retiring_group->retiring = true;` in the key-change block of `enqueuePooledTransfer()`. - `main` `179ffb2`: same block, `state->groups.erase(old_group_it);` at `tcp_transport_lane_impl.h:600`, `retiring_group->retiring = true;` at line 603. `retiring = true` has no other assignment, and `GroupState::CLOSED` is only set in `runGroupRetirement()` (and shutdown, which clears the maps). Therefore `runGroupRetirement()` cannot observe a group that is still in `state->groups`; an extra erase there is unreachable. We initially carried a 6-line patch that did this; after the source audit we dropped it. It cannot be the reason the `main` build passes — `main` itself contains `74d26b56`. #### Request: cut a patch release PyPI still serves `0.3.13.post1` for `mooncake-transfer-engine-cuda13` (checked 2026-10-28); `v0.3.14-pre` is GitHub-only. Every released TCP wheel can wedge a PD pair after a single oversized batch. Could you cut a patch release (`0.3.13.post2` or `0.3.14`) containing `74d26b56` and #4234? The `MC_TCP_ENABLE_CONNECTION_POOL=false` + larger per-peer queues + periodic restart combination is only a workaround; the pump fix removes the wedge. We can provide the EvalScope JSONL/repro scripts and the full prefill/decode logs.

bsakaguchi referenced this issue

2026-10-29 21:31:53 +00:00

[PR #1057] [MERGED] handling cudaMemcpy errors in tcp_transport.cpp #1984