[GH-ISSUE #2930] [Bug] TcpTransport connection pool: unbounded growth with threshold mass-GC; stale entries to restarted peers are never evicted #760

New issue

Open

opened 2026-10-29 21:19:13 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented 2026-10-29 21:19:13 +00:00

Owner

Copy link

Originally created by @d3nb on GitHub (Jul 15, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2930

Originally assigned to: @alogfans on GitHub.

Environment

mooncake-transfer-engine-cuda13 0.3.11.post1 (PyPI), used as SGLang PD-disaggregation KV transfer backend (MC_FORCE_TCP=1, MC_TCP_ENABLE_CONNECTION_POOL=true), same-host prefill(TP4) → decode(TP4), 8×B300, container network.

Issue 1: pool grows unboundedly, then mass-closes tens of thousands of connections at once

Under sustained load (fixed 10-way concurrency, ~27K-token KV parcels, chunked), /proc/net/tcp on the sender side shows a sawtooth with ~8-minute period:

03:15  ESTABLISHED 29024
03:17  ESTABLISHED 38952
03:18  ESTABLISHED 17724   TIME_WAIT 21089   <- ~21K connections closed within one minute
03:24  ESTABLISHED 44419
03:25  ESTABLISHED 45623   <- 83% of the ephemeral port range
03:26  ESTABLISHED 20738   TIME_WAIT 26831   <- ~27K closed at once

Two problems: (a) between flushes the pool keeps growing toward the ephemeral-port limit (we measured peaks at 83% of a widened 10240-65000 range — with the default ~28K-port range this would hit EADDRNOTAVAIL long before the flush); (b) the flush itself dumps 20K+ sockets into TIME_WAIT within seconds. A bounded pool with per-peer connection reuse would presumably hold this at a few hundred connections (we see ~566 ESTABLISHED steady when idle-ish).

Issue 2: pooled connections to a dead/restarted peer are never evicted

If one side of the PD pair is restarted alone (its listen ports change), the surviving side's pool keeps the stale entries. Small transfers still succeed (presumably via new connections), but large multi-slice transfers stall silently and permanently — the sender reports no error, the receiver times out (KVPoll.WaitingForInput timeout after 300s). It also degrades progressively: after one single-sided peer restart we only measured throughput loss at high concurrency; after a second one, all large transfers hung. Restarting both sides together fully restores normal behavior.

Ask

1. Bound the pool (per-peer cap + LRU reuse) instead of grow-then-mass-GC.
2. Evict/reconnect pool entries on connection failure or peer-identity change, so a peer restart doesn't silently poison large transfers.

Happy to provide more traces or test patches — this is reproducible on demand in our setup.

Originally created by @d3nb on GitHub (Jul 15, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2930 Originally assigned to: @alogfans on GitHub. ## Environment mooncake-transfer-engine-cuda13 0.3.11.post1 (PyPI), used as SGLang PD-disaggregation KV transfer backend (`MC_FORCE_TCP=1`, `MC_TCP_ENABLE_CONNECTION_POOL=true`), same-host prefill(TP4) → decode(TP4), 8×B300, container network. ## Issue 1: pool grows unboundedly, then mass-closes tens of thousands of connections at once Under sustained load (fixed 10-way concurrency, ~27K-token KV parcels, chunked), `/proc/net/tcp` on the sender side shows a sawtooth with ~8-minute period: ```text 03:15 ESTABLISHED 29024 03:17 ESTABLISHED 38952 03:18 ESTABLISHED 17724 TIME_WAIT 21089 <- ~21K connections closed within one minute 03:24 ESTABLISHED 44419 03:25 ESTABLISHED 45623 <- 83% of the ephemeral port range 03:26 ESTABLISHED 20738 TIME_WAIT 26831 <- ~27K closed at once ``` Two problems: (a) between flushes the pool keeps growing toward the ephemeral-port limit (we measured peaks at 83% of a widened 10240-65000 range — with the default ~28K-port range this would hit `EADDRNOTAVAIL` long before the flush); (b) the flush itself dumps 20K+ sockets into TIME_WAIT within seconds. A bounded pool with per-peer connection reuse would presumably hold this at a few hundred connections (we see ~566 ESTABLISHED steady when idle-ish). ## Issue 2: pooled connections to a dead/restarted peer are never evicted If one side of the PD pair is restarted alone (its listen ports change), the surviving side's pool keeps the stale entries. Small transfers still succeed (presumably via new connections), but **large multi-slice transfers stall silently and permanently** — the sender reports no error, the receiver times out (`KVPoll.WaitingForInput` timeout after 300s). It also degrades progressively: after one single-sided peer restart we only measured throughput loss at high concurrency; after a second one, all large transfers hung. Restarting both sides together fully restores normal behavior. ## Ask 1. Bound the pool (per-peer cap + LRU reuse) instead of grow-then-mass-GC. 2. Evict/reconnect pool entries on connection failure or peer-identity change, so a peer restart doesn't silently poison large transfers. Happy to provide more traces or test patches — this is reproducible on demand in our setup.

bsakaguchi commented 2026-10-29 21:19:14 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 15, 2026):

Thanks for opening this issue, @d3nb!

Field	Value
Issue	#2930
GitHub user ID	7304667
Reporter	@d3nb

A maintainer will triage this when possible. To help us respond faster, please include:

• Mooncake version or commit SHA
• Environment (OS, CUDA/driver, RDMA stack if relevant)
• Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

<!-- gh-comment-id:4977439917 --> @github-actions[bot] commented on GitHub (Jul 15, 2026): Thanks for opening this issue, @d3nb! | Field | Value | |-------|-------| | **Issue** | #2930 | | **GitHub user ID** | `7304667` | | **Reporter** | @d3nb | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented 2026-10-29 21:19:14 +00:00

Author

Owner

Copy link

@d3nb commented on GitHub (Jul 15, 2026):

Adding the triage details requested by the bot:

Mooncake version: mooncake-transfer-engine-cuda13 0.3.11.post1 (PyPI wheel, as bundled in the SGLang image lmsysorg/sglang:nightly-dev-cu13-20260713-b94ac87e).

Environment:

• Single node, 8×B300, two containers (prefill / decode) on a docker bridge network
• Container OS: Ubuntu 24.04.3 LTS; host kernel 6.8.0-101-generic
• CUDA 13.0 (V13.0.88), driver 590.48.01
• Transport: TCP (mooncake_tcp, MC_TCP_ENABLE_CONNECTION_POOL=true); the RDMA path is parked on this build due to a "Local segment descriptor not found" regression on long multi-segment transfers (#2177-shaped), so everything below is pure TcpTransport
• Widened container port range net.ipv4.ip_local_port_range = 10240 65000, tcp_tw_reuse=1

Repro: SGLang PD disaggregation (1 prefill + 1 decode, DeepSeek-V4 class model, KV transfer via TcpTransport), then any benchmark that ramps concurrency (we use aiperf max-concurrency-under-sla over a ~5000-request trace with ~27K-token mean prompts). Watch ss -s / per-state counts on either worker.

Actual:

1. ESTABLISHED count climbs monotonically to ~45.6K (~83% of the widened port range), then roughly every ~8 min a threshold GC flushes 21-27K connections per minute into TIME_WAIT, then the climb restarts. Under default port range this manifests as EADDRNOTAVAIL instead (that's what forced the pool on).
2. After restarting one peer, the surviving peer's pool keeps stale entries for the dead endpoint indefinitely — transfers to the restarted peer keep failing until both sides are restarted. We now operate PD workers under a mandatory paired-restart rule solely because of this.

Expected: bounded pool with idle/LRU eviction and reuse (no sawtooth mass-GC), and eviction of pool entries on connection error so a restarted peer heals without restarting the survivor.

<!-- gh-comment-id:4981463444 --> @d3nb commented on GitHub (Jul 15, 2026): Adding the triage details requested by the bot: **Mooncake version**: `mooncake-transfer-engine-cuda13 0.3.11.post1` (PyPI wheel, as bundled in the SGLang image `lmsysorg/sglang:nightly-dev-cu13-20260713-b94ac87e`). **Environment**: - Single node, 8×B300, two containers (prefill / decode) on a docker bridge network - Container OS: Ubuntu 24.04.3 LTS; host kernel 6.8.0-101-generic - CUDA 13.0 (V13.0.88), driver 590.48.01 - Transport: **TCP** (`mooncake_tcp`, `MC_TCP_ENABLE_CONNECTION_POOL=true`); the RDMA path is parked on this build due to a "Local segment descriptor not found" regression on long multi-segment transfers (#2177-shaped), so everything below is pure TcpTransport - Widened container port range `net.ipv4.ip_local_port_range = 10240 65000`, `tcp_tw_reuse=1` **Repro**: SGLang PD disaggregation (1 prefill + 1 decode, DeepSeek-V4 class model, KV transfer via TcpTransport), then any benchmark that ramps concurrency (we use aiperf max-concurrency-under-sla over a ~5000-request trace with ~27K-token mean prompts). Watch `ss -s` / per-state counts on either worker. **Actual**: 1. ESTABLISHED count climbs monotonically to ~45.6K (~83% of the widened port range), then roughly every ~8 min a threshold GC flushes 21-27K connections per minute into TIME_WAIT, then the climb restarts. Under default port range this manifests as `EADDRNOTAVAIL` instead (that's what forced the pool on). 2. After restarting one peer, the surviving peer's pool keeps stale entries for the dead endpoint indefinitely — transfers to the restarted peer keep failing until *both* sides are restarted. We now operate PD workers under a mandatory paired-restart rule solely because of this. **Expected**: bounded pool with idle/LRU eviction and reuse (no sawtooth mass-GC), and eviction of pool entries on connection error so a restarted peer heals without restarting the survivor.

bsakaguchi commented 2026-10-29 21:19:14 +00:00

Author

Owner

Copy link

@alogfans commented on GitHub (Jul 16, 2026):

Thanks for the detailed repro and socket-state timeline.

The pool is currently keyed by host/port and does not have a per-peer or global bound. When all pooled sockets for a peer are in use, getConnection() creates another socket and appends it to the peer queue. Under sustained chunked KV transfers, especially with many concurrent slices, that means the number of ESTABLISHED sockets can grow with workload pressure instead of being capped at a small reuse set.

The sawtooth/TIME_WAIT burst is also consistent with the cleanup behavior. Idle cleanup is triggered lazily from getConnection(), and expired idle sockets can be closed in one cleanup pass. So after a burst of transfers completes, many sockets can become idle around the same time, and a later cleanup pass may close a large batch together, producing the TIME_WAIT spike you measured.

For the restart case, socket->is_open() is not a sufficient liveness check for a pooled TCP connection. A socket can still look open locally after the peer process has exited or after a single-sided restart. The current pool also does not track a peer instance/generation, so it cannot reliably distinguish “same host/port, old peer” from “same logical peer after restart”. That can leave stale pooled entries eligible for reuse until a later async read/write path detects the failure, which is especially painful for large multi-slice transfers.

Planned fixes:

1. Bound the pool

   • Add a configurable per-peer connection cap.
   • Reuse idle connections before creating new sockets.
   • When the cap is reached, evict the oldest idle connection if possible.
   • If all peer connections are busy, use bounded waiting/backpressure or fail fast instead of opening unbounded sockets.

2. Make cleanup gradual

   • Track last-used time and use LRU-style idle eviction.
   • Limit the number of sockets closed in a single cleanup pass.
   • Avoid doing socket close work while holding the pool mutex where possible.

3. Evict bad connections reliably

   • Remove pooled connections on read/write/protocol/timeout failure.
   • Ensure failed sockets are discarded rather than returned to the pool.
   • Add transfer or per-chunk deadlines so large transfers fail visibly instead of hanging forever.

4. Handle peer restart

   • Clear the affected peer’s pool when its advertised TCP endpoint changes.
   • Longer term, add a peer instance ID/generation to segment metadata and include it in the pool key, so connections from a previous peer process cannot be reused after restart.

As this may incur significant changes of the tcp transport, we will perform these fixes seperately.

<!-- gh-comment-id:4987658288 --> @alogfans commented on GitHub (Jul 16, 2026): Thanks for the detailed repro and socket-state timeline. The pool is currently keyed by host/port and does not have a per-peer or global bound. When all pooled sockets for a peer are in use, getConnection() creates another socket and appends it to the peer queue. Under sustained chunked KV transfers, especially with many concurrent slices, that means the number of ESTABLISHED sockets can grow with workload pressure instead of being capped at a small reuse set. The sawtooth/TIME_WAIT burst is also consistent with the cleanup behavior. Idle cleanup is triggered lazily from getConnection(), and expired idle sockets can be closed in one cleanup pass. So after a burst of transfers completes, many sockets can become idle around the same time, and a later cleanup pass may close a large batch together, producing the TIME_WAIT spike you measured. For the restart case, socket->is_open() is not a sufficient liveness check for a pooled TCP connection. A socket can still look open locally after the peer process has exited or after a single-sided restart. The current pool also does not track a peer instance/generation, so it cannot reliably distinguish “same host/port, old peer” from “same logical peer after restart”. That can leave stale pooled entries eligible for reuse until a later async read/write path detects the failure, which is especially painful for large multi-slice transfers. Planned fixes: 1. Bound the pool - Add a configurable per-peer connection cap. - Reuse idle connections before creating new sockets. - When the cap is reached, evict the oldest idle connection if possible. - If all peer connections are busy, use bounded waiting/backpressure or fail fast instead of opening unbounded sockets. 2. Make cleanup gradual - Track last-used time and use LRU-style idle eviction. - Limit the number of sockets closed in a single cleanup pass. - Avoid doing socket close work while holding the pool mutex where possible. 3. Evict bad connections reliably - Remove pooled connections on read/write/protocol/timeout failure. - Ensure failed sockets are discarded rather than returned to the pool. - Add transfer or per-chunk deadlines so large transfers fail visibly instead of hanging forever. 4. Handle peer restart - Clear the affected peer’s pool when its advertised TCP endpoint changes. - Longer term, add a peer instance ID/generation to segment metadata and include it in the pool key, so connections from a previous peer process cannot be reused after restart. As this may incur significant changes of the tcp transport, we will perform these fixes seperately.

bsakaguchi commented 2026-10-29 21:19:15 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Oct 14, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:6160062349 --> @github-actions[bot] commented on GitHub (Oct 14, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:19:15 +00:00

Author

Owner

Copy link

@d3nb commented on GitHub (Oct 18, 2026):

Quick status from our side, since the stale bot is counting down: the fixes that came out of this thread have landed.

• #2974 (bounded per-peer connection lanes), #3542 (session progress deadline) and #3543 (endpoint refresh correctness) are all contained in v0.3.13 / v0.3.13.post1.
• #4063 (queued-bytes admission bound) merged on 09-24, so it is not in any tagged release yet.

The measurements in this issue were taken on 0.3.11.post1. We'll re-run the same socket-state timeline (ESTABLISHED growth, TIME_WAIT bursts, single-side restart) on 0.3.13.post1 and report the result here. If it reproduces nothing, we'll close this ourselves; leaving it open until then.

Thanks @alogfans for taking the four planned fixes through.

<!-- gh-comment-id:6248281468 --> @d3nb commented on GitHub (Oct 18, 2026): Quick status from our side, since the stale bot is counting down: the fixes that came out of this thread have landed. - [#2974](https://github.com/kvcache-ai/Mooncake/pull/2974) (bounded per-peer connection lanes), [#3542](https://github.com/kvcache-ai/Mooncake/pull/3542) (session progress deadline) and [#3543](https://github.com/kvcache-ai/Mooncake/pull/3543) (endpoint refresh correctness) are all contained in `v0.3.13` / `v0.3.13.post1`. - [#4063](https://github.com/kvcache-ai/Mooncake/pull/4063) (queued-bytes admission bound) merged on 09-24, so it is not in any tagged release yet. The measurements in this issue were taken on `0.3.11.post1`. We'll re-run the same socket-state timeline (ESTABLISHED growth, TIME_WAIT bursts, single-side restart) on `0.3.13.post1` and report the result here. If it reproduces nothing, we'll close this ourselves; leaving it open until then. Thanks @alogfans for taking the four planned fixes through.

bsakaguchi referenced this issue 2026-10-29 21:30:28 +00:00

[PR #760] [MERGED] [Store] Serialize/Deserialize Offset Allocator #1802

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#760

No description provided.