kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #691] [Bug]: PD failing at long context/bigger batches #222

New issue
Closed
opened 2026-10-29 21:11:05 +00:00 by bsakaguchi · 12 comments
bsakaguchi commented 2026-10-29 21:11:05 +00:00
Owner
Copy link

Originally created by @tugot17 on GitHub (Jul 29, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/691

Bug Report

Hi,

I'm currently testing the PD performance in SGLang at larger batches. The motivation being, this is a worker profile I expect to encounter in (agentic) chat. I think there is some error, but I struggle to identify the source based on the available logs.

Steps to replicate on a single H100 node (should also work with less compute if you change the TP size/smaller model):

run_decode.sh

CUDA_VISIBLE_DEVICES=4,5,6,7 python -m sglang.launch_server --model-path Qwen/Qwen3-32B-FP8 --disaggregation-mode decode --port 30001 --host 10.1.0.21 --tp-size 4 --kv-cache-dtype fp8_e5m2

run_prefill.sh

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server --model-path Qwen/Qwen3-32B-FP8 --disaggregation-mode prefill --port 30000 --host 10.1.0.21 --tp-size 4 --kv-cache-dtype fp8_e5m2

run_proxy.sh

python3 -m sglang.srt.disaggregation.mini_lb --prefill http://10.1.0.21:30000 --decode http://10.1.0.21:30001 --host 0.0.0.0 --port 8002

For a single simple setup this works fine, the problem starts when I try to simulate a batch of long context, e.g. 8x30k tokens in. Than it completely fails

import asyncio
import aiohttp
import json

async def send_request(session, request_id):
    # Generate ~30k tokens of text (roughly 120k characters)
    large_text = "This is a sample sentence for load testing. " * 2500
    
    payload = {
        "model": "Qwen/Qwen3-32B-FP8",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user", 
                "content": large_text
            }
        ],
        "max_tokens": 100,
        "temperature": 0.7
    }
    
    async with session.post(
        "http://10.1.0.21:8002/v1/chat/completions",
        json=payload,
        headers={"Content-Type": "application/json"}
    ) as response:
        result = await response.json()
        print(f"Request {request_id} completed: {response.status}")
        return result

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [send_request(session, i) for i in range(1, 32)]
        results = await asyncio.gather(*tasks)
        print(f"All {len(results)} requests completed")

if __name__ == "__main__":
    asyncio.run(main())

The first few requests complete fine:

Request 1 completed: 200
Request 2 completed: 200
Request 3 completed: 200
Request 4 completed: 200
Request 5 completed: 200

but later it just stalls.

The workers than fail to give any further responses. Errors:

decode:

E0729 05:04:30.527725 14093 rdma_endpoint.cpp:70] Failed to create QP: Cannot allocate memory [12]
Fatal Python error: Segmentation fault

Thread 0x00007f2a3ffff640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 324 in wait
  File "/usr/lib/python3.10/threading.py", line 607 in wait
  File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f2a53fff640 (most recent call first):
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 697 in heartbeat_checker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f2a5fffc640 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/socket.py", line 799 in recv_multipart
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 668 in decode_thread
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f2bf9fff640 (most recent call first):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2028 in watchdog_thread
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f2cc9fff640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 320 in wait
  File "/usr/lib/python3.10/queue.py", line 171 in get
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 141 in forward_thread_func_
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 129 in forward_thread_func
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f3c0dfff640 (most recent call first):
  File "/usr/lib/python3.10/threading.py", line 324 in wait
  File "/usr/lib/python3.10/threading.py", line 607 in wait
  File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f4028b62640 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 55 in _recv_msg
  File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 191 in _read_thread
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007f46290d8480 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2815 in all_reduce
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 81 in wrapper
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/utils.py", line 55 in poll_and_all_reduce
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/decode.py", line 555 in pop_transferred
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/decode.py", line 875 in process_decode_queue
  File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/decode.py", line 706 in event_loop_overlap_disagg_decode
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2746 in run_scheduler_process
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108 in run
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 129 in _main
  File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, uvloop.loop, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, psutil._psutil_posix, zmq.backend.cython._zmq, PIL._imaging, setproctitle._setproctitle, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, sentencepiece._sentencepiece, cuda.bindings._lib.utils, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, msgspec._core, cuda_utils, __triton_launcher (total: 110)
[2025-07-29 05:04:47] Child process unexpectedly failed with exitcode=139. pid=12409
[2025-07-29 05:04:49] Child process unexpectedly failed with exitcode=139. pid=12410
[2025-07-29 05:04:50] Child process unexpectedly failed with exitcode=139. pid=12407
[2025-07-29 05:04:52] Child process unexpectedly failed with exitcode=139. pid=12408

prefill just shows

ugin: connect()10.1.0.21:15468: Connection refused [111]
E0729 05:05:26.130410 13520 transfer_metadata_plugin.cpp:835] SocketHandShakePlugin: connect()10.1.0.21:15468: Connection refused [111]
E0729 05:05:26.131007 13507 transfer_metadata_plugin.cpp:835] SocketHandShakePlugin: connect()10.1.0.21:15468: Connection refused [111]
E0729 05:05:26.131594 13486 transfer_metadata_plugin.cpp:835] SocketHandShakePlugin: connect()10.1.0.21:15468: Connection refused [11

proxy doesn't show any errors.

I identified this error while running my benchmarking suite, but the above simplified setup also causes the same problem; hence, I included it as it is easier to replicate. It seems something under the hood fails when the KV cache size grows above some threshold.

Any ideas what might be an issue?

Originally created by @tugot17 on GitHub (Jul 29, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/691 ### Bug Report Hi, I'm currently testing the PD performance in SGLang at larger batches. The motivation being, this is a worker profile I expect to encounter in (agentic) chat. I think there is some error, but I struggle to identify the source based on the available logs. Steps to replicate on a single H100 node (should also work with less compute if you change the TP size/smaller model): `run_decode.sh` ```bash CUDA_VISIBLE_DEVICES=4,5,6,7 python -m sglang.launch_server --model-path Qwen/Qwen3-32B-FP8 --disaggregation-mode decode --port 30001 --host 10.1.0.21 --tp-size 4 --kv-cache-dtype fp8_e5m2 ``` `run_prefill.sh` ```bash CUDA_VISIBLE_DEVICES=0,1,2,3 python -m sglang.launch_server --model-path Qwen/Qwen3-32B-FP8 --disaggregation-mode prefill --port 30000 --host 10.1.0.21 --tp-size 4 --kv-cache-dtype fp8_e5m2 ``` `run_proxy.sh` ```bash python3 -m sglang.srt.disaggregation.mini_lb --prefill http://10.1.0.21:30000 --decode http://10.1.0.21:30001 --host 0.0.0.0 --port 8002 ``` For a single simple setup this works fine, the problem starts when I try to simulate a batch of long context, e.g. 8x30k tokens in. Than it completely fails ```py import asyncio import aiohttp import json async def send_request(session, request_id): # Generate ~30k tokens of text (roughly 120k characters) large_text = "This is a sample sentence for load testing. " * 2500 payload = { "model": "Qwen/Qwen3-32B-FP8", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": large_text } ], "max_tokens": 100, "temperature": 0.7 } async with session.post( "http://10.1.0.21:8002/v1/chat/completions", json=payload, headers={"Content-Type": "application/json"} ) as response: result = await response.json() print(f"Request {request_id} completed: {response.status}") return result async def main(): async with aiohttp.ClientSession() as session: tasks = [send_request(session, i) for i in range(1, 32)] results = await asyncio.gather(*tasks) print(f"All {len(results)} requests completed") if __name__ == "__main__": asyncio.run(main()) ``` The first few requests complete fine: ```bash Request 1 completed: 200 Request 2 completed: 200 Request 3 completed: 200 Request 4 completed: 200 Request 5 completed: 200 ``` but later it just stalls. The workers than fail to give any further responses. Errors: `decode`: ```bash E0729 05:04:30.527725 14093 rdma_endpoint.cpp:70] Failed to create QP: Cannot allocate memory [12] Fatal Python error: Segmentation fault Thread 0x00007f2a3ffff640 (most recent call first): File "/usr/lib/python3.10/threading.py", line 324 in wait File "/usr/lib/python3.10/threading.py", line 607 in wait File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f2a53fff640 (most recent call first): File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 697 in heartbeat_checker File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f2a5fffc640 (most recent call first): File "/usr/local/lib/python3.10/dist-packages/zmq/sugar/socket.py", line 799 in recv_multipart File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/mooncake/conn.py", line 668 in decode_thread File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f2bf9fff640 (most recent call first): File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2028 in watchdog_thread File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f2cc9fff640 (most recent call first): File "/usr/lib/python3.10/threading.py", line 320 in wait File "/usr/lib/python3.10/queue.py", line 171 in get File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 141 in forward_thread_func_ File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 129 in forward_thread_func File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f3c0dfff640 (most recent call first): File "/usr/lib/python3.10/threading.py", line 324 in wait File "/usr/lib/python3.10/threading.py", line 607 in wait File "/usr/local/lib/python3.10/dist-packages/tqdm/_monitor.py", line 60 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f4028b62640 (most recent call first): File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 55 in _recv_msg File "/usr/local/lib/python3.10/dist-packages/torch/_inductor/compile_worker/subproc_pool.py", line 191 in _read_thread File "/usr/lib/python3.10/threading.py", line 953 in run File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap Thread 0x00007f46290d8480 (most recent call first): File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2815 in all_reduce File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 81 in wrapper File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/utils.py", line 55 in poll_and_all_reduce File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/decode.py", line 555 in pop_transferred File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/decode.py", line 875 in process_decode_queue File "/sgl-workspace/sglang/python/sglang/srt/disaggregation/decode.py", line 706 in event_loop_overlap_disagg_decode File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116 in decorate_context File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2746 in run_scheduler_process File "/usr/lib/python3.10/multiprocessing/process.py", line 108 in run File "/usr/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap File "/usr/lib/python3.10/multiprocessing/spawn.py", line 129 in _main File "/usr/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main File "<string>", line 1 in <module> Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, uvloop.loop, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, psutil._psutil_posix, zmq.backend.cython._zmq, PIL._imaging, setproctitle._setproctitle, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._direct, sentencepiece._sentencepiece, cuda.bindings._lib.utils, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, msgspec._core, cuda_utils, __triton_launcher (total: 110) [2025-07-29 05:04:47] Child process unexpectedly failed with exitcode=139. pid=12409 [2025-07-29 05:04:49] Child process unexpectedly failed with exitcode=139. pid=12410 [2025-07-29 05:04:50] Child process unexpectedly failed with exitcode=139. pid=12407 [2025-07-29 05:04:52] Child process unexpectedly failed with exitcode=139. pid=12408 ``` `prefill` just shows ```bash ugin: connect()10.1.0.21:15468: Connection refused [111] E0729 05:05:26.130410 13520 transfer_metadata_plugin.cpp:835] SocketHandShakePlugin: connect()10.1.0.21:15468: Connection refused [111] E0729 05:05:26.131007 13507 transfer_metadata_plugin.cpp:835] SocketHandShakePlugin: connect()10.1.0.21:15468: Connection refused [111] E0729 05:05:26.131594 13486 transfer_metadata_plugin.cpp:835] SocketHandShakePlugin: connect()10.1.0.21:15468: Connection refused [11 ``` `proxy` doesn't show any errors. I identified this error while running my [benchmarking suite](https://github.com/tugot17/tokenomics), but the above simplified setup also causes the same problem; hence, I included it as it is easier to replicate. It seems something under the hood fails when the KV cache size grows above some threshold. Any ideas what might be an issue?
bsakaguchi 2026-10-29 21:11:05 +00:00
  • closed this issue
  • added the
    bug
         label
bsakaguchi commented 2026-10-29 21:11:05 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Jul 31, 2025):

E0729 05:04:30.527725 14093 rdma_endpoint.cpp:70] Failed to create QP: Cannot allocate memory [12]

It seems to be related to insufficient memory. Please check your system using the commands ulimit -a and free -g. In addition, you can try using rdma resource while reproducing this issue to observe RDMA resource usage.

<!-- gh-comment-id:3138382988 --> @staryxchen commented on GitHub (Jul 31, 2025): ``` E0729 05:04:30.527725 14093 rdma_endpoint.cpp:70] Failed to create QP: Cannot allocate memory [12] ``` It seems to be related to insufficient memory. Please check your system using the commands `ulimit -a` and `free -g`. In addition, you can try using `rdma resource` while reproducing this issue to observe RDMA resource usage.
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@tugot17 commented on GitHub (Jul 31, 2025):

I don't think it is the memory issue, but rather the issue with exploding number of QPs?

Normal baseline (03:01:21):

=== Thu Jul 31 03:01:21 AM PDT 2025 ===
--- Memory ---
               total        used        free      shared  buff/cache   available
Mem:            1007          19         808           1         179         980
Swap:              1           1           0
--- RDMA Resources ---
0: mlx5_0: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 
1: mlx5_1: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 
2: mlx5_2: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
4: mlx5_4: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 
5: mlx5_5: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2

QP explosion begins (03:02:21):

=== Thu Jul 31 03:02:21 AM PDT 2025 ===
--- Memory ---
               total        used        free      shared  buff/cache   available
Mem:            1007          61         766           1         179         938
Swap:              1           1           0
--- RDMA Resources ---
0: mlx5_0: pd 14 cq 178 qp 946 cm_id 0 mr 1086 ctx 18 srq 2 
1: mlx5_1: pd 14 cq 178 qp 937 cm_id 0 mr 1086 ctx 18 srq 2 
2: mlx5_2: pd 11 cq 12 qp 805 cm_id 0 mr 1080 ctx 17 srq 2 
3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
4: mlx5_4: pd 14 cq 178 qp 947 cm_id 0 mr 1086 ctx 18 srq 2 
5: mlx5_5: pd 14 cq 178 qp 939 cm_id 0 mr 1086 ctx 18 srq 2

Peak QP usage before crash (03:05:32):

=== Thu Jul 31 03:05:32 AM PDT 2025 ===
--- Memory ---
               total        used        free      shared  buff/cache   available
Mem:            1007          74         751           1         180         925
Swap:              1           1           0
--- RDMA Resources ---
0: mlx5_0: pd 14 cq 178 qp 23319 cm_id 0 mr 1086 ctx 18 srq 2 
1: mlx5_1: pd 14 cq 178 qp 23411 cm_id 0 mr 1086 ctx 18 srq 2 
2: mlx5_2: pd 11 cq 12 qp 25176 cm_id 0 mr 1080 ctx 17 srq 2 
3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
4: mlx5_4: pd 14 cq 178 qp 23766 cm_id 0 mr 1086 ctx 18 srq 2 
5: mlx5_5: pd 14 cq 178 qp 23367 cm_id 0 mr 1086 ctx 18 srq 2

System crash and cleanup (03:05:52):

=== Thu Jul 31 03:05:52 AM PDT 2025 ===
0: mlx5_0: pd 13 cq 177 qp 8797 cm_id 0 mr 951 ctx 17 srq 2 
1: mlx5_1: pd 10 cq 174 qp 106 cm_id 0 mr 546 ctx 14 srq 2

The script:

#!/bin/bash

# log file with timestamp
LOG_FILE="resource_monitor_$(date +%Y%m%d_%H%M%S).log"

echo "Starting resource monitoring - logging to $LOG_FILE"
echo "Press Ctrl+C to stop"

log_state() {
    echo "=== $(date) ===" >> $LOG_FILE
    echo "--- Memory ---" >> $LOG_FILE
    free -g >> $LOG_FILE
    echo "--- RDMA Resources ---" >> $LOG_FILE
    rdma resource >> $LOG_FILE
    echo "" >> $LOG_FILE
}

# Log initial state
log_state

# monitor every 10 seconds
while true; do
    sleep 10
    log_state
done
<!-- gh-comment-id:3139621631 --> @tugot17 commented on GitHub (Jul 31, 2025): I don't think it is the memory issue, but rather the issue with exploding number of QPs? **Normal baseline (03:01:21):** ``` === Thu Jul 31 03:01:21 AM PDT 2025 === --- Memory --- total used free shared buff/cache available Mem: 1007 19 808 1 179 980 Swap: 1 1 0 --- RDMA Resources --- 0: mlx5_0: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 1: mlx5_1: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 2: mlx5_2: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 4: mlx5_4: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 5: mlx5_5: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 ``` **QP explosion begins (03:02:21):** ``` === Thu Jul 31 03:02:21 AM PDT 2025 === --- Memory --- total used free shared buff/cache available Mem: 1007 61 766 1 179 938 Swap: 1 1 0 --- RDMA Resources --- 0: mlx5_0: pd 14 cq 178 qp 946 cm_id 0 mr 1086 ctx 18 srq 2 1: mlx5_1: pd 14 cq 178 qp 937 cm_id 0 mr 1086 ctx 18 srq 2 2: mlx5_2: pd 11 cq 12 qp 805 cm_id 0 mr 1080 ctx 17 srq 2 3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 4: mlx5_4: pd 14 cq 178 qp 947 cm_id 0 mr 1086 ctx 18 srq 2 5: mlx5_5: pd 14 cq 178 qp 939 cm_id 0 mr 1086 ctx 18 srq 2 ``` **Peak QP usage before crash (03:05:32):** ``` === Thu Jul 31 03:05:32 AM PDT 2025 === --- Memory --- total used free shared buff/cache available Mem: 1007 74 751 1 180 925 Swap: 1 1 0 --- RDMA Resources --- 0: mlx5_0: pd 14 cq 178 qp 23319 cm_id 0 mr 1086 ctx 18 srq 2 1: mlx5_1: pd 14 cq 178 qp 23411 cm_id 0 mr 1086 ctx 18 srq 2 2: mlx5_2: pd 11 cq 12 qp 25176 cm_id 0 mr 1080 ctx 17 srq 2 3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 4: mlx5_4: pd 14 cq 178 qp 23766 cm_id 0 mr 1086 ctx 18 srq 2 5: mlx5_5: pd 14 cq 178 qp 23367 cm_id 0 mr 1086 ctx 18 srq 2 ``` **System crash and cleanup (03:05:52):** ``` === Thu Jul 31 03:05:52 AM PDT 2025 === 0: mlx5_0: pd 13 cq 177 qp 8797 cm_id 0 mr 951 ctx 17 srq 2 1: mlx5_1: pd 10 cq 174 qp 106 cm_id 0 mr 546 ctx 14 srq 2 ``` The script: ```bash #!/bin/bash # log file with timestamp LOG_FILE="resource_monitor_$(date +%Y%m%d_%H%M%S).log" echo "Starting resource monitoring - logging to $LOG_FILE" echo "Press Ctrl+C to stop" log_state() { echo "=== $(date) ===" >> $LOG_FILE echo "--- Memory ---" >> $LOG_FILE free -g >> $LOG_FILE echo "--- RDMA Resources ---" >> $LOG_FILE rdma resource >> $LOG_FILE echo "" >> $LOG_FILE } # Log initial state log_state # monitor every 10 seconds while true; do sleep 10 log_state done ```
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Jul 31, 2025):

I don't think it is the memory issue, but rather the issue with exploding number of QPs?

Normal baseline (03:01:21):

=== Thu Jul 31 03:01:21 AM PDT 2025 ===
--- Memory ---
               total        used        free      shared  buff/cache   available
Mem:            1007          19         808           1         179         980
Swap:              1           1           0
--- RDMA Resources ---
0: mlx5_0: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 
1: mlx5_1: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 
2: mlx5_2: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
4: mlx5_4: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 
5: mlx5_5: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2

QP explosion begins (03:02:21):

=== Thu Jul 31 03:02:21 AM PDT 2025 ===
--- Memory ---
               total        used        free      shared  buff/cache   available
Mem:            1007          61         766           1         179         938
Swap:              1           1           0
--- RDMA Resources ---
0: mlx5_0: pd 14 cq 178 qp 946 cm_id 0 mr 1086 ctx 18 srq 2 
1: mlx5_1: pd 14 cq 178 qp 937 cm_id 0 mr 1086 ctx 18 srq 2 
2: mlx5_2: pd 11 cq 12 qp 805 cm_id 0 mr 1080 ctx 17 srq 2 
3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
4: mlx5_4: pd 14 cq 178 qp 947 cm_id 0 mr 1086 ctx 18 srq 2 
5: mlx5_5: pd 14 cq 178 qp 939 cm_id 0 mr 1086 ctx 18 srq 2

Peak QP usage before crash (03:05:32):

=== Thu Jul 31 03:05:32 AM PDT 2025 ===
--- Memory ---
               total        used        free      shared  buff/cache   available
Mem:            1007          74         751           1         180         925
Swap:              1           1           0
--- RDMA Resources ---
0: mlx5_0: pd 14 cq 178 qp 23319 cm_id 0 mr 1086 ctx 18 srq 2 
1: mlx5_1: pd 14 cq 178 qp 23411 cm_id 0 mr 1086 ctx 18 srq 2 
2: mlx5_2: pd 11 cq 12 qp 25176 cm_id 0 mr 1080 ctx 17 srq 2 
3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 
4: mlx5_4: pd 14 cq 178 qp 23766 cm_id 0 mr 1086 ctx 18 srq 2 
5: mlx5_5: pd 14 cq 178 qp 23367 cm_id 0 mr 1086 ctx 18 srq 2

System crash and cleanup (03:05:52):

=== Thu Jul 31 03:05:52 AM PDT 2025 ===
0: mlx5_0: pd 13 cq 177 qp 8797 cm_id 0 mr 951 ctx 17 srq 2 
1: mlx5_1: pd 10 cq 174 qp 106 cm_id 0 mr 546 ctx 14 srq 2

The script:

#!/bin/bash

log file with timestamp

LOG_FILE="resource_monitor_$(date +%Y%m%d_%H%M%S).log"

echo "Starting resource monitoring - logging to $LOG_FILE"
echo "Press Ctrl+C to stop"

log_state() {
echo "=== $(date) ===" >> $LOG_FILE
echo "--- Memory ---" >> $LOG_FILE
free -g >> $LOG_FILE
echo "--- RDMA Resources ---" >> $LOG_FILE
rdma resource >> $LOG_FILE
echo "" >> $LOG_FILE
}

Log initial state

log_state

monitor every 10 seconds

while true; do
sleep 10
log_state
done

Hi @tugot17
What version of mooncake are you using?
Can you try using version 0.3.5 and setting the environment variable MC_ENABLE_DEST_DEVICE_AFFINITY=1 to see if that solves the problem?

<!-- gh-comment-id:3139678252 --> @staryxchen commented on GitHub (Jul 31, 2025): > I don't think it is the memory issue, but rather the issue with exploding number of QPs? > > **Normal baseline (03:01:21):** > > ``` > === Thu Jul 31 03:01:21 AM PDT 2025 === > --- Memory --- > total used free shared buff/cache available > Mem: 1007 19 808 1 179 980 > Swap: 1 1 0 > --- RDMA Resources --- > 0: mlx5_0: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 > 1: mlx5_1: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 > 2: mlx5_2: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 > 3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 > 4: mlx5_4: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 > 5: mlx5_5: pd 6 cq 170 qp 86 cm_id 0 mr 6 ctx 2 srq 2 > ``` > > **QP explosion begins (03:02:21):** > > ``` > === Thu Jul 31 03:02:21 AM PDT 2025 === > --- Memory --- > total used free shared buff/cache available > Mem: 1007 61 766 1 179 938 > Swap: 1 1 0 > --- RDMA Resources --- > 0: mlx5_0: pd 14 cq 178 qp 946 cm_id 0 mr 1086 ctx 18 srq 2 > 1: mlx5_1: pd 14 cq 178 qp 937 cm_id 0 mr 1086 ctx 18 srq 2 > 2: mlx5_2: pd 11 cq 12 qp 805 cm_id 0 mr 1080 ctx 17 srq 2 > 3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 > 4: mlx5_4: pd 14 cq 178 qp 947 cm_id 0 mr 1086 ctx 18 srq 2 > 5: mlx5_5: pd 14 cq 178 qp 939 cm_id 0 mr 1086 ctx 18 srq 2 > ``` > > **Peak QP usage before crash (03:05:32):** > > ``` > === Thu Jul 31 03:05:32 AM PDT 2025 === > --- Memory --- > total used free shared buff/cache available > Mem: 1007 74 751 1 180 925 > Swap: 1 1 0 > --- RDMA Resources --- > 0: mlx5_0: pd 14 cq 178 qp 23319 cm_id 0 mr 1086 ctx 18 srq 2 > 1: mlx5_1: pd 14 cq 178 qp 23411 cm_id 0 mr 1086 ctx 18 srq 2 > 2: mlx5_2: pd 11 cq 12 qp 25176 cm_id 0 mr 1080 ctx 17 srq 2 > 3: mlx5_3: pd 3 cq 4 qp 1 cm_id 0 mr 0 ctx 1 srq 2 > 4: mlx5_4: pd 14 cq 178 qp 23766 cm_id 0 mr 1086 ctx 18 srq 2 > 5: mlx5_5: pd 14 cq 178 qp 23367 cm_id 0 mr 1086 ctx 18 srq 2 > ``` > > **System crash and cleanup (03:05:52):** > > ``` > === Thu Jul 31 03:05:52 AM PDT 2025 === > 0: mlx5_0: pd 13 cq 177 qp 8797 cm_id 0 mr 951 ctx 17 srq 2 > 1: mlx5_1: pd 10 cq 174 qp 106 cm_id 0 mr 546 ctx 14 srq 2 > ``` > > The script: > > #!/bin/bash > > # log file with timestamp > LOG_FILE="resource_monitor_$(date +%Y%m%d_%H%M%S).log" > > echo "Starting resource monitoring - logging to $LOG_FILE" > echo "Press Ctrl+C to stop" > > log_state() { > echo "=== $(date) ===" >> $LOG_FILE > echo "--- Memory ---" >> $LOG_FILE > free -g >> $LOG_FILE > echo "--- RDMA Resources ---" >> $LOG_FILE > rdma resource >> $LOG_FILE > echo "" >> $LOG_FILE > } > > # Log initial state > log_state > > # monitor every 10 seconds > while true; do > sleep 10 > log_state > done Hi @tugot17 What version of mooncake are you using? Can you try using version 0.3.5 and setting the environment variable `MC_ENABLE_DEST_DEVICE_AFFINITY=1` to see if that solves the problem?
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@tugot17 commented on GitHub (Jul 31, 2025):

mooncake-transfer-engine 0.3.4.post2

I updated the version and used the MC_ENABLE_DEST_DEVICE_AFFINITY=1 and it seems it is now fully functional.

<!-- gh-comment-id:3140000483 --> @tugot17 commented on GitHub (Jul 31, 2025): `mooncake-transfer-engine 0.3.4.post2` I updated the version and used the `MC_ENABLE_DEST_DEVICE_AFFINITY=1` and it seems it is now fully functional.
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@tugot17 commented on GitHub (Jul 31, 2025):

thanks @staryxchen

Image
<!-- gh-comment-id:3140001973 --> @tugot17 commented on GitHub (Jul 31, 2025): thanks @staryxchen <img width="500" height="500" alt="Image" src="https://github.com/user-attachments/assets/1f37b02b-4801-4eb4-a1d0-bdb6f94649b9" />
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Aug 1, 2025):

thanks @staryxchen

Image

Interesting meme 🤣
BTW, you can report the problem and solution to the sglang community to help others with similar problems, if you confirm the problem has been solved.

<!-- gh-comment-id:3141959148 --> @staryxchen commented on GitHub (Aug 1, 2025): > thanks [@staryxchen](https://github.com/staryxchen) > > <img alt="Image" width="500" height="500" src="https://private-user-images.githubusercontent.com/27293258/473014766-1f37b02b-4801-4eb4-a1d0-bdb6f94649b9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTQwMTU3MTksIm5iZiI6MTc1NDAxNTQxOSwicGF0aCI6Ii8yNzI5MzI1OC80NzMwMTQ3NjYtMWYzN2IwMmItNDgwMS00ZWI0LWExZDAtYmRiNmY5NDY0OWI5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA4MDElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwODAxVDAyMzAxOVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJkOTMxOWM2ZDlkMzEyNmM1NzA4OWQ3ODVhYThkMTRiZTI5YzAyZjBiYmQ1YzIwNTZlMDM5YjI2ZWFmZGQ0NzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.XXNbWe6VOCEXsGHrFuoCKLg6kpz9645GJEOgLH60AGI"> Interesting meme 🤣 BTW, you can report the problem and solution to the sglang community to help others with similar problems, if you confirm the problem has been solved.
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@tugot17 commented on GitHub (Aug 1, 2025):

How would I report it? just make a PR to the docs of SGLang, or are you aware of some other process to do so?

<!-- gh-comment-id:3142708040 --> @tugot17 commented on GitHub (Aug 1, 2025): How would I report it? just make a PR to the docs of SGLang, or are you aware of some other process to do so?
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Aug 1, 2025):

How would I report it? just make a PR to the docs of SGLang, or are you aware of some other process to do so?

I'm not familiar with the sglang community. If they have some kind of troubleshooting docs, you can add relevant content. Or you can raise an issue to report this problem and see what the community thinks.

<!-- gh-comment-id:3142804861 --> @staryxchen commented on GitHub (Aug 1, 2025): > How would I report it? just make a PR to the docs of SGLang, or are you aware of some other process to do so? I'm not familiar with the sglang community. If they have some kind of troubleshooting docs, you can add relevant content. Or you can raise an issue to report this problem and see what the community thinks.
bsakaguchi commented 2026-10-29 21:11:07 +00:00
Author
Owner
Copy link

@jimmy-evo commented on GitHub (Aug 4, 2025):

Great issue! I've been looking this for 2 weeks

<!-- gh-comment-id:3150687338 --> @jimmy-evo commented on GitHub (Aug 4, 2025): Great issue! I've been looking this for 2 weeks
bsakaguchi commented 2026-10-29 21:11:07 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Aug 5, 2025):

Great issue! I've been looking this for 2 weeks

Hi @jinmingyi1998 I'm glad to be of help.
I am curious to know how much QP has decreased in your case. If you don't mind, could you share the changes in the QP before and after setting MC_ENABLE_DEST_DEVICE_AFFINITY=1?

<!-- gh-comment-id:3153054983 --> @staryxchen commented on GitHub (Aug 5, 2025): > Great issue! I've been looking this for 2 weeks Hi @jinmingyi1998 I'm glad to be of help. I am curious to know how much QP has decreased in your case. If you don't mind, could you share the changes in the QP before and after setting MC_ENABLE_DEST_DEVICE_AFFINITY=1?
bsakaguchi commented 2026-10-29 21:11:07 +00:00
Author
Owner
Copy link

@jimmy-evo commented on GitHub (Aug 5, 2025):

Great issue! I've been looking this for 2 weeks

Hi @jinmingyi1998 I'm glad to be of help. I am curious to know how much QP has decreased in your case. If you don't mind, could you share the changes in the QP before and after setting MC_ENABLE_DEST_DEVICE_AFFINITY=1?

it does useful!
num_qp < 250 and didn't increase

<!-- gh-comment-id:3155890920 --> @jimmy-evo commented on GitHub (Aug 5, 2025): > > Great issue! I've been looking this for 2 weeks > > Hi [@jinmingyi1998](https://github.com/jinmingyi1998) I'm glad to be of help. I am curious to know how much QP has decreased in your case. If you don't mind, could you share the changes in the QP before and after setting MC_ENABLE_DEST_DEVICE_AFFINITY=1? it does useful! num_qp < 250 and didn't increase
bsakaguchi commented 2026-10-29 21:11:07 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Aug 6, 2025):

Great issue! I've been looking this for 2 weeks

Hi @jinmingyi1998 I'm glad to be of help. I am curious to know how much QP has decreased in your case. If you don't mind, could you share the changes in the QP before and after setting MC_ENABLE_DEST_DEVICE_AFFINITY=1?

it does useful! num_qp < 250 and didn't increase

Got it. Thanks for letting me know. 🙂

<!-- gh-comment-id:3157175848 --> @staryxchen commented on GitHub (Aug 6, 2025): > > > Great issue! I've been looking this for 2 weeks > > > > > > Hi [@jinmingyi1998](https://github.com/jinmingyi1998) I'm glad to be of help. I am curious to know how much QP has decreased in your case. If you don't mind, could you share the changes in the QP before and after setting MC_ENABLE_DEST_DEVICE_AFFINITY=1? > > it does useful! num_qp < 250 and didn't increase Got it. Thanks for letting me know. 🙂
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#222
No description provided.