kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #602] [Usage]: How can i use tcp protocol in sglang RTX4090 #201

New issue
Closed
opened 2026-10-29 21:10:48 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-29 21:10:48 +00:00
Owner
Copy link

Originally created by @ZhenshengWu on GitHub (Jul 8, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/602

Describe your usage question

8 * 4090
sglang-0.4.8-post1
mooncake-transfer-engine 0.3.4.post2

prefill

PROTOCOL=tcp CUDA_VISIBLE_DEVICES=0,1 python3 -m sglang.launch_server  --model-path /models/Qwen/Qwen3-30B-A3B-FP8 --tp 2 --disaggregation-mode prefill  --host 0.0.0.0 --port 30000 --dist-init-addr 172.16.16.63:9999 --nnodes 1 --node-rank 0 --mem-fraction-static 0.75 --disable-radix-cache

decode

PROTOCOL=tcp CUDA_VISIBLE_DEVICES=2,3  python3 -m sglang.launch_server  --model-path /models/Qwen/Qwen3-30B-A3B-FP8 --tp 2 --disaggregation-mode decode  --host 0.0.0.0 --port 30001 --dist-init-addr 172.16.16.63:9998 --nnodes 1 --node-rank 0 --mem-fraction-static 0.75 --disable-radix-cache

error_info

[2025-07-08 06:14:34] End of prefill warmup with status 200, resp: [{'text': '%', 'meta_info': {'id': 'd94d6945758240cebb0865da99808830', 'finish_reason': {'type': 'length', 'length': 0}, 'prompt_tokens': 4, 'completion_tokens': 1, 'cached_tokens': 0, 'e2e_latency': 6.842517852783203}}]
[2025-07-08 06:14:34] The server is fired up and ready to roll!
[2025-07-08 06:14:38 TP0] Prefill batch. #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0, input throughput (token/s): 0.37
E0708 06:14:39.147976 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 06:14:39.148039 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 06:14:39.148120 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 06:14:39.148180 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 06:14:39.148198 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
[2025-07-08 06:14:39 TP1] Session 172.16.16.63:16275 failed.
[2025-07-08 06:14:39 TP1] Prefill transfer failed for request rank=1 req.rid='7920d264921a456885604c58f0f8f27b' req.bootstrap_room=3652412862429320087 with exception KVTransferError(bootstrap_room=3652412862429320087): Failed to send kv chunk of 3652412862429320087 to 172.16.16.63:35489
[2025-07-08 06:14:39 TP0] Prefill transfer failed for request rank=0 req.rid='7920d264921a456885604c58f0f8f27b' req.bootstrap_room=3652412862429320087 with exception KVTransferError(bootstrap_room=3652412862429320087): Failed due to an unknown reason from another rank
E0708 06:14:39.159231 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 06:14:39.159271 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 06:14:39.159385 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 06:14:39.159590 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
[2025-07-08 06:14:39 TP0] Session 172.16.16.63:15849 failed.

MoonCake test Case has same error

target:

root@lts-4090:/home/Mooncake_0707/build/mooncake-transfer-engine/example# MC_TE_METRIC=1 ./transfer_engine_bench --metadata_server=http://172.16.16.63:18001/metadata --mode=target --local_server_name=172.16.16.63:12345  --protocol tcp
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0708 09:25:56.158710   366 transfer_engine.cpp:44] Transfer Engine starting. Server: 172.16.16.63:12345, Metadata: http://172.16.16.63:18001/metadata, ip_or_host_name: 172.16.16.63, rpc_port: 12345
I0708 09:25:56.158730   367 transfer_engine.cpp:394] Metrics reporting thread started (interval: 5s)
I0708 09:25:56.159404   366 transfer_metadata_plugin.cpp:1053] Found active interface eth0 with IP 172.16.16.63
I0708 09:25:56.159416   366 transfer_metadata_plugin.cpp:1053] Found active interface docker0 with IP 172.17.0.1
I0708 09:25:56.159439   366 transfer_engine.cpp:112] Transfer Engine RPC using new RPC mapping, listening on 172.16.16.63:15605
I0708 09:25:56.160874   366 tcp_transport.cpp:241] TcpTransport: listen on port 15606
I0708 09:25:56.160989   366 transfer_engine_bench.cpp:405] VRAM is used
I0708 09:25:56.327559   366 transfer_engine_bench.cpp:422] numa node num: 4
E0708 09:26:26.525108   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525174   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525197   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525218   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525238   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525259   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525280   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525300   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525321   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525339   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525360   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525379   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525400   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525419   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525440   368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0

initiator:

root@lts-4090://home/Mooncake_0707/build/mooncake-transfer-engine/example# ./transfer_engine_bench --metadata_server=http://172.16.16.63:18001/metadata --segment_id=172.16.16.63:12345 --local_server_name=172.16.16.63:12346 --protocol tcp --threads 2
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0708 09:26:26.330760   362 transfer_engine.cpp:387] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0708 09:26:26.330791   362 transfer_engine.cpp:44] Transfer Engine starting. Server: 172.16.16.63:12346, Metadata: http://172.16.16.63:18001/metadata, ip_or_host_name: 172.16.16.63, rpc_port: 12346
I0708 09:26:26.331456   362 transfer_metadata_plugin.cpp:1053] Found active interface eth0 with IP 172.16.16.63
I0708 09:26:26.331470   362 transfer_metadata_plugin.cpp:1053] Found active interface docker0 with IP 172.17.0.1
I0708 09:26:26.331507   362 transfer_engine.cpp:112] Transfer Engine RPC using new RPC mapping, listening on 172.16.16.63:16049
I0708 09:26:26.332921   362 tcp_transport.cpp:241] TcpTransport: listen on port 16050
I0708 09:26:26.333029   362 transfer_engine_bench.cpp:312] VRAM is used
E0708 09:26:26.525163   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525243   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525266   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525290   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525319   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525337   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525362   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525389   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525413   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525430   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525450   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525481   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525501   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525518   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0
E0708 09:26:26.525537   363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0

Mooncake test case would be ok if set use_vram to false

Before submitting a new issue...

  • Make sure you already searched for relevant issues and read the documentation
Originally created by @ZhenshengWu on GitHub (Jul 8, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/602 ### Describe your usage question 8 * 4090 sglang-0.4.8-post1 mooncake-transfer-engine 0.3.4.post2 prefill ``` PROTOCOL=tcp CUDA_VISIBLE_DEVICES=0,1 python3 -m sglang.launch_server --model-path /models/Qwen/Qwen3-30B-A3B-FP8 --tp 2 --disaggregation-mode prefill --host 0.0.0.0 --port 30000 --dist-init-addr 172.16.16.63:9999 --nnodes 1 --node-rank 0 --mem-fraction-static 0.75 --disable-radix-cache ``` decode ``` PROTOCOL=tcp CUDA_VISIBLE_DEVICES=2,3 python3 -m sglang.launch_server --model-path /models/Qwen/Qwen3-30B-A3B-FP8 --tp 2 --disaggregation-mode decode --host 0.0.0.0 --port 30001 --dist-init-addr 172.16.16.63:9998 --nnodes 1 --node-rank 0 --mem-fraction-static 0.75 --disable-radix-cache ``` error_info ``` [2025-07-08 06:14:34] End of prefill warmup with status 200, resp: [{'text': '%', 'meta_info': {'id': 'd94d6945758240cebb0865da99808830', 'finish_reason': {'type': 'length', 'length': 0}, 'prompt_tokens': 4, 'completion_tokens': 1, 'cached_tokens': 0, 'e2e_latency': 6.842517852783203}}] [2025-07-08 06:14:34] The server is fired up and ready to roll! [2025-07-08 06:14:38 TP0] Prefill batch. #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #unbootstrapped-req: 0, #queue-req: 0, #transferring-req: 0, input throughput (token/s): 0.37 E0708 06:14:39.147976 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 06:14:39.148039 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 06:14:39.148120 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 06:14:39.148180 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 06:14:39.148198 11748 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 [2025-07-08 06:14:39 TP1] Session 172.16.16.63:16275 failed. [2025-07-08 06:14:39 TP1] Prefill transfer failed for request rank=1 req.rid='7920d264921a456885604c58f0f8f27b' req.bootstrap_room=3652412862429320087 with exception KVTransferError(bootstrap_room=3652412862429320087): Failed to send kv chunk of 3652412862429320087 to 172.16.16.63:35489 [2025-07-08 06:14:39 TP0] Prefill transfer failed for request rank=0 req.rid='7920d264921a456885604c58f0f8f27b' req.bootstrap_room=3652412862429320087 with exception KVTransferError(bootstrap_room=3652412862429320087): Failed due to an unknown reason from another rank E0708 06:14:39.159231 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 06:14:39.159271 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 06:14:39.159385 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 06:14:39.159590 11737 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 [2025-07-08 06:14:39 TP0] Session 172.16.16.63:15849 failed. ``` MoonCake test Case has same error target: ``` root@lts-4090:/home/Mooncake_0707/build/mooncake-transfer-engine/example# MC_TE_METRIC=1 ./transfer_engine_bench --metadata_server=http://172.16.16.63:18001/metadata --mode=target --local_server_name=172.16.16.63:12345 --protocol tcp WARNING: Logging before InitGoogleLogging() is written to STDERR I0708 09:25:56.158710 366 transfer_engine.cpp:44] Transfer Engine starting. Server: 172.16.16.63:12345, Metadata: http://172.16.16.63:18001/metadata, ip_or_host_name: 172.16.16.63, rpc_port: 12345 I0708 09:25:56.158730 367 transfer_engine.cpp:394] Metrics reporting thread started (interval: 5s) I0708 09:25:56.159404 366 transfer_metadata_plugin.cpp:1053] Found active interface eth0 with IP 172.16.16.63 I0708 09:25:56.159416 366 transfer_metadata_plugin.cpp:1053] Found active interface docker0 with IP 172.17.0.1 I0708 09:25:56.159439 366 transfer_engine.cpp:112] Transfer Engine RPC using new RPC mapping, listening on 172.16.16.63:15605 I0708 09:25:56.160874 366 tcp_transport.cpp:241] TcpTransport: listen on port 15606 I0708 09:25:56.160989 366 transfer_engine_bench.cpp:405] VRAM is used I0708 09:25:56.327559 366 transfer_engine_bench.cpp:422] numa node num: 4 E0708 09:26:26.525108 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525174 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525197 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525218 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525238 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525259 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525280 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525300 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525321 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525339 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525360 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525379 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525400 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525419 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525440 368 tcp_transport.cpp:136] Session::writeBody failed. Error: Bad address (value: 14), total_transferred_bytes_: 0, current transferred_bytes: 0 ``` initiator: ``` root@lts-4090://home/Mooncake_0707/build/mooncake-transfer-engine/example# ./transfer_engine_bench --metadata_server=http://172.16.16.63:18001/metadata --segment_id=172.16.16.63:12345 --local_server_name=172.16.16.63:12346 --protocol tcp --threads 2 WARNING: Logging before InitGoogleLogging() is written to STDERR I0708 09:26:26.330760 362 transfer_engine.cpp:387] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0708 09:26:26.330791 362 transfer_engine.cpp:44] Transfer Engine starting. Server: 172.16.16.63:12346, Metadata: http://172.16.16.63:18001/metadata, ip_or_host_name: 172.16.16.63, rpc_port: 12346 I0708 09:26:26.331456 362 transfer_metadata_plugin.cpp:1053] Found active interface eth0 with IP 172.16.16.63 I0708 09:26:26.331470 362 transfer_metadata_plugin.cpp:1053] Found active interface docker0 with IP 172.17.0.1 I0708 09:26:26.331507 362 transfer_engine.cpp:112] Transfer Engine RPC using new RPC mapping, listening on 172.16.16.63:16049 I0708 09:26:26.332921 362 tcp_transport.cpp:241] TcpTransport: listen on port 16050 I0708 09:26:26.333029 362 transfer_engine_bench.cpp:312] VRAM is used E0708 09:26:26.525163 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525243 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525266 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525290 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525319 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525337 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525362 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525389 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525413 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525430 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525450 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525481 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525501 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525518 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 E0708 09:26:26.525537 363 tcp_transport.cpp:170] Session::readBody failed. Error: End of file (value: 2), total_transferred_bytes_: 0, current transferred_bytes: 0 ``` Mooncake test case would be ok if set use_vram to false ### Before submitting a new issue... - [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
bsakaguchi commented 2026-10-29 21:10:49 +00:00
Author
Owner
Copy link

@alogfans commented on GitHub (Jul 10, 2025):

Currently TCP transport requires that the memory region should be DRAM only. We will fix soon to support VRAM buffer by adding extra copies.

Note: We strongly recommend to use PD disaggregation under the RDMA network for best performance.

<!-- gh-comment-id:3054914593 --> @alogfans commented on GitHub (Jul 10, 2025): Currently TCP transport requires that the memory region should be DRAM only. We will fix soon to support VRAM buffer by adding extra copies. Note: We strongly recommend to use PD disaggregation under the RDMA network for best performance.
bsakaguchi commented 2026-10-29 21:10:49 +00:00
Author
Owner
Copy link

@ZhenshengWu commented on GitHub (Jul 10, 2025):

Currently TCP transport requires that the memory region should be DRAM only. We will fix soon to support VRAM buffer by adding extra copies.

Note: We strongly recommend to use PD disaggregation under the RDMA network for best performance.

Thank you very much, and I look forward to your excellent work!

<!-- gh-comment-id:3057155073 --> @ZhenshengWu commented on GitHub (Jul 10, 2025): > Currently TCP transport requires that the memory region should be DRAM only. We will fix soon to support VRAM buffer by adding extra copies. > > Note: We strongly recommend to use PD disaggregation under the RDMA network for best performance. Thank you very much, and I look forward to your excellent work!
bsakaguchi commented 2026-10-29 21:10:49 +00:00
Author
Owner
Copy link

@YangZeyu95 commented on GitHub (Jul 11, 2025):

Hi @alogfans, thx for your work. And I have a question, when I run P/D with sglang in a single node. GPU 0,1,2,3 for prefill and 4,5,6,7 for decode. Why still need a RDMA device to communication because it's on one node.

<!-- gh-comment-id:3060011132 --> @YangZeyu95 commented on GitHub (Jul 11, 2025): Hi @alogfans, thx for your work. And I have a question, when I run P/D with sglang in a single node. GPU 0,1,2,3 for prefill and 4,5,6,7 for decode. Why still need a RDMA device to communication because it's on one node.
bsakaguchi commented 2026-10-29 21:10:49 +00:00
Author
Owner
Copy link

@alogfans commented on GitHub (Jul 14, 2025):

In the current implementation, we require RDMA devices to communicate. However, in the next major revision (expected to be released in Q3), we will allow to use NVLink instead.

<!-- gh-comment-id:3067527597 --> @alogfans commented on GitHub (Jul 14, 2025): In the current implementation, we require RDMA devices to communicate. However, in the next major revision (expected to be released in Q3), we will allow to use NVLink instead.
bsakaguchi commented 2026-10-29 21:10:49 +00:00
Author
Owner
Copy link

@CSEEduanyu commented on GitHub (Jul 22, 2025):

In the current implementation, we require RDMA devices to communicate. However, in the next major revision (expected to be released in Q3), we will allow to use NVLink instead.

Why use RDMA without crossing machines? Isn't NVLink better?

<!-- gh-comment-id:3101203243 --> @CSEEduanyu commented on GitHub (Jul 22, 2025): > In the current implementation, we require RDMA devices to communicate. However, in the next major revision (expected to be released in Q3), we will allow to use NVLink instead. Why use RDMA without crossing machines? Isn't NVLink better?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#201
No description provided.