kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #682] [Bug]: TcpTransport::startTransfer encountered an ASIO exception.Exception: connect: Connection timed out #221

New issue
Closed
opened 2026-10-29 21:11:05 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-29 21:11:05 +00:00
Owner
Copy link

Originally created by @Marovlo on GitHub (Jul 28, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/682

Bug Report

question

I have two server called as 16 (30.221.162.16) and 198 (30.221.161.198) in short for vllm+lmcache+mooncakeStore PD disaggregation. The mooncake_master and mooncake_http_metadata_server are running at 16. During experiment when decoder try to use store.get_buffer() to get kv_cache, it always encounters a TimeoutError, while the store.is_exists() operation returns True successfully before store.get_buffer(), which means that decoder can't establish connection with prefiller using mooncake.
I write a simple example to reproduce the problem as below:

In server 16 (prefiller-test)

mooncake_master -v=1
mooncake_http_metadata_server

from mooncake.store import MooncakeDistributedStore
import time

store = MooncakeDistributedStore()

store.setup(
    "prefiller-test",           # Your node's address
    "http://30.221.162.16:8080/metadata",    # HTTP metadata server
    512*1024*1024,          # 512MB segment size
    128*1024*1024,          # 128MB local buffer
    "tcp",                             # Use TCP (RDMA for high performance)
    "",                            # Empty for TCP, specify device for RDMA
    "30.221.162.16:50051"        # Master service
)

store.put("hello_key", b"Hello, Mooncake Store!")
print(store.is_exist("hello_key"))
while(1):
    time.sleep(1)

store.close()

In server 198 (decoder-test)

from mooncake.store import MooncakeDistributedStore

store = MooncakeDistributedStore()

store.setup(
    "decoder-test",           # Your node's address
    "http://30.221.162.16:8080/metadata",    # HTTP metadata server
    512*1024*1024,          # 512MB segment size
    128*1024*1024,          # 128MB local buffer
    "tcp",                             # Use TCP (RDMA for high performance)
    "",                            # Empty for TCP, specify device for RDMA
    "30.221.162.16:50051"        # Master service
)

print(store.is_exist("hello_key")) # return True successfully
data = store.get_buffer("hello_key") # failed to get or get_buffer, eventually Timeout
print(data)  

store.close()

outputs

master

(venv) root@TENCENT64:/workspace# mooncake_master -v=1
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0728 08:17:06.862074  5658 master.cpp:78] Master service started on port 50051, enable_gc=0, max_threads=4, enable_metric_reporting=1, metrics_port=9003, default_kv_lease_ttl=5000, default_kv_soft_pin_ttl=1800000, allow_evict_soft_pinned_objects=1, eviction_ratio=0.1, eviction_high_watermark_ratio=1, enable_ha=0, etcd_endpoints=, client_ttl=10, rpc_thread_num=0, rpc_port=0, rpc_address=0.0.0.0, rpc_conn_timeout_seconds=0, rpc_enable_tcp_no_delay=1, cluster_id=mooncake_cluster, memory_allocator=offset
I0728 08:17:06.872241  5658 master_service.cpp:49] action=start_gc_thread
I0728 08:17:06.872283  5669 master_service.cpp:619] action=gc_thread_started
I0728 08:17:06.872455  5658 rpc_service.cpp:180] HTTP metrics server started on port 9003
I0728 08:17:06.872704  5673 rpc_service.cpp:48] Master Metrics: Storage: 0.00 B / 0.00 B | Keys: 0 (soft-pinned: 0) | Requests (Success/Total): Put=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0,  | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B
I0728 08:17:14.448499  5678 scoped_vlog_timer.h:46] GetFsdir request: action=get_fsdir
I0728 08:17:14.448534  5678 scoped_vlog_timer.h:90] GetFsdir response: status=success, value="mooncake_cluster", latency=46us
I0728 08:17:14.459617  5678 scoped_vlog_timer.h:46] MountSegment request: base=140408302600192, size=536870912, segment_name=prefiller-test:13665, id=16376810002321721829-5443441755540039838
I0728 08:17:14.459640  5678 allocator.cpp:115] initializing_offset_buffer_allocator segment_name=prefiller-test:13665 base_address=0x7fb35b000000 size=536870912
I0728 08:17:14.460888  5678 allocator.cpp:132] offset_buffer_allocator_initialized segment_name=prefiller-test:13665
I0728 08:17:14.460899  5678 scoped_vlog_timer.h:90] MountSegment response: status=success, value=void, latency=1284us
I0728 08:17:14.461048  5678 scoped_vlog_timer.h:46] PutStart request: key=hello_key, slice_lengths=1
I0728 08:17:14.461053  5678 master_service.cpp:355] key=hello_key, value_length=22, slice_count=1, config=ReplicateConfig: { replica_num: 1, with_soft_pin: 0, preferred_segment:  }, action=put_start_begin
I0728 08:17:14.461062  5678 allocator.cpp:167] allocation_succeeded size=22 segment=prefiller-test:13665 address=0x7fb35b000000
I0728 08:17:14.461066  5678 master_service.cpp:400] key=hello_key, replica_id=0, slice_index=0, handle=AllocatedBuffer: { segment_name: prefiller-test:13665, size: 22, status: INIT, buffer_ptr: 0x7fb35b000000 }, action=slice_allocated
I0728 08:17:14.461081  5678 scoped_vlog_timer.h:90] PutStart response: status=success, value=[{"descriptor_variant":{"buffer_descriptors":[{"segment_name_":"prefiller-test:13665","size_":22,"buffer_address_":140408302600192,"status_":0}]},"status":2}], latency=29us
I0728 08:17:14.461516  5678 scoped_vlog_timer.h:46] PutEnd request: key=hello_key
I0728 08:17:14.461521  5678 scoped_vlog_timer.h:90] PutEnd response: status=success, value=void, latency=4us
I0728 08:17:14.461594  5678 scoped_vlog_timer.h:46] ExistKey request: key=hello_key
I0728 08:17:14.461598  5678 scoped_vlog_timer.h:90] ExistKey response: status=success, value=true, latency=4us
I0728 08:17:16.872822  5673 rpc_service.cpp:48] Master Metrics: Storage: 22.00 B / 512.00 MB (0.0%) | Keys: 1 (soft-pinned: 0) | Requests (Success/Total): Put=2/2, Get=0/0, Exist=1/1, Del=0/0, DelAll=0/0,  | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B
I0728 08:17:18.534184  5679 scoped_vlog_timer.h:46] GetFsdir request: action=get_fsdir
I0728 08:17:18.534211  5679 scoped_vlog_timer.h:90] GetFsdir response: status=success, value="mooncake_cluster", latency=37us
I0728 08:17:18.544878  5679 scoped_vlog_timer.h:46] MountSegment request: base=140299653349376, size=536870912, segment_name=decoder-test:13579, id=8739332299496202095-12528480269511449496
I0728 08:17:18.544893  5679 allocator.cpp:115] initializing_offset_buffer_allocator segment_name=decoder-test:13579 base_address=0x7f9a0f000000 size=536870912
I0728 08:17:18.546083  5679 allocator.cpp:132] offset_buffer_allocator_initialized segment_name=decoder-test:13579
I0728 08:17:18.546096  5679 scoped_vlog_timer.h:90] MountSegment response: status=success, value=void, latency=1223us
I0728 08:17:18.546295  5679 scoped_vlog_timer.h:46] ExistKey request: key=hello_key
I0728 08:17:18.546301  5679 scoped_vlog_timer.h:90] ExistKey response: status=success, value=true, latency=5us
I0728 08:17:18.546415  5679 scoped_vlog_timer.h:46] GetReplicaList request: key=hello_key
I0728 08:17:18.546423  5679 scoped_vlog_timer.h:90] GetReplicaList response: status=success, value=[{"descriptor_variant":{"buffer_descriptors":[{"segment_name_":"prefiller-test:13665","size_":22,"buffer_address_":140408302600192,"status_":1}]},"status":3}], latency=5us
I0728 08:17:26.872953  5673 rpc_service.cpp:48] Master Metrics: Storage: 22.00 B / 1.00 GB (0.0%) | Keys: 1 (soft-pinned: 0) | Requests (Success/Total): Put=2/2, Get=1/1, Exist=2/2, Del=0/0, DelAll=0/0,  | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B

metadata_server

(venv) root@TENCENT64:/workspace# mooncake_http_metadata_server 
2025-07-28 08:17:08,574 - root - INFO - HTTP Metadata Server started on 0.0.0.0:8080
2025-07-28 08:17:14,455 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Frpc_meta%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:14,456 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:14,459 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:14,459 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:14,461 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "GET /metadata?key=mooncake%2Frpc_meta%2Fprefiller-test%3A13665 HTTP/1.1" 200 199 "-" "-"
2025-07-28 08:17:18,542 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Frpc_meta%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:18,542 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Fram%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:18,544 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Fram%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:18,544 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Fram%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-"
2025-07-28 08:17:18,546 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "GET /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 418 "-" "-"
2025-07-28 08:17:18,546 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "GET /metadata?key=mooncake%2Frpc_meta%2Fprefiller-test%3A13665 HTTP/1.1" 200 199 "-" "-"

prefiller-test

(venv) root@TENCENT64:/workspace/vllm-test-16-198/vllm-lmcache-test/mooncake-store# python mooncake-test.py 
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0728 08:17:14.447731  5685 transfer_engine.cpp:422] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0728 08:17:14.447830  5685 client.cpp:43] client_id=10903118352813309828-13144224583129692052
I0728 08:17:14.448625  5685 client.cpp:236] Storage root directory is not set. persisting data is disabled.
I0728 08:17:14.448634  5685 transfer_engine.cpp:44] Transfer Engine starting. Server: prefiller-test:13665, Metadata: http://30.221.162.16:8080/metadata, ip_or_host_name: prefiller-test, rpc_port: 13665
I0728 08:17:14.448640  5685 transfer_engine.cpp:63] Transfer Engine parseHostNameWithPort. server_name: prefiller-test port: 13665
I0728 08:17:14.454468  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond8 with IP 30.221.239.158
I0728 08:17:14.454484  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond2 with IP 30.221.238.29
I0728 08:17:14.454488  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond7 with IP 30.221.239.94
I0728 08:17:14.454493  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond9 with IP 30.221.239.221
I0728 08:17:14.454496  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond3 with IP 30.221.238.93
I0728 08:17:14.454504  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond1 with IP 30.221.162.16
I0728 08:17:14.454509  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond4 with IP 30.221.238.158
I0728 08:17:14.454510  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond5 with IP 30.221.238.221
I0728 08:17:14.454514  5685 transfer_metadata_plugin.cpp:1053] Found active interface bond6 with IP 30.221.239.29
I0728 08:17:14.454515  5685 transfer_metadata_plugin.cpp:1053] Found active interface flannel.1 with IP 11.157.18.48
I0728 08:17:14.454536  5685 transfer_engine.cpp:114] Transfer Engine RPC using new RPC mapping, listening on 30.221.239.158:16137
I0728 08:17:14.455914  5685 client.cpp:192] transport_type=tcp
I0728 08:17:14.456252  5685 tcp_transport.cpp:250] TcpTransport: listen on port 16071
I0728 08:17:14.459372  5685 store_py.cpp:249] Mounting segment: 536870912 bytes, 536870912 of 536870912
1

decoder-test

(venv) root@TENCENT64:/workspace/vllm-test-16-198/vllm-lmcache-test/mooncake-store# python mooncake-test.py 
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0728 08:17:18.532800   320 transfer_engine.cpp:422] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I0728 08:17:18.532917   320 client.cpp:43] client_id=8377625392771374067-6105932400794069431
I0728 08:17:18.534300   320 client.cpp:236] Storage root directory is not set. persisting data is disabled.
I0728 08:17:18.534313   320 transfer_engine.cpp:44] Transfer Engine starting. Server: decoder-test:13579, Metadata: http://30.221.162.16:8080/metadata, ip_or_host_name: decoder-test, rpc_port: 13579
I0728 08:17:18.534318   320 transfer_engine.cpp:63] Transfer Engine parseHostNameWithPort. server_name: decoder-test port: 13579
I0728 08:17:18.541168   320 transfer_metadata_plugin.cpp:1053] Found active interface bond8 with IP 30.221.237.139
I0728 08:17:18.541185   320 transfer_metadata_plugin.cpp:1053] Found active interface bond2 with IP 30.221.236.10
I0728 08:17:18.541191   320 transfer_metadata_plugin.cpp:1053] Found active interface bond7 with IP 30.221.237.72
I0728 08:17:18.541195   320 transfer_metadata_plugin.cpp:1053] Found active interface bond9 with IP 30.221.237.202
I0728 08:17:18.541200   320 transfer_metadata_plugin.cpp:1053] Found active interface bond3 with IP 30.221.236.74
I0728 08:17:18.541203   320 transfer_metadata_plugin.cpp:1053] Found active interface bond1 with IP 30.221.161.198
I0728 08:17:18.541205   320 transfer_metadata_plugin.cpp:1053] Found active interface bond4 with IP 30.221.236.137
I0728 08:17:18.541208   320 transfer_metadata_plugin.cpp:1053] Found active interface bond5 with IP 30.221.236.203
I0728 08:17:18.541210   320 transfer_metadata_plugin.cpp:1053] Found active interface bond6 with IP 30.221.237.13
I0728 08:17:18.541211   320 transfer_metadata_plugin.cpp:1042] Skipping interface br-49c505bbe127 (not UP or not RUNNING)
I0728 08:17:18.541229   320 transfer_engine.cpp:114] Transfer Engine RPC using new RPC mapping, listening on 30.221.237.139:15522
I0728 08:17:18.542601   320 client.cpp:192] transport_type=tcp
I0728 08:17:18.542896   320 tcp_transport.cpp:250] TcpTransport: listen on port 15167
I0728 08:17:18.544556   320 store_py.cpp:249] Mounting segment: 536870912 bytes, 536870912 of 536870912
1
E0728 08:19:28.863303   320 tcp_transport.cpp:430] TcpTransport::startTransfer encountered an ASIO exception. Slice details - source_addr: 0x7f9a2f7fe040, length: 22, opcode: 0, target_id: 1. Exception: connect: Connection timed out
E0728 08:19:28.863345   320 transfer_task.cpp:244] Transfer failed for batch 118913840 task 0 with status 6
E0728 08:19:28.863354   320 client.cpp:378] transfer_read_failed key=hello_key
E0728 08:19:28.863389   320 store_py.cpp:857] Get failed for key: hello_key with error: TRANSFER_FAIL
None

I am wondering whether the multi net interface is the cause. As we can see that in the prefiller-test output, which says the "Transfer Engine RPC using new RPC mapping, listening on 30.221.239.158:16137" that is interface "bond8", but I prefer to use "bond1" for both prefiller(30.221.162.16) and deocder(30.221.161.198).

I don't kown how it works specificlly, could you please help me out? Thanks a lot.

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @Marovlo on GitHub (Jul 28, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/682 ### Bug Report ## question I have two server called as 16 (30.221.162.16) and 198 (30.221.161.198) in short for vllm+lmcache+mooncakeStore PD disaggregation. The mooncake_master and mooncake_http_metadata_server are running at 16. During experiment when decoder try to use store.get_buffer() to get kv_cache, it always encounters a **TimeoutError**, while the store.is_exists() operation returns **True** successfully before store.get_buffer(), which means that decoder can't establish connection with prefiller using mooncake. I write a simple example to reproduce the problem as below: ## In server 16 (prefiller-test) ```shell mooncake_master -v=1 mooncake_http_metadata_server ``` ```python from mooncake.store import MooncakeDistributedStore import time store = MooncakeDistributedStore() store.setup( "prefiller-test", # Your node's address "http://30.221.162.16:8080/metadata", # HTTP metadata server 512*1024*1024, # 512MB segment size 128*1024*1024, # 128MB local buffer "tcp", # Use TCP (RDMA for high performance) "", # Empty for TCP, specify device for RDMA "30.221.162.16:50051" # Master service ) store.put("hello_key", b"Hello, Mooncake Store!") print(store.is_exist("hello_key")) while(1): time.sleep(1) store.close() ``` ## In server 198 (decoder-test) ```python from mooncake.store import MooncakeDistributedStore store = MooncakeDistributedStore() store.setup( "decoder-test", # Your node's address "http://30.221.162.16:8080/metadata", # HTTP metadata server 512*1024*1024, # 512MB segment size 128*1024*1024, # 128MB local buffer "tcp", # Use TCP (RDMA for high performance) "", # Empty for TCP, specify device for RDMA "30.221.162.16:50051" # Master service ) print(store.is_exist("hello_key")) # return True successfully data = store.get_buffer("hello_key") # failed to get or get_buffer, eventually Timeout print(data) store.close() ``` ## outputs ### master ```shell (venv) root@TENCENT64:/workspace# mooncake_master -v=1 WARNING: Logging before InitGoogleLogging() is written to STDERR I0728 08:17:06.862074 5658 master.cpp:78] Master service started on port 50051, enable_gc=0, max_threads=4, enable_metric_reporting=1, metrics_port=9003, default_kv_lease_ttl=5000, default_kv_soft_pin_ttl=1800000, allow_evict_soft_pinned_objects=1, eviction_ratio=0.1, eviction_high_watermark_ratio=1, enable_ha=0, etcd_endpoints=, client_ttl=10, rpc_thread_num=0, rpc_port=0, rpc_address=0.0.0.0, rpc_conn_timeout_seconds=0, rpc_enable_tcp_no_delay=1, cluster_id=mooncake_cluster, memory_allocator=offset I0728 08:17:06.872241 5658 master_service.cpp:49] action=start_gc_thread I0728 08:17:06.872283 5669 master_service.cpp:619] action=gc_thread_started I0728 08:17:06.872455 5658 rpc_service.cpp:180] HTTP metrics server started on port 9003 I0728 08:17:06.872704 5673 rpc_service.cpp:48] Master Metrics: Storage: 0.00 B / 0.00 B | Keys: 0 (soft-pinned: 0) | Requests (Success/Total): Put=0/0, Get=0/0, Exist=0/0, Del=0/0, DelAll=0/0, | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B I0728 08:17:14.448499 5678 scoped_vlog_timer.h:46] GetFsdir request: action=get_fsdir I0728 08:17:14.448534 5678 scoped_vlog_timer.h:90] GetFsdir response: status=success, value="mooncake_cluster", latency=46us I0728 08:17:14.459617 5678 scoped_vlog_timer.h:46] MountSegment request: base=140408302600192, size=536870912, segment_name=prefiller-test:13665, id=16376810002321721829-5443441755540039838 I0728 08:17:14.459640 5678 allocator.cpp:115] initializing_offset_buffer_allocator segment_name=prefiller-test:13665 base_address=0x7fb35b000000 size=536870912 I0728 08:17:14.460888 5678 allocator.cpp:132] offset_buffer_allocator_initialized segment_name=prefiller-test:13665 I0728 08:17:14.460899 5678 scoped_vlog_timer.h:90] MountSegment response: status=success, value=void, latency=1284us I0728 08:17:14.461048 5678 scoped_vlog_timer.h:46] PutStart request: key=hello_key, slice_lengths=1 I0728 08:17:14.461053 5678 master_service.cpp:355] key=hello_key, value_length=22, slice_count=1, config=ReplicateConfig: { replica_num: 1, with_soft_pin: 0, preferred_segment: }, action=put_start_begin I0728 08:17:14.461062 5678 allocator.cpp:167] allocation_succeeded size=22 segment=prefiller-test:13665 address=0x7fb35b000000 I0728 08:17:14.461066 5678 master_service.cpp:400] key=hello_key, replica_id=0, slice_index=0, handle=AllocatedBuffer: { segment_name: prefiller-test:13665, size: 22, status: INIT, buffer_ptr: 0x7fb35b000000 }, action=slice_allocated I0728 08:17:14.461081 5678 scoped_vlog_timer.h:90] PutStart response: status=success, value=[{"descriptor_variant":{"buffer_descriptors":[{"segment_name_":"prefiller-test:13665","size_":22,"buffer_address_":140408302600192,"status_":0}]},"status":2}], latency=29us I0728 08:17:14.461516 5678 scoped_vlog_timer.h:46] PutEnd request: key=hello_key I0728 08:17:14.461521 5678 scoped_vlog_timer.h:90] PutEnd response: status=success, value=void, latency=4us I0728 08:17:14.461594 5678 scoped_vlog_timer.h:46] ExistKey request: key=hello_key I0728 08:17:14.461598 5678 scoped_vlog_timer.h:90] ExistKey response: status=success, value=true, latency=4us I0728 08:17:16.872822 5673 rpc_service.cpp:48] Master Metrics: Storage: 22.00 B / 512.00 MB (0.0%) | Keys: 1 (soft-pinned: 0) | Requests (Success/Total): Put=2/2, Get=0/0, Exist=1/1, Del=0/0, DelAll=0/0, | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B I0728 08:17:18.534184 5679 scoped_vlog_timer.h:46] GetFsdir request: action=get_fsdir I0728 08:17:18.534211 5679 scoped_vlog_timer.h:90] GetFsdir response: status=success, value="mooncake_cluster", latency=37us I0728 08:17:18.544878 5679 scoped_vlog_timer.h:46] MountSegment request: base=140299653349376, size=536870912, segment_name=decoder-test:13579, id=8739332299496202095-12528480269511449496 I0728 08:17:18.544893 5679 allocator.cpp:115] initializing_offset_buffer_allocator segment_name=decoder-test:13579 base_address=0x7f9a0f000000 size=536870912 I0728 08:17:18.546083 5679 allocator.cpp:132] offset_buffer_allocator_initialized segment_name=decoder-test:13579 I0728 08:17:18.546096 5679 scoped_vlog_timer.h:90] MountSegment response: status=success, value=void, latency=1223us I0728 08:17:18.546295 5679 scoped_vlog_timer.h:46] ExistKey request: key=hello_key I0728 08:17:18.546301 5679 scoped_vlog_timer.h:90] ExistKey response: status=success, value=true, latency=5us I0728 08:17:18.546415 5679 scoped_vlog_timer.h:46] GetReplicaList request: key=hello_key I0728 08:17:18.546423 5679 scoped_vlog_timer.h:90] GetReplicaList response: status=success, value=[{"descriptor_variant":{"buffer_descriptors":[{"segment_name_":"prefiller-test:13665","size_":22,"buffer_address_":140408302600192,"status_":1}]},"status":3}], latency=5us I0728 08:17:26.872953 5673 rpc_service.cpp:48] Master Metrics: Storage: 22.00 B / 1.00 GB (0.0%) | Keys: 1 (soft-pinned: 0) | Requests (Success/Total): Put=2/2, Get=1/1, Exist=2/2, Del=0/0, DelAll=0/0, | Eviction: Success/Attempts=0/0, keys=0, size=0.00 B ``` ### metadata_server ```shell (venv) root@TENCENT64:/workspace# mooncake_http_metadata_server 2025-07-28 08:17:08,574 - root - INFO - HTTP Metadata Server started on 0.0.0.0:8080 2025-07-28 08:17:14,455 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Frpc_meta%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:14,456 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:14,459 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:14,459 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "PUT /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:14,461 - aiohttp.access - INFO - 30.221.162.16 [28/Jul/2025:08:17:14 +0000] "GET /metadata?key=mooncake%2Frpc_meta%2Fprefiller-test%3A13665 HTTP/1.1" 200 199 "-" "-" 2025-07-28 08:17:18,542 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Frpc_meta%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:18,542 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Fram%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:18,544 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Fram%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:18,544 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "PUT /metadata?key=mooncake%2Fram%2Fdecoder-test%3A13579 HTTP/1.1" 200 176 "-" "-" 2025-07-28 08:17:18,546 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "GET /metadata?key=mooncake%2Fram%2Fprefiller-test%3A13665 HTTP/1.1" 200 418 "-" "-" 2025-07-28 08:17:18,546 - aiohttp.access - INFO - 30.221.161.198 [28/Jul/2025:08:17:18 +0000] "GET /metadata?key=mooncake%2Frpc_meta%2Fprefiller-test%3A13665 HTTP/1.1" 200 199 "-" "-" ``` ### prefiller-test ```shell (venv) root@TENCENT64:/workspace/vllm-test-16-198/vllm-lmcache-test/mooncake-store# python mooncake-test.py WARNING: Logging before InitGoogleLogging() is written to STDERR I0728 08:17:14.447731 5685 transfer_engine.cpp:422] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0728 08:17:14.447830 5685 client.cpp:43] client_id=10903118352813309828-13144224583129692052 I0728 08:17:14.448625 5685 client.cpp:236] Storage root directory is not set. persisting data is disabled. I0728 08:17:14.448634 5685 transfer_engine.cpp:44] Transfer Engine starting. Server: prefiller-test:13665, Metadata: http://30.221.162.16:8080/metadata, ip_or_host_name: prefiller-test, rpc_port: 13665 I0728 08:17:14.448640 5685 transfer_engine.cpp:63] Transfer Engine parseHostNameWithPort. server_name: prefiller-test port: 13665 I0728 08:17:14.454468 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond8 with IP 30.221.239.158 I0728 08:17:14.454484 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond2 with IP 30.221.238.29 I0728 08:17:14.454488 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond7 with IP 30.221.239.94 I0728 08:17:14.454493 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond9 with IP 30.221.239.221 I0728 08:17:14.454496 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond3 with IP 30.221.238.93 I0728 08:17:14.454504 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond1 with IP 30.221.162.16 I0728 08:17:14.454509 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond4 with IP 30.221.238.158 I0728 08:17:14.454510 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond5 with IP 30.221.238.221 I0728 08:17:14.454514 5685 transfer_metadata_plugin.cpp:1053] Found active interface bond6 with IP 30.221.239.29 I0728 08:17:14.454515 5685 transfer_metadata_plugin.cpp:1053] Found active interface flannel.1 with IP 11.157.18.48 I0728 08:17:14.454536 5685 transfer_engine.cpp:114] Transfer Engine RPC using new RPC mapping, listening on 30.221.239.158:16137 I0728 08:17:14.455914 5685 client.cpp:192] transport_type=tcp I0728 08:17:14.456252 5685 tcp_transport.cpp:250] TcpTransport: listen on port 16071 I0728 08:17:14.459372 5685 store_py.cpp:249] Mounting segment: 536870912 bytes, 536870912 of 536870912 1 ``` ### decoder-test ```shell (venv) root@TENCENT64:/workspace/vllm-test-16-198/vllm-lmcache-test/mooncake-store# python mooncake-test.py WARNING: Logging before InitGoogleLogging() is written to STDERR I0728 08:17:18.532800 320 transfer_engine.cpp:422] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I0728 08:17:18.532917 320 client.cpp:43] client_id=8377625392771374067-6105932400794069431 I0728 08:17:18.534300 320 client.cpp:236] Storage root directory is not set. persisting data is disabled. I0728 08:17:18.534313 320 transfer_engine.cpp:44] Transfer Engine starting. Server: decoder-test:13579, Metadata: http://30.221.162.16:8080/metadata, ip_or_host_name: decoder-test, rpc_port: 13579 I0728 08:17:18.534318 320 transfer_engine.cpp:63] Transfer Engine parseHostNameWithPort. server_name: decoder-test port: 13579 I0728 08:17:18.541168 320 transfer_metadata_plugin.cpp:1053] Found active interface bond8 with IP 30.221.237.139 I0728 08:17:18.541185 320 transfer_metadata_plugin.cpp:1053] Found active interface bond2 with IP 30.221.236.10 I0728 08:17:18.541191 320 transfer_metadata_plugin.cpp:1053] Found active interface bond7 with IP 30.221.237.72 I0728 08:17:18.541195 320 transfer_metadata_plugin.cpp:1053] Found active interface bond9 with IP 30.221.237.202 I0728 08:17:18.541200 320 transfer_metadata_plugin.cpp:1053] Found active interface bond3 with IP 30.221.236.74 I0728 08:17:18.541203 320 transfer_metadata_plugin.cpp:1053] Found active interface bond1 with IP 30.221.161.198 I0728 08:17:18.541205 320 transfer_metadata_plugin.cpp:1053] Found active interface bond4 with IP 30.221.236.137 I0728 08:17:18.541208 320 transfer_metadata_plugin.cpp:1053] Found active interface bond5 with IP 30.221.236.203 I0728 08:17:18.541210 320 transfer_metadata_plugin.cpp:1053] Found active interface bond6 with IP 30.221.237.13 I0728 08:17:18.541211 320 transfer_metadata_plugin.cpp:1042] Skipping interface br-49c505bbe127 (not UP or not RUNNING) I0728 08:17:18.541229 320 transfer_engine.cpp:114] Transfer Engine RPC using new RPC mapping, listening on 30.221.237.139:15522 I0728 08:17:18.542601 320 client.cpp:192] transport_type=tcp I0728 08:17:18.542896 320 tcp_transport.cpp:250] TcpTransport: listen on port 15167 I0728 08:17:18.544556 320 store_py.cpp:249] Mounting segment: 536870912 bytes, 536870912 of 536870912 1 E0728 08:19:28.863303 320 tcp_transport.cpp:430] TcpTransport::startTransfer encountered an ASIO exception. Slice details - source_addr: 0x7f9a2f7fe040, length: 22, opcode: 0, target_id: 1. Exception: connect: Connection timed out E0728 08:19:28.863345 320 transfer_task.cpp:244] Transfer failed for batch 118913840 task 0 with status 6 E0728 08:19:28.863354 320 client.cpp:378] transfer_read_failed key=hello_key E0728 08:19:28.863389 320 store_py.cpp:857] Get failed for key: hello_key with error: TRANSFER_FAIL None ``` I am wondering whether the **multi net interface** is the cause. As we can see that in the prefiller-test output, which says the "Transfer Engine RPC using new RPC mapping, listening on 30.221.239.158:16137" that is interface "bond8", but I prefer to use "bond1" for both prefiller(30.221.162.16) and deocder(30.221.161.198). I don't kown how it works specificlly, could you please help me out? Thanks a lot. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi 2026-10-29 21:11:05 +00:00
  • closed this issue
  • added the
    bug
         label
bsakaguchi commented 2026-10-29 21:11:05 +00:00
Author
Owner
Copy link

@xiaguan commented on GitHub (Jul 28, 2025):

maybe you could try set MC_TCP_BIND_ADDRESS

<!-- gh-comment-id:3126445491 --> @xiaguan commented on GitHub (Jul 28, 2025): maybe you could try set `MC_TCP_BIND_ADDRESS`
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@Marovlo commented on GitHub (Jul 28, 2025):

maybe you could try set MC_TCP_BIND_ADDRESS

thanks! This helps a lot.

<!-- gh-comment-id:3126454047 --> @Marovlo commented on GitHub (Jul 28, 2025): > maybe you could try set `MC_TCP_BIND_ADDRESS` thanks! This helps a lot.
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@xiaguan commented on GitHub (Jul 28, 2025):

By the way, we recommend using RDMA for optimal performance.

<!-- gh-comment-id:3126463303 --> @xiaguan commented on GitHub (Jul 28, 2025): By the way, we recommend using RDMA for optimal performance.
bsakaguchi commented 2026-10-29 21:11:06 +00:00
Author
Owner
Copy link

@Marovlo commented on GitHub (Jul 28, 2025):

Yeah，we should try rdma once the whole process can be completed by using tcp as an example. Thanks again!

<!-- gh-comment-id:3126575424 --> @Marovlo commented on GitHub (Jul 28, 2025): Yeah，we should try rdma once the whole process can be completed by using tcp as an example. Thanks again!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#221
No description provided.