[GH-ISSUE #1169] [Usage]: RDMA usage #397

New issue

Closed

opened 2026-10-29 21:13:52 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:13:52 +00:00

Owner

Originally created by @khuonglm on GitHub (Dec 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1169

Describe your usage question

I have some questions about using mooncake store with rdma.

Here is the code I use to run the experiment

import torch
import time
import safetensors
from safetensors.torch import load_file, save_file
from mooncake.store import MooncakeDistributedStore

# Initialize store with RDMA protocol for maximum performance
print("RDMA test")
store_put = MooncakeDistributedStore()
store_put.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "rdma", "mlx5_8,mlx5_9", "localhost:50051")
store_get = MooncakeDistributedStore()
store_get.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "rdma", "mlx5_10,mlx5_11", "localhost:50051")

# Create data to store
original_data = torch.randn((1000, 1000), dtype=torch.float32)
buffer_ptr = original_data.data_ptr()
size = original_data.numel() * original_data.element_size()

# Step 1: Register the buffer
reg_s = time.perf_counter()
result = store_put.register_buffer(buffer_ptr, size)
reg_e = time.perf_counter()
print(f"register buffer elapsed time in s: {reg_e - reg_s}")
if result != 0:
    raise RuntimeError(f"Failed to register buffer: {result}")

# Step 2: Zero-copy store
reg_s = time.perf_counter()
result = store_put.put_from("large_tensor", buffer_ptr, size)
reg_e = time.perf_counter()
print(f"store op elapsed time in s: {reg_e - reg_s}")
if result == 0:
    print(f"Successfully stored {size} bytes with zero-copy")
else:
    raise RuntimeError(f"Store failed with code: {result}")

# Step 3: Pre-allocate buffer for retrieval
retrieved_data = torch.empty((1000, 1000), dtype=torch.float32)
recv_buffer_ptr = retrieved_data.data_ptr()
recv_size = retrieved_data.numel() * retrieved_data.element_size()

# Step 4: Register receive buffer
reg_s = time.perf_counter()
result = store_get.register_buffer(recv_buffer_ptr, recv_size)
reg_e = time.perf_counter()
print(f"register buffer elapsed time in s: {reg_e - reg_s}")
if result != 0:
    raise RuntimeError(f"Failed to register receive buffer: {result}")

# Step 5: Zero-copy retrieval
reg_s = time.perf_counter()
bytes_read = store_get.get_into("large_tensor", recv_buffer_ptr, recv_size)
reg_e = time.perf_counter()
print(f"load ops elapsed time in s: {reg_e - reg_s}")
if bytes_read > 0:
    print(f"Successfully retrieved {bytes_read} bytes with zero-copy")
    # Verify the data
    print(f"Data matches: {torch.equal(original_data, retrieved_data)}")
else:
    raise RuntimeError(f"Retrieval failed with code: {bytes_read}")

# Step 6: Clean up - unregister both buffers
store_put.unregister_buffer(buffer_ptr)
store_get.unregister_buffer(recv_buffer_ptr)
store_put.close()
store_get.close()

print("TCP test")
store_put = MooncakeDistributedStore()
store_put.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "tcp", "", "localhost:50051")
store_get = MooncakeDistributedStore()
store_get.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "tcp", "", "localhost:50051")

# Create data to store
original_data = torch.randn((1000, 1000), dtype=torch.float32)
buffer_ptr = original_data.data_ptr()
size = original_data.numel() * original_data.element_size()

# Step 1: Register the buffer
reg_s = time.perf_counter()
result = store_put.register_buffer(buffer_ptr, size)
reg_e = time.perf_counter()
print(f"register buffer elapsed time in s: {reg_e - reg_s}")
if result != 0:
    raise RuntimeError(f"Failed to register buffer: {result}")

# Step 2: Zero-copy store
reg_s = time.perf_counter()
result = store_put.put_from("large_tensor", buffer_ptr, size)
reg_e = time.perf_counter()
print(f"store op elapsed time in s: {reg_e - reg_s}")
if result == 0:
    print(f"Successfully stored {size} bytes with zero-copy")
else:
    raise RuntimeError(f"Store failed with code: {result}")

# Step 3: Pre-allocate buffer for retrieval
retrieved_data = torch.empty((1000, 1000), dtype=torch.float32)
recv_buffer_ptr = retrieved_data.data_ptr()
recv_size = retrieved_data.numel() * retrieved_data.element_size()

# Step 4: Register receive buffer
reg_s = time.perf_counter()
result = store_get.register_buffer(recv_buffer_ptr, recv_size)
reg_e = time.perf_counter()
print(f"register buffer elapsed time in s: {reg_e - reg_s}")
if result != 0:
    raise RuntimeError(f"Failed to register receive buffer: {result}")

# Step 5: Zero-copy retrieval
reg_s = time.perf_counter()
bytes_read = store_get.get_into("large_tensor", recv_buffer_ptr, recv_size)
reg_e = time.perf_counter()
print(f"load ops elapsed time in s: {reg_e - reg_s}")
if bytes_read > 0:
    print(f"Successfully retrieved {bytes_read} bytes with zero-copy")
    # Verify the data
    print(f"Data matches: {torch.equal(original_data, retrieved_data)}")
else:
    raise RuntimeError(f"Retrieval failed with code: {bytes_read}")

# Step 6: Clean up - unregister both buffers
store_put.unregister_buffer(buffer_ptr)
store_get.unregister_buffer(recv_buffer_ptr)
store_put.close()
store_get.close()

original_data = torch.randn((1000, 1000), dtype=torch.float32)

reg_s = time.perf_counter()
filename ='/workspace/vllm/analysis/encoder_cache.safetensors'
tensors = {"ec_cache": original_data.detach()}
save_file(tensors, filename)
reg_e = time.perf_counter()
print(f"save disk elapsed time in s: {reg_e - reg_s}")

reg_s = time.perf_counter()
retrieved_data = load_file(filename)["ec_cache"]
reg_e = time.perf_counter()
print(f"load disk elapsed time in s: {reg_e - reg_s}")

print(f"Data matches: {torch.equal(original_data, retrieved_data)}")

RDMA test
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251205 03:33:19.084900  2021 client_metric.cpp:80] Client metrics enabled (default enabled)
I20251205 03:33:19.085592  2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I20251205 03:33:19.085604  2021 client_service.cpp:47] client_id=11691934756937292448-5686628881794908341
I20251205 03:33:19.085609  2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0)
I20251205 03:33:19.088817  2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled.
I20251205 03:33:19.089025  2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I20251205 03:33:19.089041  2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 14216
I20251205 03:33:19.089483  2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3
I20251205 03:33:19.089507  2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:16310
I20251205 03:33:19.090808  2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: rdma
I20251205 03:33:19.090865  2021 client_service.cpp:285] Using specified RDMA devices: mlx5_8,mlx5_9
I20251205 03:33:19.097743  2021 client_service.cpp:295] Topology discovery complete with specified devices. Found 2 HCAs
I20251205 03:33:19.097980  2021 rdma_context.cpp:77] Using SIEVE endpoint store
I20251205 03:33:19.123669  2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_8/ (network state: without network device)
I20251205 03:33:19.191383  2021 rdma_context.cpp:140] RDMA device: mlx5_8, LID: 1, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d8:50
I20251205 03:33:19.191447  2021 rdma_context.cpp:77] Using SIEVE endpoint store
I20251205 03:33:19.222651  2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_9/ (network state: without network device)
I20251205 03:33:19.241377  2021 rdma_context.cpp:140] RDMA device: mlx5_9, LID: 16, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d8:51
I20251205 03:33:19.261952  2021 real_client.cpp:204] Registering local memory: 16777216 bytes
I20251205 03:33:19.294965  2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912
I20251205 03:33:19.500952  2021 client_metric.cpp:80] Client metrics enabled (default enabled)
I20251205 03:33:19.501535  2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I20251205 03:33:19.501547  2021 client_service.cpp:47] client_id=13494744637921797233-3815788281693438653
I20251205 03:33:19.501552  2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0)
I20251205 03:33:19.502620  2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled.
I20251205 03:33:19.502727  2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I20251205 03:33:19.502743  2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 13196
I20251205 03:33:19.502988  2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3
I20251205 03:33:19.503003  2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:16557
I20251205 03:33:19.503353  2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: rdma
I20251205 03:33:19.503366  2021 client_service.cpp:285] Using specified RDMA devices: mlx5_10,mlx5_11
I20251205 03:33:19.504050  2021 client_service.cpp:295] Topology discovery complete with specified devices. Found 2 HCAs
I20251205 03:33:19.504067  2021 rdma_context.cpp:77] Using SIEVE endpoint store
I20251205 03:33:19.515302  2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_10/ (network state: without network device)
I20251205 03:33:19.570784  2021 rdma_context.cpp:140] RDMA device: mlx5_10, LID: 25, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d7:d0
I20251205 03:33:19.570848  2021 rdma_context.cpp:77] Using SIEVE endpoint store
I20251205 03:33:19.589025  2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_11/ (network state: without network device)
I20251205 03:33:19.596918  2021 rdma_context.cpp:140] RDMA device: mlx5_11, LID: 26, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d7:d1
I20251205 03:33:19.611696  2021 real_client.cpp:204] Registering local memory: 16777216 bytes
I20251205 03:33:19.627089  2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912
register buffer elapsed time in s: 0.20800824277102947
store op elapsed time in s: 0.4269195403903723
Successfully stored 4000000 bytes with zero-copy
register buffer elapsed time in s: 0.03758341632783413
load ops elapsed time in s: 0.03459075465798378
Successfully retrieved 4000000 bytes with zero-copy
Data matches: True
TCP test
I20251205 03:33:21.823161  2021 client_metric.cpp:80] Client metrics enabled (default enabled)
I20251205 03:33:21.823657  2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I20251205 03:33:21.823668  2021 client_service.cpp:47] client_id=2829310088265572082-16050854898572728492
I20251205 03:33:21.823733  2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0)
I20251205 03:33:21.825008  2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled.
I20251205 03:33:21.825060  2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I20251205 03:33:21.825153  2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 13722
I20251205 03:33:21.825485  2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3
I20251205 03:33:21.825502  2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:16368
I20251205 03:33:21.825819  2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: tcp
I20251205 03:33:21.826097  2021 tcp_transport.cpp:343] TcpTransport: listen on port 15439
I20251205 03:33:21.840912  2021 real_client.cpp:204] Registering local memory: 16777216 bytes
I20251205 03:33:21.841358  2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912
I20251205 03:33:21.870105  2021 client_metric.cpp:80] Client metrics enabled (default enabled)
I20251205 03:33:21.870543  2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view
I20251205 03:33:21.870553  2021 client_service.cpp:47] client_id=8957369257660210342-1153799996855156152
I20251205 03:33:21.870559  2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0)
I20251205 03:33:21.871601  2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled.
I20251205 03:33:21.871649  2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I20251205 03:33:21.871663  2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 13569
I20251205 03:33:21.872006  2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3
I20251205 03:33:21.872028  2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:15722
I20251205 03:33:21.872414  2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: tcp
I20251205 03:33:21.872871  2021 tcp_transport.cpp:343] TcpTransport: listen on port 15710
I20251205 03:33:21.884755  2021 real_client.cpp:204] Registering local memory: 16777216 bytes
I20251205 03:33:21.885214  2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912
register buffer elapsed time in s: 0.0002044104039669037
store op elapsed time in s: 0.0021700309589505196
Successfully stored 4000000 bytes with zero-copy
register buffer elapsed time in s: 0.00018815044313669205
load ops elapsed time in s: 0.005244561471045017
Successfully retrieved 4000000 bytes with zero-copy
Data matches: True
save disk elapsed time in s: 0.02478019054979086
load disk elapsed time in s: 0.0004493109881877899
Data matches: True

Q1. It seems that I can not use device to device in rdma. When I try to add device = 'cuda' when creating tensor, I can not register buffer. In my understanding, I think I will able to move data from tensor on gpu to another on gpu directly. Did I setup correctly? How should I use mooncake store with rdma (I only find tcp examples)?
Q2. How many rdma devices are enough (matches the tcp performance)?
Q3. Is it true that more devices is better?
Q4. Do I need to allocate distinctly rdma devices to different stores if I test on one machine?
Q5. I found that the rdma is worse than tcp. Could you give me the best practice or something?

Thank you.

Before submitting a new issue...

Make sure you already searched for relevant issues and read the documentation

Originally created by @khuonglm on GitHub (Dec 5, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1169 ### Describe your usage question I have some questions about using mooncake store with rdma. Here is the code I use to run the experiment ```python import torch import time import safetensors from safetensors.torch import load_file, save_file from mooncake.store import MooncakeDistributedStore # Initialize store with RDMA protocol for maximum performance print("RDMA test") store_put = MooncakeDistributedStore() store_put.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "rdma", "mlx5_8,mlx5_9", "localhost:50051") store_get = MooncakeDistributedStore() store_get.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "rdma", "mlx5_10,mlx5_11", "localhost:50051") # Create data to store original_data = torch.randn((1000, 1000), dtype=torch.float32) buffer_ptr = original_data.data_ptr() size = original_data.numel() * original_data.element_size() # Step 1: Register the buffer reg_s = time.perf_counter() result = store_put.register_buffer(buffer_ptr, size) reg_e = time.perf_counter() print(f"register buffer elapsed time in s: {reg_e - reg_s}") if result != 0: raise RuntimeError(f"Failed to register buffer: {result}") # Step 2: Zero-copy store reg_s = time.perf_counter() result = store_put.put_from("large_tensor", buffer_ptr, size) reg_e = time.perf_counter() print(f"store op elapsed time in s: {reg_e - reg_s}") if result == 0: print(f"Successfully stored {size} bytes with zero-copy") else: raise RuntimeError(f"Store failed with code: {result}") # Step 3: Pre-allocate buffer for retrieval retrieved_data = torch.empty((1000, 1000), dtype=torch.float32) recv_buffer_ptr = retrieved_data.data_ptr() recv_size = retrieved_data.numel() * retrieved_data.element_size() # Step 4: Register receive buffer reg_s = time.perf_counter() result = store_get.register_buffer(recv_buffer_ptr, recv_size) reg_e = time.perf_counter() print(f"register buffer elapsed time in s: {reg_e - reg_s}") if result != 0: raise RuntimeError(f"Failed to register receive buffer: {result}") # Step 5: Zero-copy retrieval reg_s = time.perf_counter() bytes_read = store_get.get_into("large_tensor", recv_buffer_ptr, recv_size) reg_e = time.perf_counter() print(f"load ops elapsed time in s: {reg_e - reg_s}") if bytes_read > 0: print(f"Successfully retrieved {bytes_read} bytes with zero-copy") # Verify the data print(f"Data matches: {torch.equal(original_data, retrieved_data)}") else: raise RuntimeError(f"Retrieval failed with code: {bytes_read}") # Step 6: Clean up - unregister both buffers store_put.unregister_buffer(buffer_ptr) store_get.unregister_buffer(recv_buffer_ptr) store_put.close() store_get.close() print("TCP test") store_put = MooncakeDistributedStore() store_put.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "tcp", "", "localhost:50051") store_get = MooncakeDistributedStore() store_get.setup("localhost", "http://localhost:8080/metadata", 512*1024*1024, 16*1024*1024, "tcp", "", "localhost:50051") # Create data to store original_data = torch.randn((1000, 1000), dtype=torch.float32) buffer_ptr = original_data.data_ptr() size = original_data.numel() * original_data.element_size() # Step 1: Register the buffer reg_s = time.perf_counter() result = store_put.register_buffer(buffer_ptr, size) reg_e = time.perf_counter() print(f"register buffer elapsed time in s: {reg_e - reg_s}") if result != 0: raise RuntimeError(f"Failed to register buffer: {result}") # Step 2: Zero-copy store reg_s = time.perf_counter() result = store_put.put_from("large_tensor", buffer_ptr, size) reg_e = time.perf_counter() print(f"store op elapsed time in s: {reg_e - reg_s}") if result == 0: print(f"Successfully stored {size} bytes with zero-copy") else: raise RuntimeError(f"Store failed with code: {result}") # Step 3: Pre-allocate buffer for retrieval retrieved_data = torch.empty((1000, 1000), dtype=torch.float32) recv_buffer_ptr = retrieved_data.data_ptr() recv_size = retrieved_data.numel() * retrieved_data.element_size() # Step 4: Register receive buffer reg_s = time.perf_counter() result = store_get.register_buffer(recv_buffer_ptr, recv_size) reg_e = time.perf_counter() print(f"register buffer elapsed time in s: {reg_e - reg_s}") if result != 0: raise RuntimeError(f"Failed to register receive buffer: {result}") # Step 5: Zero-copy retrieval reg_s = time.perf_counter() bytes_read = store_get.get_into("large_tensor", recv_buffer_ptr, recv_size) reg_e = time.perf_counter() print(f"load ops elapsed time in s: {reg_e - reg_s}") if bytes_read > 0: print(f"Successfully retrieved {bytes_read} bytes with zero-copy") # Verify the data print(f"Data matches: {torch.equal(original_data, retrieved_data)}") else: raise RuntimeError(f"Retrieval failed with code: {bytes_read}") # Step 6: Clean up - unregister both buffers store_put.unregister_buffer(buffer_ptr) store_get.unregister_buffer(recv_buffer_ptr) store_put.close() store_get.close() original_data = torch.randn((1000, 1000), dtype=torch.float32) reg_s = time.perf_counter() filename ='/workspace/vllm/analysis/encoder_cache.safetensors' tensors = {"ec_cache": original_data.detach()} save_file(tensors, filename) reg_e = time.perf_counter() print(f"save disk elapsed time in s: {reg_e - reg_s}") reg_s = time.perf_counter() retrieved_data = load_file(filename)["ec_cache"] reg_e = time.perf_counter() print(f"load disk elapsed time in s: {reg_e - reg_s}") print(f"Data matches: {torch.equal(original_data, retrieved_data)}") ``` ```log RDMA test WARNING: Logging before InitGoogleLogging() is written to STDERR I20251205 03:33:19.084900 2021 client_metric.cpp:80] Client metrics enabled (default enabled) I20251205 03:33:19.085592 2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view I20251205 03:33:19.085604 2021 client_service.cpp:47] client_id=11691934756937292448-5686628881794908341 I20251205 03:33:19.085609 2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0) I20251205 03:33:19.088817 2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled. I20251205 03:33:19.089025 2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I20251205 03:33:19.089041 2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 14216 I20251205 03:33:19.089483 2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3 I20251205 03:33:19.089507 2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:16310 I20251205 03:33:19.090808 2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: rdma I20251205 03:33:19.090865 2021 client_service.cpp:285] Using specified RDMA devices: mlx5_8,mlx5_9 I20251205 03:33:19.097743 2021 client_service.cpp:295] Topology discovery complete with specified devices. Found 2 HCAs I20251205 03:33:19.097980 2021 rdma_context.cpp:77] Using SIEVE endpoint store I20251205 03:33:19.123669 2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_8/ (network state: without network device) I20251205 03:33:19.191383 2021 rdma_context.cpp:140] RDMA device: mlx5_8, LID: 1, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d8:50 I20251205 03:33:19.191447 2021 rdma_context.cpp:77] Using SIEVE endpoint store I20251205 03:33:19.222651 2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_9/ (network state: without network device) I20251205 03:33:19.241377 2021 rdma_context.cpp:140] RDMA device: mlx5_9, LID: 16, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d8:51 I20251205 03:33:19.261952 2021 real_client.cpp:204] Registering local memory: 16777216 bytes I20251205 03:33:19.294965 2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912 I20251205 03:33:19.500952 2021 client_metric.cpp:80] Client metrics enabled (default enabled) I20251205 03:33:19.501535 2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view I20251205 03:33:19.501547 2021 client_service.cpp:47] client_id=13494744637921797233-3815788281693438653 I20251205 03:33:19.501552 2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0) I20251205 03:33:19.502620 2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled. I20251205 03:33:19.502727 2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I20251205 03:33:19.502743 2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 13196 I20251205 03:33:19.502988 2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3 I20251205 03:33:19.503003 2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:16557 I20251205 03:33:19.503353 2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: rdma I20251205 03:33:19.503366 2021 client_service.cpp:285] Using specified RDMA devices: mlx5_10,mlx5_11 I20251205 03:33:19.504050 2021 client_service.cpp:295] Topology discovery complete with specified devices. Found 2 HCAs I20251205 03:33:19.504067 2021 rdma_context.cpp:77] Using SIEVE endpoint store I20251205 03:33:19.515302 2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_10/ (network state: without network device) I20251205 03:33:19.570784 2021 rdma_context.cpp:140] RDMA device: mlx5_10, LID: 25, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d7:d0 I20251205 03:33:19.570848 2021 rdma_context.cpp:77] Using SIEVE endpoint store I20251205 03:33:19.589025 2021 rdma_context.cpp:567] Find best gid index: 0 on mlx5_11/ (network state: without network device) I20251205 03:33:19.596918 2021 rdma_context.cpp:140] RDMA device: mlx5_11, LID: 26, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:08:c0:eb:03:00:91:d7:d1 I20251205 03:33:19.611696 2021 real_client.cpp:204] Registering local memory: 16777216 bytes I20251205 03:33:19.627089 2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912 register buffer elapsed time in s: 0.20800824277102947 store op elapsed time in s: 0.4269195403903723 Successfully stored 4000000 bytes with zero-copy register buffer elapsed time in s: 0.03758341632783413 load ops elapsed time in s: 0.03459075465798378 Successfully retrieved 4000000 bytes with zero-copy Data matches: True TCP test I20251205 03:33:21.823161 2021 client_metric.cpp:80] Client metrics enabled (default enabled) I20251205 03:33:21.823657 2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view I20251205 03:33:21.823668 2021 client_service.cpp:47] client_id=2829310088265572082-16050854898572728492 I20251205 03:33:21.823733 2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0) I20251205 03:33:21.825008 2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled. I20251205 03:33:21.825060 2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I20251205 03:33:21.825153 2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 13722 I20251205 03:33:21.825485 2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3 I20251205 03:33:21.825502 2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:16368 I20251205 03:33:21.825819 2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: tcp I20251205 03:33:21.826097 2021 tcp_transport.cpp:343] TcpTransport: listen on port 15439 I20251205 03:33:21.840912 2021 real_client.cpp:204] Registering local memory: 16777216 bytes I20251205 03:33:21.841358 2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912 I20251205 03:33:21.870105 2021 client_metric.cpp:80] Client metrics enabled (default enabled) I20251205 03:33:21.870543 2021 ha_helper.cpp:20] Master view key: mooncake-store/mooncake/master_view I20251205 03:33:21.870553 2021 client_service.cpp:47] client_id=8957369257660210342-1153799996855156152 I20251205 03:33:21.870559 2021 client_service.cpp:55] Client metrics enabled but reporting disabled (interval=0) I20251205 03:33:21.871601 2021 client_service.cpp:404] Storage root directory is not set. persisting data is disabled. I20251205 03:33:21.871649 2021 transfer_engine.cpp:559] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I20251205 03:33:21.871663 2021 transfer_engine.cpp:101] Transfer Engine parseHostNameWithPort. server_name: localhost port: 13569 I20251205 03:33:21.872006 2021 transfer_metadata_plugin.cpp:1127] Found active interface eth0 with IP 172.17.0.3 I20251205 03:33:21.872028 2021 transfer_engine.cpp:168] Transfer Engine RPC using new RPC mapping, listening on 172.17.0.3:15722 I20251205 03:33:21.872414 2021 client_service.cpp:273] Transfer engine auto discovery is disabled for protocol: tcp I20251205 03:33:21.872871 2021 tcp_transport.cpp:343] TcpTransport: listen on port 15710 I20251205 03:33:21.884755 2021 real_client.cpp:204] Registering local memory: 16777216 bytes I20251205 03:33:21.885214 2021 real_client.cpp:228] Mounting segment: 536870912 bytes, 536870912 of 536870912 register buffer elapsed time in s: 0.0002044104039669037 store op elapsed time in s: 0.0021700309589505196 Successfully stored 4000000 bytes with zero-copy register buffer elapsed time in s: 0.00018815044313669205 load ops elapsed time in s: 0.005244561471045017 Successfully retrieved 4000000 bytes with zero-copy Data matches: True save disk elapsed time in s: 0.02478019054979086 load disk elapsed time in s: 0.0004493109881877899 Data matches: True ``` Q1. It seems that I can not use device to device in rdma. When I try to add device = 'cuda' when creating tensor, I can not register buffer. In my understanding, I think I will able to move data from tensor on gpu to another on gpu directly. Did I setup correctly? How should I use mooncake store with rdma (I only find tcp examples)? Q2. How many rdma devices are enough (matches the tcp performance)? Q3. Is it true that more devices is better? Q4. Do I need to allocate distinctly rdma devices to different stores if I test on one machine? Q5. I found that the rdma is worse than tcp. Could you give me the best practice or something? Thank you. ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

bsakaguchi closed this issue

2026-10-29 21:13:53 +00:00

bsakaguchi commented

2026-10-29 21:13:53 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 5, 2025):

A1. In current implementation of Mooncake Store, all registered memory should be in host. However, if you use Mooncake TE directly, it's okay to register device memory.
A2. Typically 1 is enough. You can remain it as empty as it tries to use all devices.
A3. It depends on whether data transfer is bottleneck. More device occasionally costs more resource.
A4. No needed.
A5. You can try to add the block size for each transfer. Also you can start with TE rather than Store to directly measure the transfer performance.

@alogfans commented on GitHub (Dec 5, 2025): A1. In current implementation of Mooncake Store, all registered memory should be in host. However, if you use Mooncake TE directly, it's okay to register device memory. A2. Typically 1 is enough. You can remain it as empty as it tries to use all devices. A3. It depends on whether data transfer is bottleneck. More device occasionally costs more resource. A4. No needed. A5. You can try to add the block size for each transfer. Also you can start with TE rather than Store to directly measure the transfer performance.

bsakaguchi commented

2026-10-29 21:13:53 +00:00

Author

Owner

@khuonglm commented on GitHub (Dec 5, 2025):

Thank you

@khuonglm commented on GitHub (Dec 5, 2025): Thank you