kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #887] [Bug]: Specified the topology.json, but it doesn't seem to take effect. #277

New issue

Closed

opened 2026-10-29 21:11:54 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:11:54 +00:00

Owner

Originally created by @lirong-lirong on GitHub (Sep 25, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/887

Bug Report

I specified the topology.json, but it doesn't seem to take effect.

I'm using Mooncake Store to transfer tensors between two GPUs, and there appears to be a system issue where NICs can only access GPU memory within the same PIX domain (PIX = Connection traversing at most a single PCIe bridge). For example, mlx5_bond_0 can only access cuda:0, and attempting to access cuda:1's memory results in errors. There are related issues in the community (https://github.com/kvcache-ai/Mooncake/issues/851, https://github.com/kvcache-ai/Mooncake/issues/284).

To work around this problem, I specified the topology.json via environment variables.

Here is my topology.json:

{  
    "cuda:0": [["mlx5_bond_0"], []], 
    "cuda:1": [["mlx5_bond_1"], []],  
    "cuda:2": [["mlx5_bond_2"], []],
    "cuda:3": [["mlx5_bond_3"], []], 
    "cuda:4": [["mlx5_bond_4"], []],  
    "cuda:5": [["mlx5_bond_5"], []],
    "cuda:6": [["mlx5_bond_6"], []], 
    "cuda:7": [["mlx5_bond_7"], []]
}

Here is my code:

import numpy as np  
import torch  
from mooncake.store import MooncakeDistributedStore, ReplicateConfig  
import time  
import os  
os.environ['GLOG_v'] = '3' 
os.environ['MC_TE_METRIC'] = '1'
os.environ['GLOG_logtostderr'] = '1'
# 初始化 Mooncake Store  
store = MooncakeDistributedStore()  
store.setup(  
    "10.32.5.183:12340",                    # 本地节点地址  
    "http://10.32.5.183:9757/metadata",     # 元数据服务器  
    512*1024*1024,                       # Segment 大小  
    128*1024*1024,                       # 本地缓冲区  
    "rdma",                              # 协议 (rdma/tcp)  
    "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2",                                  # RDMA 设备 (自动发现)  
    "10.32.5.183:50051"                    # Master 服务  
)

# === 发送端 (权重服务器) ===  
# 1. 创建或加载模型权重 tensor (在 GPU 上)  
device = torch.device("cuda:1")
weight_tensor = torch.randn(100, 100, dtype=torch.float32, device=device)

# 2. 获取 GPU 内存地址和大小  
weight_ptr = weight_tensor.data_ptr()  
weight_size = weight_tensor.numel() * weight_tensor.element_size()  
  
# 3. 注册 GPU 内存为可访问的缓冲区  
result = store.register_buffer(weight_ptr, weight_size)  
if result != 0:  
    raise RuntimeError(f"Failed to register source buffer: {result}")  

print(f"GPU tensor is valid: {weight_tensor.is_cuda}")  
print(f"GPU memory address: {hex(weight_tensor.data_ptr())}")

print("1")
# 4. 使用 put_from() 直接从 GPU 地址存储数据  
result = store.put_from("model_weight_layer1", weight_ptr, weight_size)  
if result != 0:  
    raise RuntimeError(f"Failed to store tensor: {result}")  
print(f"Successfully stored {weight_size} bytes from GPU address {hex(weight_ptr)}")  

# === 接收端 (客户端节点) ===  
# 1. 预分配目标 GPU 内存  
target_device = torch.device("cuda:2")
target_tensor = torch.empty(100, 100, dtype=torch.float32, device=target_device)  
  
# 2. 获取目标 GPU 地址  
target_ptr = target_tensor.data_ptr()  
target_size = target_tensor.numel() * target_tensor.element_size()  


print(2)
# 3. 注册目标 GPU 内存  
result = store.register_buffer(target_ptr, target_size)  
if result != 0:  
    raise RuntimeError(f"Failed to register target buffer: {result}")  

print(f"GPU tensor is valid: {target_tensor.is_cuda}")  
print(f"GPU memory address: {hex(target_tensor.data_ptr())}")

print(3)
# 4. 使用 get_into() 直接传输到指定 GPU 地址  
bytes_read = store.get_into("model_weight_layer1", target_ptr, target_size)  
if bytes_read <= 0:  
    raise RuntimeError(f"Failed to retrieve tensor: {bytes_read}")  
print(f"Successfully retrieved {bytes_read} bytes to GPU address {hex(target_ptr)}")  
print(4)
# 5. 验证数据传输正确性  
print(f"Data transfer successful: {torch.allclose(weight_tensor.cpu(), target_tensor.cpu())}")  

try: 
    print(f"Data transfer successful: {torch.allclose(weight_tensor.cpu(), target_tensor.cpu())}")
    print("开始清理...")  
    result1 = store.unregister_buffer(weight_ptr)  
    print(f"unregister weight_ptr result: {result1}")  
      
    result2 = store.unregister_buffer(target_ptr)    
    print(f"unregister target_ptr result: {result2}")  
      
    print("开始关闭store...")  
    store.close()  
    print("store关闭成功")  
except Exception as e:  
    print(f"清理过程中出错: {e}")

The execution log shows:

WARNING: Logging before InitGoogleLogging() is written to STDERR
I0925 00:24:39.140286  4418 client_metric.cpp:76] Client metrics enabled (default enabled)
I0925 00:24:39.140830  4485 transfer_engine.cpp:493] Metrics reporting thread started (interval: 5s)
I0925 00:24:39.140914  4418 client.cpp:46] client_id=3622122671379780837-14715045264966180993
I0925 00:24:39.140924  4418 client.cpp:54] Client metrics enabled but reporting disabled (interval=0)
I0925 00:24:39.140945  4418 master_client.cpp:214] Connecting to master at 10.32.5.183:50051 from mooncake::ErrorCode mooncake::MasterClient::Connect(const string&)
E0925 00:24:39.141773  4418 client.cpp:353] Failed to get fsdir from master
I0925 00:24:39.141781  4418 client.cpp:103] auto discovery set by env MC_MS_AUTO_DISC
I0925 00:24:39.141789  4418 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: 10.32.5.183 port: 12340
I0925 00:24:39.144016  4418 transfer_metadata_plugin.cpp:1100] Found active interface eth0 with IP 10.32.5.183
I0925 00:24:39.144028  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond0 with IP 29.194.30.138
I0925 00:24:39.144032  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond1 with IP 29.194.30.178
I0925 00:24:39.144035  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond2 with IP 29.194.30.166
I0925 00:24:39.144038  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond3 with IP 29.194.30.142
I0925 00:24:39.144042  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond4 with IP 29.194.30.218
I0925 00:24:39.144045  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond5 with IP 29.194.30.214
I0925 00:24:39.144050  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond6 with IP 29.194.30.222
I0925 00:24:39.144052  4418 transfer_metadata_plugin.cpp:1100] Found active interface bond7 with IP 29.194.30.186
I0925 00:24:39.144057  4418 transfer_metadata_plugin.cpp:1089] Skipping interface docker0 (not UP or not RUNNING)
I0925 00:24:39.144073  4418 transfer_engine.cpp:146] Transfer Engine RPC using new RPC mapping, listening on 10.32.5.183:15556
I0925 00:24:39.144482  4418 transfer_engine.cpp:185] Auto-discovering topology...
I0925 00:24:39.144490  4418 transfer_engine.cpp:188] Using custom topology from: /data0/pengdrumli/topology.json
I0925 00:24:39.144560  4418 transfer_engine.cpp:200] Topology discovery complete. Found 8 HCAs.
I0925 00:24:39.149134  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_7/
I0925 00:24:39.149914  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:ba
I0925 00:24:39.153369  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_6/
I0925 00:24:39.154105  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:de
I0925 00:24:39.157409  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_4/
I0925 00:24:39.158092  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:da
I0925 00:24:39.161506  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_3/
I0925 00:24:39.162213  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:8e
I0925 00:24:39.165632  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_2/
I0925 00:24:39.166342  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:a6
I0925 00:24:39.169688  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_5/
I0925 00:24:39.170370  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:d6
I0925 00:24:39.173784  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_1/
I0925 00:24:39.174489  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:b2
I0925 00:24:39.177912  4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_0/
I0925 00:24:39.178597  4418 rdma_context.cpp:129] RDMA device: mlx5_bond_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:8a
I0925 00:24:39.179338  4418 client.cpp:250] Transfer engine auto discovery is enabled for protocol: rdma
I0925 00:24:39.226794  4418 pybind_client.cpp:217] Mounting segment: 536870912 bytes, 536870912 of 536870912
GPU tensor is valid: True
GPU memory address: 0x7f2303400000
1
E0925 00:24:43.616580  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_6, dest_rkey: 36685, retry_cnt: 0): local protection error
E0925 00:24:43.624086  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_7, dest_rkey: 38224, retry_cnt: 1): local protection error
E0925 00:24:43.629307  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_6, dest_rkey: 36685, retry_cnt: 2): local protection error
E0925 00:24:43.633308  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_4, dest_rkey: 32829, retry_cnt: 3): local protection error
E0925 00:24:43.640805  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_3, dest_rkey: 37713, retry_cnt: 4): local protection error
E0925 00:24:43.647084  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_2, dest_rkey: 43106, retry_cnt: 5): local protection error
E0925 00:24:43.653498  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_5, dest_rkey: 42083, retry_cnt: 6): local protection error
E0925 00:24:43.659596  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_1, dest_rkey: 60843, retry_cnt: 7): local protection error
E0925 00:24:43.665987  4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_0, dest_rkey: 60332, retry_cnt: 8): local protection error
E0925 00:24:43.666009  4418 transfer_task.cpp:246] Transfer failed for batch 150903152 task 0 with status 6
E0925 00:24:43.666206  4418 pybind_client.cpp:898] Put operation failed with error: TRANSFER_FAIL
Traceback (most recent call last):
  File "/data0/pengdrumli/master-test.py", line 42, in <module>
    raise RuntimeError(f"Failed to store tensor: {result}")  
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Failed to store tensor: -800
I0925 00:24:44.144174  4418 transfer_engine.cpp:535] Waiting for metrics reporting thread to join...
I0925 00:24:45.141355  4485 transfer_engine.cpp:528] Metrics reporting thread stopped
I0925 00:24:45.141541  4418 transfer_engine.cpp:537] Metrics reporting thread joined

According to the logs, the topology.json doesn't seem to be taking effect. Based on the topology.json configuration, it should use mlx_bond_1 to access the memory of cuda:1, but it actually used mlx_bond_4 instead. This behavior persists regardless of whether automatic topology discovery is enabled or disabled. Am I using it incorrectly?

In fact, by disabling automatic topology discovery and forcibly specifying the NIC, the program can execute successfully.

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @lirong-lirong on GitHub (Sep 25, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/887 ### Bug Report I specified the topology.json, but it doesn't seem to take effect. I'm using Mooncake Store to transfer tensors between two GPUs, and there appears to be a system issue where NICs can only access GPU memory within the same PIX domain (PIX = Connection traversing at most a single PCIe bridge). For example, mlx5_bond_0 can only access cuda:0, and attempting to access cuda:1's memory results in errors. There are related issues in the community (https://github.com/kvcache-ai/Mooncake/issues/851, https://github.com/kvcache-ai/Mooncake/issues/284). To work around this problem, I specified the topology.json via environment variables. Here is my topology.json: ```json { "cuda:0": [["mlx5_bond_0"], []], "cuda:1": [["mlx5_bond_1"], []], "cuda:2": [["mlx5_bond_2"], []], "cuda:3": [["mlx5_bond_3"], []], "cuda:4": [["mlx5_bond_4"], []], "cuda:5": [["mlx5_bond_5"], []], "cuda:6": [["mlx5_bond_6"], []], "cuda:7": [["mlx5_bond_7"], []] } ``` Here is my code: ```python import numpy as np import torch from mooncake.store import MooncakeDistributedStore, ReplicateConfig import time import os os.environ['GLOG_v'] = '3' os.environ['MC_TE_METRIC'] = '1' os.environ['GLOG_logtostderr'] = '1' # 初始化 Mooncake Store store = MooncakeDistributedStore() store.setup( "10.32.5.183:12340", # 本地节点地址 "http://10.32.5.183:9757/metadata", # 元数据服务器 512*1024*1024, # Segment 大小 128*1024*1024, # 本地缓冲区 "rdma", # 协议 (rdma/tcp) "mlx5_bond_0,mlx5_bond_1,mlx5_bond_2", # RDMA 设备 (自动发现) "10.32.5.183:50051" # Master 服务 ) # === 发送端 (权重服务器) === # 1. 创建或加载模型权重 tensor (在 GPU 上) device = torch.device("cuda:1") weight_tensor = torch.randn(100, 100, dtype=torch.float32, device=device) # 2. 获取 GPU 内存地址和大小 weight_ptr = weight_tensor.data_ptr() weight_size = weight_tensor.numel() * weight_tensor.element_size() # 3. 注册 GPU 内存为可访问的缓冲区 result = store.register_buffer(weight_ptr, weight_size) if result != 0: raise RuntimeError(f"Failed to register source buffer: {result}") print(f"GPU tensor is valid: {weight_tensor.is_cuda}") print(f"GPU memory address: {hex(weight_tensor.data_ptr())}") print("1") # 4. 使用 put_from() 直接从 GPU 地址存储数据 result = store.put_from("model_weight_layer1", weight_ptr, weight_size) if result != 0: raise RuntimeError(f"Failed to store tensor: {result}") print(f"Successfully stored {weight_size} bytes from GPU address {hex(weight_ptr)}") # === 接收端 (客户端节点) === # 1. 预分配目标 GPU 内存 target_device = torch.device("cuda:2") target_tensor = torch.empty(100, 100, dtype=torch.float32, device=target_device) # 2. 获取目标 GPU 地址 target_ptr = target_tensor.data_ptr() target_size = target_tensor.numel() * target_tensor.element_size() print(2) # 3. 注册目标 GPU 内存 result = store.register_buffer(target_ptr, target_size) if result != 0: raise RuntimeError(f"Failed to register target buffer: {result}") print(f"GPU tensor is valid: {target_tensor.is_cuda}") print(f"GPU memory address: {hex(target_tensor.data_ptr())}") print(3) # 4. 使用 get_into() 直接传输到指定 GPU 地址 bytes_read = store.get_into("model_weight_layer1", target_ptr, target_size) if bytes_read <= 0: raise RuntimeError(f"Failed to retrieve tensor: {bytes_read}") print(f"Successfully retrieved {bytes_read} bytes to GPU address {hex(target_ptr)}") print(4) # 5. 验证数据传输正确性 print(f"Data transfer successful: {torch.allclose(weight_tensor.cpu(), target_tensor.cpu())}") try: print(f"Data transfer successful: {torch.allclose(weight_tensor.cpu(), target_tensor.cpu())}") print("开始清理...") result1 = store.unregister_buffer(weight_ptr) print(f"unregister weight_ptr result: {result1}") result2 = store.unregister_buffer(target_ptr) print(f"unregister target_ptr result: {result2}") print("开始关闭store...") store.close() print("store关闭成功") except Exception as e: print(f"清理过程中出错: {e}") ``` The execution log shows: ```text WARNING: Logging before InitGoogleLogging() is written to STDERR I0925 00:24:39.140286 4418 client_metric.cpp:76] Client metrics enabled (default enabled) I0925 00:24:39.140830 4485 transfer_engine.cpp:493] Metrics reporting thread started (interval: 5s) I0925 00:24:39.140914 4418 client.cpp:46] client_id=3622122671379780837-14715045264966180993 I0925 00:24:39.140924 4418 client.cpp:54] Client metrics enabled but reporting disabled (interval=0) I0925 00:24:39.140945 4418 master_client.cpp:214] Connecting to master at 10.32.5.183:50051 from mooncake::ErrorCode mooncake::MasterClient::Connect(const string&) E0925 00:24:39.141773 4418 client.cpp:353] Failed to get fsdir from master I0925 00:24:39.141781 4418 client.cpp:103] auto discovery set by env MC_MS_AUTO_DISC I0925 00:24:39.141789 4418 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: 10.32.5.183 port: 12340 I0925 00:24:39.144016 4418 transfer_metadata_plugin.cpp:1100] Found active interface eth0 with IP 10.32.5.183 I0925 00:24:39.144028 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond0 with IP 29.194.30.138 I0925 00:24:39.144032 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond1 with IP 29.194.30.178 I0925 00:24:39.144035 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond2 with IP 29.194.30.166 I0925 00:24:39.144038 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond3 with IP 29.194.30.142 I0925 00:24:39.144042 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond4 with IP 29.194.30.218 I0925 00:24:39.144045 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond5 with IP 29.194.30.214 I0925 00:24:39.144050 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond6 with IP 29.194.30.222 I0925 00:24:39.144052 4418 transfer_metadata_plugin.cpp:1100] Found active interface bond7 with IP 29.194.30.186 I0925 00:24:39.144057 4418 transfer_metadata_plugin.cpp:1089] Skipping interface docker0 (not UP or not RUNNING) I0925 00:24:39.144073 4418 transfer_engine.cpp:146] Transfer Engine RPC using new RPC mapping, listening on 10.32.5.183:15556 I0925 00:24:39.144482 4418 transfer_engine.cpp:185] Auto-discovering topology... I0925 00:24:39.144490 4418 transfer_engine.cpp:188] Using custom topology from: /data0/pengdrumli/topology.json I0925 00:24:39.144560 4418 transfer_engine.cpp:200] Topology discovery complete. Found 8 HCAs. I0925 00:24:39.149134 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_7/ I0925 00:24:39.149914 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_7, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:ba I0925 00:24:39.153369 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_6/ I0925 00:24:39.154105 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_6, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:de I0925 00:24:39.157409 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_4/ I0925 00:24:39.158092 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_4, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:da I0925 00:24:39.161506 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_3/ I0925 00:24:39.162213 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_3, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:8e I0925 00:24:39.165632 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_2/ I0925 00:24:39.166342 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_2, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:a6 I0925 00:24:39.169688 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_5/ I0925 00:24:39.170370 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_5, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:d6 I0925 00:24:39.173784 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_1/ I0925 00:24:39.174489 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_1, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:b2 I0925 00:24:39.177912 4418 rdma_context.cpp:491] Find best gid index: 3 on mlx5_bond_0/ I0925 00:24:39.178597 4418 rdma_context.cpp:129] RDMA device: mlx5_bond_0, LID: 0, GID: (GID_Index 3) 00:00:00:00:00:00:00:00:00:00:ff:ff:1d:c2:1e:8a I0925 00:24:39.179338 4418 client.cpp:250] Transfer engine auto discovery is enabled for protocol: rdma I0925 00:24:39.226794 4418 pybind_client.cpp:217] Mounting segment: 536870912 bytes, 536870912 of 536870912 GPU tensor is valid: True GPU memory address: 0x7f2303400000 1 E0925 00:24:43.616580 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_6, dest_rkey: 36685, retry_cnt: 0): local protection error E0925 00:24:43.624086 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_7, dest_rkey: 38224, retry_cnt: 1): local protection error E0925 00:24:43.629307 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_6, dest_rkey: 36685, retry_cnt: 2): local protection error E0925 00:24:43.633308 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_4, dest_rkey: 32829, retry_cnt: 3): local protection error E0925 00:24:43.640805 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_3, dest_rkey: 37713, retry_cnt: 4): local protection error E0925 00:24:43.647084 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_2, dest_rkey: 43106, retry_cnt: 5): local protection error E0925 00:24:43.653498 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_5, dest_rkey: 42083, retry_cnt: 6): local protection error E0925 00:24:43.659596 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_1, dest_rkey: 60843, retry_cnt: 7): local protection error E0925 00:24:43.665987 4700 worker_pool.cpp:294] Worker: Process failed for slice (opcode: 1, source_addr: 0x7f2303400000, length: 40000, dest_addr: 0x7f252c000000, local_nic: mlx5_bond_4, peer_nic: 10.32.5.183:12340@mlx5_bond_0, dest_rkey: 60332, retry_cnt: 8): local protection error E0925 00:24:43.666009 4418 transfer_task.cpp:246] Transfer failed for batch 150903152 task 0 with status 6 E0925 00:24:43.666206 4418 pybind_client.cpp:898] Put operation failed with error: TRANSFER_FAIL Traceback (most recent call last): File "/data0/pengdrumli/master-test.py", line 42, in <module> raise RuntimeError(f"Failed to store tensor: {result}") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Failed to store tensor: -800 I0925 00:24:44.144174 4418 transfer_engine.cpp:535] Waiting for metrics reporting thread to join... I0925 00:24:45.141355 4485 transfer_engine.cpp:528] Metrics reporting thread stopped I0925 00:24:45.141541 4418 transfer_engine.cpp:537] Metrics reporting thread joined ``` According to the logs, the topology.json doesn't seem to be taking effect. Based on the topology.json configuration, it should use mlx_bond_1 to access the memory of cuda:1, but it actually used mlx_bond_4 instead. This behavior persists regardless of whether automatic topology discovery is enabled or disabled. Am I using it incorrectly? In fact, by disabling automatic topology discovery and forcibly specifying the NIC, the program can execute successfully. ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:11:54 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:11:55 +00:00

Author

Owner

@xiaguan commented on GitHub (Sep 25, 2025):

Thanks for reporting the issue!

You can check the topology info actually stored by the Transfer Engine by printing the topology info from the HTTP metadata. We'll improve the logging around topology discovery.

@xiaguan commented on GitHub (Sep 25, 2025): Thanks for reporting the issue! You can check the topology info actually stored by the Transfer Engine by printing the topology info from the HTTP metadata. We'll improve the logging around topology discovery.

bsakaguchi commented

2026-10-29 21:11:56 +00:00

Author

Owner

@lirong-lirong commented on GitHub (Sep 25, 2025):

@xiaguan

Thanks for your response!

I'm not sure how to retrieve the topology information from the HTTP server - when I tried curling the HTTP server, it returned a 404 error.

However, I used the transfer_engine_topology_dumpscript to get the topology discovery results from the Transfer Engine:

root@h20-183:/data0/pengdrumli# transfer_engine_topology_dump
Local topology:  {
        "cpu:0" : 
        [
                [
                        "mlx5_bond_0",
                        "mlx5_bond_1",
                        "mlx5_bond_2",
                        "mlx5_bond_3"
                ],
                [
                        "mlx5_bond_4",
                        "mlx5_bond_5",
                        "mlx5_bond_6",
                        "mlx5_bond_7"
                ]
        ],
        "cpu:1" : 
        [
                [
                        "mlx5_bond_4",
                        "mlx5_bond_5",
                        "mlx5_bond_6",
                        "mlx5_bond_7"
                ],
                [
                        "mlx5_bond_0",
                        "mlx5_bond_1",
                        "mlx5_bond_2",
                        "mlx5_bond_3"
                ]
        ]
}

I checked the script and found that it doesn't consider the MC_CUSTOM_TOPO_JSONenvironment variable, so I wrote a simple test script:

import sys
import os

def main():
    os.environ['MC_LOG_LEVEL'] = 'ERROR'
    # os.environ['MC_CUSTOM_TOPO_JSON'] = ''
    from mooncake.engine import TransferEngine
    engine = TransferEngine()
    print('Local topology: ', engine.get_local_topology())

if __name__ == "__main__":
    sys.exit(main())

The topology result is:

root@h20-183:/data0/pengdrumli# python3 topology.py 
Local topology:  {
        "cuda:0" : 
        [
                [
                        "mlx5_bond_0"
                ],
                []
        ],
        "cuda:1" : 
        [
                [
                        "mlx5_bond_1"
                ],
                []
        ],
        "cuda:2" : 
        [
                [
                        "mlx5_bond_2"
                ],
                []
        ],
        "cuda:3" : 
        [
                [
                        "mlx5_bond_3"
                ],
                []
        ],
        "cuda:4" : 
        [
                [
                        "mlx5_bond_4"
                ],
                []
        ],
        "cuda:5" : 
        [
                [
                        "mlx5_bond_5"
                ],
                []
        ],
        "cuda:6" : 
        [
                [
                        "mlx5_bond_6"
                ],
                []
        ],
        "cuda:7" : 
        [
                [
                        "mlx5_bond_7"
                ],
                []
        ]
}

As you can see, the topology looks correct.

I also noticed some discussions in the issues and recompiled Mooncake with -DUSE_CUDA=ON. After recompilation, the automatic topology discovery in Mooncake Store works normally, and my test code can also run successfully.

After examining the code, I believe the issue might lie in the store.register_buffer interface. When this Python interface calls the underlying C++ RegisterLocalMemory function to register memory, it first uses getMemoryLocationto detect the memory location. If USE_CUDA is not enabled, it skips the CUDA-related detection logic and directly proceeds to the NUMA node detection for CPU memory, incorrectly identifying GPU memory as Host memory.

Thank you very much for your reply!

By the way, I'd like to ask: do the developers have plans to provide CUDA-enabled pip packages? Currently, the precompiled packages available via pip have CUDA disabled by default, which prevents them from correctly identifying GPU memory locations. In some specific environments where NICs can only access GPU memory within the same PIX domain due to hardware or driver limitations, this leads to various local protection errors (#851, #284). Providing CUDA-enabled pip packages might help reduce some of these issues.

@lirong-lirong commented on GitHub (Sep 25, 2025): @xiaguan Thanks for your response! I'm not sure how to retrieve the topology information from the HTTP server - when I tried curling the HTTP server, it returned a 404 error. However, I used the transfer_engine_topology_dumpscript to get the topology discovery results from the Transfer Engine: ```bash root@h20-183:/data0/pengdrumli# transfer_engine_topology_dump Local topology: { "cpu:0" : [ [ "mlx5_bond_0", "mlx5_bond_1", "mlx5_bond_2", "mlx5_bond_3" ], [ "mlx5_bond_4", "mlx5_bond_5", "mlx5_bond_6", "mlx5_bond_7" ] ], "cpu:1" : [ [ "mlx5_bond_4", "mlx5_bond_5", "mlx5_bond_6", "mlx5_bond_7" ], [ "mlx5_bond_0", "mlx5_bond_1", "mlx5_bond_2", "mlx5_bond_3" ] ] } ``` I checked the script and found that it doesn't consider the MC_CUSTOM_TOPO_JSONenvironment variable, so I wrote a simple test script: ```python import sys import os def main(): os.environ['MC_LOG_LEVEL'] = 'ERROR' # os.environ['MC_CUSTOM_TOPO_JSON'] = '' from mooncake.engine import TransferEngine engine = TransferEngine() print('Local topology: ', engine.get_local_topology()) if __name__ == "__main__": sys.exit(main()) ``` The topology result is: ```bash root@h20-183:/data0/pengdrumli# python3 topology.py Local topology: { "cuda:0" : [ [ "mlx5_bond_0" ], [] ], "cuda:1" : [ [ "mlx5_bond_1" ], [] ], "cuda:2" : [ [ "mlx5_bond_2" ], [] ], "cuda:3" : [ [ "mlx5_bond_3" ], [] ], "cuda:4" : [ [ "mlx5_bond_4" ], [] ], "cuda:5" : [ [ "mlx5_bond_5" ], [] ], "cuda:6" : [ [ "mlx5_bond_6" ], [] ], "cuda:7" : [ [ "mlx5_bond_7" ], [] ] } ``` As you can see, the topology looks correct. I also noticed some discussions in the issues and recompiled Mooncake with -DUSE_CUDA=ON. After recompilation, the automatic topology discovery in Mooncake Store works normally, and my test code can also run successfully. After examining the code, I believe the issue might lie in the `store.register_buffer` interface. When this Python interface calls the underlying C++ `RegisterLocalMemory` function to register memory, it first uses `getMemoryLocationto` detect the memory location. If `USE_CUDA` is not enabled, it skips the CUDA-related detection logic and directly proceeds to the NUMA node detection for CPU memory, incorrectly identifying GPU memory as Host memory. Thank you very much for your reply! By the way, I'd like to ask: do the developers have plans to provide CUDA-enabled pip packages? Currently, the precompiled packages available via pip have CUDA disabled by default, which prevents them from correctly identifying GPU memory locations. In some specific environments where NICs can only access GPU memory within the same PIX domain due to hardware or driver limitations, this leads to various `local protection errors` (#851, #284). Providing CUDA-enabled pip packages might help reduce some of these issues.

bsakaguchi commented

2026-10-29 21:11:56 +00:00

Author

Owner

@xiaguan commented on GitHub (Sep 25, 2025):

Great insight—you nailed the reason!

Good suggestion. We could take care of it.

In fact, we already have a CUDA pip package (for aarch64, it's manually built and uploaded). For x86_64 (the most commonly used arch), we need to figure out how to package CUDA—specifically, which version to use and how to do it.

I'll give it a try. Could you tell me your CUDA version?

@xiaguan commented on GitHub (Sep 25, 2025): Great insight—you nailed the reason! Good suggestion. We could take care of it. In fact, we already have a CUDA pip package (for aarch64, it's manually built and uploaded). For x86_64 (the most commonly used arch), we need to figure out how to package CUDA—specifically, which version to use and how to do it. I'll give it a try. Could you tell me your CUDA version?

bsakaguchi commented

2026-10-29 21:11:56 +00:00

Author

Owner

@lirong-lirong commented on GitHub (Sep 25, 2025):

The container image I'm using is based on CUDA 12.8.1. The test was conducted on host machines with different CUDA versions - some running 12.8 and others 12.2.

ARG CUDA_VERSION=12.8.1
ARG PYTHON_VERSION=3.10
ARG BUILD_BASE_IMAGE=nvidia/cuda:${CUDA_VERSION}-devel-ubuntu22.04  

FROM ${BUILD_BASE_IMAGE}
WORKDIR /base-build

# COPY . .

ENV GOROOT=/usr/local/go
ENV PATH=$GOROOT/bin:$PATH

RUN apt update \
     && apt install -y unzip wget cmake git sudo python3 pip \
     && pip install pybind11 \
     && rm -rf /tmp/* /var/tmp/* /var/lib/apt/lists/*

# Execute installation in the container
RUN git clone https://github.com/kvcache-ai/Mooncake.git 

RUN cd Mooncake && bash dependencies.sh \
     && apt autoremove -y \
     && apt clean -y \
     && rm -rf /tmp/* /var/tmp/* \
     && find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete \
     && find /var/cache -type f -delete

RUN cd Mooncake && rm -rf build || true \
     && mkdir build && cd build \
     && cmake .. -DSTORE_USE_ETCD=ON -DUSE_ETCD=ON -DUSE_HTTP=ON -DUSE_CUDA=ON && make -j$(nproc) && make install

RUN export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/base-build/Mooncake/build/mooncake-common/etcd

# OTHERS
# RUN  pip install torch numpy -i https://pypi.tuna.tsinghua.edu.cn/simple 

# Tencent Cloud
# RUN pip config set global.index-url http://mirrors.cloud.tencent.com/pypi/simple \
#      && pip config set global.trusted-host mirrors.cloud.tencent.com \
#      && pip install torch numpy

@lirong-lirong commented on GitHub (Sep 25, 2025): The container image I'm using is based on CUDA 12.8.1. The test was conducted on host machines with different CUDA versions - some running 12.8 and others 12.2. ```dockerfile ARG CUDA_VERSION=12.8.1 ARG PYTHON_VERSION=3.10 ARG BUILD_BASE_IMAGE=nvidia/cuda:${CUDA_VERSION}-devel-ubuntu22.04 FROM ${BUILD_BASE_IMAGE} WORKDIR /base-build # COPY . . ENV GOROOT=/usr/local/go ENV PATH=$GOROOT/bin:$PATH RUN apt update \ && apt install -y unzip wget cmake git sudo python3 pip \ && pip install pybind11 \ && rm -rf /tmp/* /var/tmp/* /var/lib/apt/lists/* # Execute installation in the container RUN git clone https://github.com/kvcache-ai/Mooncake.git RUN cd Mooncake && bash dependencies.sh \ && apt autoremove -y \ && apt clean -y \ && rm -rf /tmp/* /var/tmp/* \ && find /var/cache/apt/archives /var/lib/apt/lists -not -name lock -type f -delete \ && find /var/cache -type f -delete RUN cd Mooncake && rm -rf build || true \ && mkdir build && cd build \ && cmake .. -DSTORE_USE_ETCD=ON -DUSE_ETCD=ON -DUSE_HTTP=ON -DUSE_CUDA=ON && make -j$(nproc) && make install RUN export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/base-build/Mooncake/build/mooncake-common/etcd # OTHERS # RUN pip install torch numpy -i https://pypi.tuna.tsinghua.edu.cn/simple # Tencent Cloud # RUN pip config set global.index-url http://mirrors.cloud.tencent.com/pypi/simple \ # && pip config set global.trusted-host mirrors.cloud.tencent.com \ # && pip install torch numpy ```