kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #1254] [Usage]: how to register gpu address with transfer engine #421

New issue
Closed
opened 2026-10-29 21:14:18 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-29 21:14:18 +00:00
Owner
Copy link

Originally created by @khuonglm on GitHub (Dec 22, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1254

Describe your usage question

Hello. I read that with transfer engine, we could register cuda memory and transfer zero copy between gpu and gpu. I have tried this, but it does not work.

How to resolve this?

mooncake version: 0.3.7.post2

import torch
from mooncake.engine import TransferEngine

engine = TransferEngine()
ret_value = engine.initialize("localhost", "P2PHANDSHAKE", "rdma", "mlx5_6")

tensor = torch.empty(
    (16, 16), dtype=torch.float32, device="cuda"
)
ret_value = engine.register_memory(
    tensor.data_ptr(), tensor.numel() * tensor.element_size()
)
print(ret_value)

WARNING: Logging before InitGoogleLogging() is written to STDERR
I1222 07:16:50.622046 60597 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I1222 07:16:50.622078 60597 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 12001
I1222 07:16:50.622102 60597 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:16027
I1222 07:16:50.622311 60597 transfer_engine.cpp:185] Auto-discovering topology...
I1222 07:16:53.143682 60597 transfer_engine.cpp:200] Topology discovery complete. Found 1 HCAs.
I1222 07:16:53.155242 60597 rdma_context.cpp:533] Find best gid index: 0 on mlx5_6/
I1222 07:16:53.156589 60597 rdma_context.cpp:126] RDMA device: mlx5_6, LID: 25, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:10:70:fd:03:00:16:32:4c
E1222 07:16:55.993026 60597 rdma_context.cpp:243] Failed to register memory 0x7fe75f200000: Bad address [14]
-202

Before submitting a new issue...

  • Make sure you already searched for relevant issues and read the documentation
Originally created by @khuonglm on GitHub (Dec 22, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1254 ### Describe your usage question Hello. I read that with transfer engine, we could register cuda memory and transfer zero copy between gpu and gpu. I have tried this, but it does not work. How to resolve this? mooncake version: 0.3.7.post2 ```python import torch from mooncake.engine import TransferEngine engine = TransferEngine() ret_value = engine.initialize("localhost", "P2PHANDSHAKE", "rdma", "mlx5_6") tensor = torch.empty( (16, 16), dtype=torch.float32, device="cuda" ) ret_value = engine.register_memory( tensor.data_ptr(), tensor.numel() * tensor.element_size() ) print(ret_value) ``` ```log WARNING: Logging before InitGoogleLogging() is written to STDERR I1222 07:16:50.622046 60597 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I1222 07:16:50.622078 60597 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 12001 I1222 07:16:50.622102 60597 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:16027 I1222 07:16:50.622311 60597 transfer_engine.cpp:185] Auto-discovering topology... I1222 07:16:53.143682 60597 transfer_engine.cpp:200] Topology discovery complete. Found 1 HCAs. I1222 07:16:53.155242 60597 rdma_context.cpp:533] Find best gid index: 0 on mlx5_6/ I1222 07:16:53.156589 60597 rdma_context.cpp:126] RDMA device: mlx5_6, LID: 25, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:10:70:fd:03:00:16:32:4c E1222 07:16:55.993026 60597 rdma_context.cpp:243] Failed to register memory 0x7fe75f200000: Bad address [14] -202 ``` ### Before submitting a new issue... - [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
bsakaguchi commented 2026-10-29 21:14:19 +00:00
Author
Owner
Copy link

@ykwd commented on GitHub (Dec 22, 2025):

It looks like your code is correct. You might want to check whether the issue is caused by other factors. For example, on some systems, accessing RDMA requires root privileges.

<!-- gh-comment-id:3680957741 --> @ykwd commented on GitHub (Dec 22, 2025): It looks like your code is correct. You might want to check whether the issue is caused by other factors. For example, on some systems, accessing RDMA requires root privileges.
bsakaguchi commented 2026-10-29 21:14:19 +00:00
Author
Owner
Copy link

@khuonglm commented on GitHub (Dec 22, 2025):

Thank you for your comment. Could you help me to check the details? I think I have root privileges when accessing rdma.

I run a simple code. It seems everything is in good conditions.

import os
import sys

print("=== RDMA Environment Check ===\n")

# 1. Root privileges
print(f"Running as root: {os.geteuid() == 0}")
print(f"UID: {os.getuid()}, EUID: {os.geteuid()}\n")

# 2. Device access
print("Device Access:")
for device in ['uverbs6', 'rdma_cm']:
    path = f"/dev/infiniband/{device}"
    if os.path.exists(path):
        stat = os.stat(path)
        can_read = os.access(path, os.R_OK)
        can_write = os.access(path, os.W_OK)
        print(f"  {device}: exists=✓, readable={can_read}, writable={can_write}, mode={oct(stat.st_mode)}")
    else:
        print(f"  {device}: NOT FOUND")

# 3. Memory limits
import resource
soft, hard = resource.getrlimit(resource.RLIMIT_MEMLOCK)
print(f"\nMemory Lock Limits:")
print(f"  Soft: {soft} ({'unlimited' if soft == resource.RLIM_INFINITY else 'bytes'})")
print(f"  Hard: {hard} ({'unlimited' if hard == resource.RLIM_INFINITY else 'bytes'})")

# 4. Try to import and check the library
print("\n=== Testing TransferEngine ===")
try:
    from mooncake.engine import TransferEngine
    print("✓ TransferEngine imported successfully")
    
    engine = TransferEngine()
    print("✓ TransferEngine instantiated")
    
    # Try initialization
    print("\nAttempting initialization...")
    ret_value = engine.initialize("localhost", "P2PHANDSHAKE", "rdma", "mlx5_6")
    print(f"Initialize returned: {ret_value}")
    
except ImportError as e:
    print(f"✗ Failed to import TransferEngine: {e}")
except Exception as e:
    print(f"✗ Error during initialization: {type(e).__name__}: {e}")
    import traceback
    traceback.print_exc()

# 5. Check if mlx5_6 directory exists
print("\n=== mlx5_6 Device Info ===")
mlx5_path = "/sys/class/infiniband/mlx5_6"
if os.path.exists(mlx5_path):
    print(f"✓ {mlx5_path} exists")
    
    # Check port state
    for port in [1, 2]:
        state_file = f"{mlx5_path}/ports/{port}/state"
        if os.path.exists(state_file):
            with open(state_file) as f:
                state = f.read().strip()
            print(f"  Port {port} state: {state}")
else:
    print(f"✗ {mlx5_path} NOT FOUND")

=== RDMA Environment Check ===

Running as root: True
UID: 0, EUID: 0

Device Access:
  uverbs6: exists=✓, readable=True, writable=True, mode=0o20666
  rdma_cm: exists=✓, readable=True, writable=True, mode=0o20666

Memory Lock Limits:
  Soft: -1 (unlimited)
  Hard: -1 (unlimited)

=== Testing TransferEngine ===
✓ TransferEngine imported successfully
✓ TransferEngine instantiated

Attempting initialization...
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1222 08:39:30.661171  6101 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I1222 08:39:30.661202  6101 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 12001
I1222 08:39:30.661229  6101 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:16141
I1222 08:39:30.661280  6101 transfer_engine.cpp:185] Auto-discovering topology...
I1222 08:39:30.664053  6101 transfer_engine.cpp:200] Topology discovery complete. Found 1 HCAs.
I1222 08:39:30.701478  6101 rdma_context.cpp:491] Find best gid index: 0 on mlx5_6/
I1222 08:39:30.702906  6101 rdma_context.cpp:129] RDMA device: mlx5_6, LID: 25, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:10:70:fd:03:00:16:32:4c
Initialize returned: 0

=== mlx5_6 Device Info ===
✓ /sys/class/infiniband/mlx5_6 exists
  Port 1 state: 4: ACTIVE
This is the rdma information. 
ls -la /dev/infiniband/

total 0
drwxr-xr-x  2 root root     1140 Dec 18 01:55 .
drwxr-xr-x 16 root root     5600 Dec 18 01:56 ..
crw-------  1 root root 231,  64 Dec 18 01:55 issm0
crw-------  1 root root 231,  65 Dec 18 01:55 issm1
crw-------  1 root root 231,  74 Dec 18 01:55 issm10
crw-------  1 root root 231,  75 Dec 18 01:55 issm11
crw-------  1 root root 231,  76 Dec 18 01:55 issm12
crw-------  1 root root 231,  77 Dec 18 01:55 issm13
crw-------  1 root root 231,  78 Dec 18 01:55 issm14
crw-------  1 root root 231,  79 Dec 18 01:55 issm15
crw-------  1 root root 231,  80 Dec 18 01:55 issm16
crw-------  1 root root 231,  81 Dec 18 01:55 issm17
crw-------  1 root root 231,  66 Dec 18 01:55 issm2
crw-------  1 root root 231,  67 Dec 18 01:55 issm3
crw-------  1 root root 231,  68 Dec 18 01:55 issm4
crw-------  1 root root 231,  69 Dec 18 01:55 issm5
crw-------  1 root root 231,  70 Dec 18 01:55 issm6
crw-------  1 root root 231,  71 Dec 18 01:55 issm7
crw-------  1 root root 231,  72 Dec 18 01:55 issm8
crw-------  1 root root 231,  73 Dec 18 01:55 issm9
crw-rw-rw-  1 root root  10,  56 Dec 18 01:55 rdma_cm
crw-------  1 root root 231,   0 Dec 18 01:55 umad0
crw-------  1 root root 231,   1 Dec 18 01:55 umad1
crw-------  1 root root 231,  10 Dec 18 01:55 umad10
crw-------  1 root root 231,  11 Dec 18 01:55 umad11
crw-------  1 root root 231,  12 Dec 18 01:55 umad12
crw-------  1 root root 231,  13 Dec 18 01:55 umad13
crw-------  1 root root 231,  14 Dec 18 01:55 umad14
crw-------  1 root root 231,  15 Dec 18 01:55 umad15
crw-------  1 root root 231,  16 Dec 18 01:55 umad16
crw-------  1 root root 231,  17 Dec 18 01:55 umad17
crw-------  1 root root 231,   2 Dec 18 01:55 umad2
crw-------  1 root root 231,   3 Dec 18 01:55 umad3
crw-------  1 root root 231,   4 Dec 18 01:55 umad4
crw-------  1 root root 231,   5 Dec 18 01:55 umad5
crw-------  1 root root 231,   6 Dec 18 01:55 umad6
crw-------  1 root root 231,   7 Dec 18 01:55 umad7
crw-------  1 root root 231,   8 Dec 18 01:55 umad8
crw-------  1 root root 231,   9 Dec 18 01:55 umad9
crw-rw-rw-  1 root root 231, 192 Dec 18 01:55 uverbs0
crw-rw-rw-  1 root root 231, 193 Dec 18 01:55 uverbs1
crw-rw-rw-  1 root root 231, 202 Dec 18 01:55 uverbs10
crw-rw-rw-  1 root root 231, 203 Dec 18 01:55 uverbs11
crw-rw-rw-  1 root root 231, 204 Dec 18 01:55 uverbs12
crw-rw-rw-  1 root root 231, 205 Dec 18 01:55 uverbs13
crw-rw-rw-  1 root root 231, 206 Dec 18 01:55 uverbs14
crw-rw-rw-  1 root root 231, 207 Dec 18 01:55 uverbs15
crw-rw-rw-  1 root root 231, 208 Dec 18 01:55 uverbs16
crw-rw-rw-  1 root root 231, 209 Dec 18 01:55 uverbs17
crw-rw-rw-  1 root root 231, 194 Dec 18 01:55 uverbs2
crw-rw-rw-  1 root root 231, 195 Dec 18 01:55 uverbs3
crw-rw-rw-  1 root root 231, 196 Dec 18 01:55 uverbs4
crw-rw-rw-  1 root root 231, 197 Dec 18 01:55 uverbs5
crw-rw-rw-  1 root root 231, 198 Dec 18 01:55 uverbs6
crw-rw-rw-  1 root root 231, 199 Dec 18 01:55 uverbs7
crw-rw-rw-  1 root root 231, 200 Dec 18 01:55 uverbs8
crw-rw-rw-  1 root root 231, 201 Dec 18 01:55 uverbs9
Capabilities: Current: =ep
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read
Ambient set =
Current IAB: 
Securebits: 00/0x0/1'b0 (no-new-privs=0)
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
 secure-no-ambient-raise: no (unlocked)
uid=0(root) euid=0(root)
gid=0(root)
groups=0(root)
Guessed mode: HYBRID (4)

ls -l /sys/class/infiniband/*/device/infiniband_verbs/uverbs*

/sys/class/infiniband/mlx5_0/device/infiniband_verbs/uverbs0:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:54:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_1/device/infiniband_verbs/uverbs1:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:54:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_10/device/infiniband_verbs/uverbs10:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:d1:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_11/device/infiniband_verbs/uverbs11:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:d1:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_12/device/infiniband_verbs/uverbs12:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:8e:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_13/device/infiniband_verbs/uverbs13:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:8e:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_14/device/infiniband_verbs/uverbs14:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:91:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_15/device/infiniband_verbs/uverbs15:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:91:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_16/device/infiniband_verbs/uverbs16:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:b1:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_17/device/infiniband_verbs/uverbs17:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:b1:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_2/device/infiniband_verbs/uverbs2:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:55:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_3/device/infiniband_verbs/uverbs3:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:55:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_4/device/infiniband_verbs/uverbs4:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:0e:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_5/device/infiniband_verbs/uverbs5:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:0e:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_6/device/infiniband_verbs/uverbs6:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:11:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_7/device/infiniband_verbs/uverbs7:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:11:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_8/device/infiniband_verbs/uverbs8:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:ce:00.0
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent

/sys/class/infiniband/mlx5_9/device/infiniband_verbs/uverbs9:
total 0
-r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version
-r--r--r-- 1 root root 4096 Dec 22 08:21 dev
lrwxrwxrwx 1 root root    0 Dec 22 08:21 device -> ../../../0000:ce:00.1
-r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev
drwxr-xr-x 2 root root    0 Dec 22 08:21 power
lrwxrwxrwx 1 root root    0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs
-rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent
<!-- gh-comment-id:3681018430 --> @khuonglm commented on GitHub (Dec 22, 2025): Thank you for your comment. Could you help me to check the details? I think I have root privileges when accessing rdma. I run a simple code. It seems everything is in good conditions. ```python import os import sys print("=== RDMA Environment Check ===\n") # 1. Root privileges print(f"Running as root: {os.geteuid() == 0}") print(f"UID: {os.getuid()}, EUID: {os.geteuid()}\n") # 2. Device access print("Device Access:") for device in ['uverbs6', 'rdma_cm']: path = f"/dev/infiniband/{device}" if os.path.exists(path): stat = os.stat(path) can_read = os.access(path, os.R_OK) can_write = os.access(path, os.W_OK) print(f" {device}: exists=✓, readable={can_read}, writable={can_write}, mode={oct(stat.st_mode)}") else: print(f" {device}: NOT FOUND") # 3. Memory limits import resource soft, hard = resource.getrlimit(resource.RLIMIT_MEMLOCK) print(f"\nMemory Lock Limits:") print(f" Soft: {soft} ({'unlimited' if soft == resource.RLIM_INFINITY else 'bytes'})") print(f" Hard: {hard} ({'unlimited' if hard == resource.RLIM_INFINITY else 'bytes'})") # 4. Try to import and check the library print("\n=== Testing TransferEngine ===") try: from mooncake.engine import TransferEngine print("✓ TransferEngine imported successfully") engine = TransferEngine() print("✓ TransferEngine instantiated") # Try initialization print("\nAttempting initialization...") ret_value = engine.initialize("localhost", "P2PHANDSHAKE", "rdma", "mlx5_6") print(f"Initialize returned: {ret_value}") except ImportError as e: print(f"✗ Failed to import TransferEngine: {e}") except Exception as e: print(f"✗ Error during initialization: {type(e).__name__}: {e}") import traceback traceback.print_exc() # 5. Check if mlx5_6 directory exists print("\n=== mlx5_6 Device Info ===") mlx5_path = "/sys/class/infiniband/mlx5_6" if os.path.exists(mlx5_path): print(f"✓ {mlx5_path} exists") # Check port state for port in [1, 2]: state_file = f"{mlx5_path}/ports/{port}/state" if os.path.exists(state_file): with open(state_file) as f: state = f.read().strip() print(f" Port {port} state: {state}") else: print(f"✗ {mlx5_path} NOT FOUND") ``` ```log === RDMA Environment Check === Running as root: True UID: 0, EUID: 0 Device Access: uverbs6: exists=✓, readable=True, writable=True, mode=0o20666 rdma_cm: exists=✓, readable=True, writable=True, mode=0o20666 Memory Lock Limits: Soft: -1 (unlimited) Hard: -1 (unlimited) === Testing TransferEngine === ✓ TransferEngine imported successfully ✓ TransferEngine instantiated Attempting initialization... WARNING: Logging before InitGoogleLogging() is written to STDERR I1222 08:39:30.661171 6101 transfer_engine.cpp:486] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I1222 08:39:30.661202 6101 transfer_engine.cpp:91] Transfer Engine parseHostNameWithPort. server_name: localhost port: 12001 I1222 08:39:30.661229 6101 transfer_engine.cpp:146] Transfer Engine RPC using P2P handshake, listening on localhost:16141 I1222 08:39:30.661280 6101 transfer_engine.cpp:185] Auto-discovering topology... I1222 08:39:30.664053 6101 transfer_engine.cpp:200] Topology discovery complete. Found 1 HCAs. I1222 08:39:30.701478 6101 rdma_context.cpp:491] Find best gid index: 0 on mlx5_6/ I1222 08:39:30.702906 6101 rdma_context.cpp:129] RDMA device: mlx5_6, LID: 25, GID: (GID_Index 0) fe:80:00:00:00:00:00:00:10:70:fd:03:00:16:32:4c Initialize returned: 0 === mlx5_6 Device Info === ✓ /sys/class/infiniband/mlx5_6 exists Port 1 state: 4: ACTIVE ``` <details open> <summary>This is the rdma information.</summary> ```bash ls -la /dev/infiniband/ ``` ```log total 0 drwxr-xr-x 2 root root 1140 Dec 18 01:55 . drwxr-xr-x 16 root root 5600 Dec 18 01:56 .. crw------- 1 root root 231, 64 Dec 18 01:55 issm0 crw------- 1 root root 231, 65 Dec 18 01:55 issm1 crw------- 1 root root 231, 74 Dec 18 01:55 issm10 crw------- 1 root root 231, 75 Dec 18 01:55 issm11 crw------- 1 root root 231, 76 Dec 18 01:55 issm12 crw------- 1 root root 231, 77 Dec 18 01:55 issm13 crw------- 1 root root 231, 78 Dec 18 01:55 issm14 crw------- 1 root root 231, 79 Dec 18 01:55 issm15 crw------- 1 root root 231, 80 Dec 18 01:55 issm16 crw------- 1 root root 231, 81 Dec 18 01:55 issm17 crw------- 1 root root 231, 66 Dec 18 01:55 issm2 crw------- 1 root root 231, 67 Dec 18 01:55 issm3 crw------- 1 root root 231, 68 Dec 18 01:55 issm4 crw------- 1 root root 231, 69 Dec 18 01:55 issm5 crw------- 1 root root 231, 70 Dec 18 01:55 issm6 crw------- 1 root root 231, 71 Dec 18 01:55 issm7 crw------- 1 root root 231, 72 Dec 18 01:55 issm8 crw------- 1 root root 231, 73 Dec 18 01:55 issm9 crw-rw-rw- 1 root root 10, 56 Dec 18 01:55 rdma_cm crw------- 1 root root 231, 0 Dec 18 01:55 umad0 crw------- 1 root root 231, 1 Dec 18 01:55 umad1 crw------- 1 root root 231, 10 Dec 18 01:55 umad10 crw------- 1 root root 231, 11 Dec 18 01:55 umad11 crw------- 1 root root 231, 12 Dec 18 01:55 umad12 crw------- 1 root root 231, 13 Dec 18 01:55 umad13 crw------- 1 root root 231, 14 Dec 18 01:55 umad14 crw------- 1 root root 231, 15 Dec 18 01:55 umad15 crw------- 1 root root 231, 16 Dec 18 01:55 umad16 crw------- 1 root root 231, 17 Dec 18 01:55 umad17 crw------- 1 root root 231, 2 Dec 18 01:55 umad2 crw------- 1 root root 231, 3 Dec 18 01:55 umad3 crw------- 1 root root 231, 4 Dec 18 01:55 umad4 crw------- 1 root root 231, 5 Dec 18 01:55 umad5 crw------- 1 root root 231, 6 Dec 18 01:55 umad6 crw------- 1 root root 231, 7 Dec 18 01:55 umad7 crw------- 1 root root 231, 8 Dec 18 01:55 umad8 crw------- 1 root root 231, 9 Dec 18 01:55 umad9 crw-rw-rw- 1 root root 231, 192 Dec 18 01:55 uverbs0 crw-rw-rw- 1 root root 231, 193 Dec 18 01:55 uverbs1 crw-rw-rw- 1 root root 231, 202 Dec 18 01:55 uverbs10 crw-rw-rw- 1 root root 231, 203 Dec 18 01:55 uverbs11 crw-rw-rw- 1 root root 231, 204 Dec 18 01:55 uverbs12 crw-rw-rw- 1 root root 231, 205 Dec 18 01:55 uverbs13 crw-rw-rw- 1 root root 231, 206 Dec 18 01:55 uverbs14 crw-rw-rw- 1 root root 231, 207 Dec 18 01:55 uverbs15 crw-rw-rw- 1 root root 231, 208 Dec 18 01:55 uverbs16 crw-rw-rw- 1 root root 231, 209 Dec 18 01:55 uverbs17 crw-rw-rw- 1 root root 231, 194 Dec 18 01:55 uverbs2 crw-rw-rw- 1 root root 231, 195 Dec 18 01:55 uverbs3 crw-rw-rw- 1 root root 231, 196 Dec 18 01:55 uverbs4 crw-rw-rw- 1 root root 231, 197 Dec 18 01:55 uverbs5 crw-rw-rw- 1 root root 231, 198 Dec 18 01:55 uverbs6 crw-rw-rw- 1 root root 231, 199 Dec 18 01:55 uverbs7 crw-rw-rw- 1 root root 231, 200 Dec 18 01:55 uverbs8 crw-rw-rw- 1 root root 231, 201 Dec 18 01:55 uverbs9 Capabilities: Current: =ep Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read Ambient set = Current IAB: Securebits: 00/0x0/1'b0 (no-new-privs=0) secure-noroot: no (unlocked) secure-no-suid-fixup: no (unlocked) secure-keep-caps: no (unlocked) secure-no-ambient-raise: no (unlocked) uid=0(root) euid=0(root) gid=0(root) groups=0(root) Guessed mode: HYBRID (4) ``` ```bash ls -l /sys/class/infiniband/*/device/infiniband_verbs/uverbs* ``` ```log /sys/class/infiniband/mlx5_0/device/infiniband_verbs/uverbs0: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:54:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_1/device/infiniband_verbs/uverbs1: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:54:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_10/device/infiniband_verbs/uverbs10: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:d1:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_11/device/infiniband_verbs/uverbs11: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:d1:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_12/device/infiniband_verbs/uverbs12: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:8e:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_13/device/infiniband_verbs/uverbs13: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:8e:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_14/device/infiniband_verbs/uverbs14: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:91:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_15/device/infiniband_verbs/uverbs15: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:91:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_16/device/infiniband_verbs/uverbs16: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:b1:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_17/device/infiniband_verbs/uverbs17: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:b1:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_2/device/infiniband_verbs/uverbs2: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:55:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_3/device/infiniband_verbs/uverbs3: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:55:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_4/device/infiniband_verbs/uverbs4: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:0e:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_5/device/infiniband_verbs/uverbs5: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:0e:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_6/device/infiniband_verbs/uverbs6: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:11:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_7/device/infiniband_verbs/uverbs7: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:11:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_8/device/infiniband_verbs/uverbs8: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:ce:00.0 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent /sys/class/infiniband/mlx5_9/device/infiniband_verbs/uverbs9: total 0 -r--r--r-- 1 root root 4096 Dec 22 08:21 abi_version -r--r--r-- 1 root root 4096 Dec 22 08:21 dev lrwxrwxrwx 1 root root 0 Dec 22 08:21 device -> ../../../0000:ce:00.1 -r--r--r-- 1 root root 4096 Dec 22 08:21 ibdev drwxr-xr-x 2 root root 0 Dec 22 08:21 power lrwxrwxrwx 1 root root 0 Dec 22 08:21 subsystem -> ../../../../../../../../../../class/infiniband_verbs -rw-r--r-- 1 root root 4096 Dec 22 08:21 uevent ``` </details>
bsakaguchi commented 2026-10-29 21:14:19 +00:00
Author
Owner
Copy link

@staryxchen commented on GitHub (Dec 22, 2025):

Is the nvidia_peermem driver installed?

<!-- gh-comment-id:3681158914 --> @staryxchen commented on GitHub (Dec 22, 2025): Is the nvidia_peermem driver installed?
bsakaguchi commented 2026-10-29 21:14:19 +00:00
Author
Owner
Copy link

@khuonglm commented on GitHub (Dec 22, 2025):

Thank you. it is not installed. I will try to install first and reopen if it is not resolved.

<!-- gh-comment-id:3681196124 --> @khuonglm commented on GitHub (Dec 22, 2025): Thank you. it is not installed. I will try to install first and reopen if it is not resolved.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#421
No description provided.