[GH-ISSUE #4980] [Bug][Store]: dummy-client CUDA IPC buffers are opened on the exporter's device ordinal, which is wrong whenever the real client and the dummy client see different `CUDA_VISIBLE_DEVICES` #1263

New issue

Open

opened 2026-10-29 21:24:35 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:35 +00:00

Owner

Originally created by @0z5a on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4980

Describe the bug

The dummy client exports a registered GPU buffer with cudaIpcGetMemHandle and records attr.device (its own ordinal, cuda_ipc_buffer.cpp). The real client then does cudaSetDevice(handle.device_id) + cudaIpcOpenMemHandle with that number (cuda_ipc_buffer.cpp), and the dummy path has no staged fallback (dummy_client.cpp). Device ordinals are per-process: in the common layout where the real client (store service) sees all GPUs and each worker process sees only its own GPU (CUDA_VISIBLE_DEVICES=<i>), every worker reports device 0, so the real client maps every worker's buffer on its own GPU 0.

Without GPU P2P the open fails and put/get return INVALID_PARAMS (-600).
With NVLink/P2P it "works", but the mapping lives in the wrong device context, so every put/get on workers 1..N crosses NVLink instead of staying local (not measured yet).

TENT's NVLink transport documents exactly this pitfall and deliberately does not switch to the peer's ordinal before cudaIpcOpenMemHandle (tent/src/transport/nvlink/nvlink_transport.cpp, comment above the IPC open: "that ordinal is the PEER's device index, relative to the peer's CUDA_VISIBLE_DEVICES").

To reproduce

2× L40S (PCIe, P2P disabled: topo -p2p r = NS), nightly wheel 0.3.14.dev20261025, TCP protocol, mooncake_master + mooncake_client (real client) with CUDA_VISIBLE_DEVICES=0,1, and a dummy client (MooncakeDistributedStore().setup_dummy(...)) that registers a 1 MiB GPU buffer and does put_from / get_into:

real client CVD	dummy CVD	buffer on physical GPU	real client opens on physical GPU	put / get
0,1	1	1	0	-600 / -600
0,1	0	0	0	OK / OK (1 MiB, byte-exact)
0,1	0,1 (dev 1)	1	1	OK / OK

Expected behavior

Do not reuse the exporter's ordinal: either open on the real client's intended device as TENT does, or carry the GPU UUID / PCI bus id in the exported handle and resolve it in the real client's own numbering; fail with a clear message if that GPU is not visible to the real client.

AI assistance disclosure

Found by AI-assisted code reading (Claude) and confirmed on the hardware above.

Originally created by @0z5a on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4980 ### Describe the bug The dummy client exports a registered GPU buffer with `cudaIpcGetMemHandle` and records `attr.device` (its own ordinal, `cuda_ipc_buffer.cpp`). The real client then does `cudaSetDevice(handle.device_id)` + `cudaIpcOpenMemHandle` with that number (`cuda_ipc_buffer.cpp`), and the dummy path has no staged fallback (`dummy_client.cpp`). Device ordinals are per-process: in the common layout where the real client (store service) sees all GPUs and each worker process sees only its own GPU (`CUDA_VISIBLE_DEVICES=<i>`), every worker reports device 0, so the real client maps every worker's buffer on its own GPU 0. - Without GPU P2P the open fails and put/get return `INVALID_PARAMS` (-600). - With NVLink/P2P it "works", but the mapping lives in the wrong device context, so every put/get on workers 1..N crosses NVLink instead of staying local (not measured yet). TENT's NVLink transport documents exactly this pitfall and deliberately does not switch to the peer's ordinal before `cudaIpcOpenMemHandle` (`tent/src/transport/nvlink/nvlink_transport.cpp`, comment above the IPC open: "that ordinal is the PEER's device index, relative to the peer's CUDA_VISIBLE_DEVICES"). ### To reproduce 2× L40S (PCIe, P2P disabled: `topo -p2p r` = NS), nightly wheel `0.3.14.dev20261025`, TCP protocol, `mooncake_master` + `mooncake_client` (real client) with `CUDA_VISIBLE_DEVICES=0,1`, and a dummy client (`MooncakeDistributedStore().setup_dummy(...)`) that registers a 1 MiB GPU buffer and does `put_from` / `get_into`: | real client CVD | dummy CVD | buffer on physical GPU | real client opens on physical GPU | put / get | |---|---|---|---|---| | 0,1 | 1 | 1 | 0 | **-600 / -600** | | 0,1 | 0 | 0 | 0 | OK / OK (1 MiB, byte-exact) | | 0,1 | 0,1 (dev 1) | 1 | 1 | OK / OK | ### Expected behavior Do not reuse the exporter's ordinal: either open on the real client's intended device as TENT does, or carry the GPU UUID / PCI bus id in the exported handle and resolve it in the real client's own numbering; fail with a clear message if that GPU is not visible to the real client. ### AI assistance disclosure Found by AI-assisted code reading (Claude) and confirmed on the hardware above.

bsakaguchi commented

2026-10-29 21:24:36 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 27, 2026):

Thanks for opening this issue, @0z5a!

Field	Value
Issue	#4980
GitHub user ID	`192209249`
Reporter	@0z5a

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 27, 2026): Thanks for opening this issue, @0z5a! | Field | Value | |-------|-------| | **Issue** | #4980 | | **GitHub user ID** | `192209249` | | **Reporter** | @0z5a | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:55 +00:00

[PR #1263] [MERGED] [CI] Disable certain tests in CI configuration #2117