[GH-ISSUE #4596] [Bug]: TENT MnnvlTransport passes &fd to cuMemImportFromShareableHandle, POSIX-FD mode never imports #1173

New issue

Closed

opened 2026-10-29 21:23:35 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:23:35 +00:00

Owner

Originally created by @SongOf on GitHub (Oct 12, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4596

Bug Report

Version: origin/main b198a191
Environment: TENT MNNVL transport with transports/nvlink/handle_type=posix_fd. Observed on 2× NVIDIA L20, driver 580.126.09, CUDA 12.8, Ubuntu 22.04; not hardware specific.

Description

MnnvlTransport::relocateSharedMemoryAddress imports a peer buffer in POSIX-FD mode with

github.com/kvcache-ai/Mooncake@b198a191/mooncake-transfer-engine/tent/src/transport/mnnvl/mnnvl_transport.cpp (L818)

CHECK_CU(cuMemImportFromShareableHandle(&handle, &local_fd, handle_type_));

For CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR, cuMemImportFromShareableHandle expects osHandle to be the fd value itself cast to void * (the export side, cuMemExportToShareableHandle, is the one that takes a pointer to an int). Passing &local_fd makes the driver treat a stack address as the fd and fail with CUDA_ERROR_UNKNOWN (999). CHECK_CU returns an error status and submitTransferTasks then hits LOG(FATAL), aborting the process.

Consequence: MNNVL in POSIX-FD mode cannot import any peer buffer. The export side (line 562) and importFdFromProcess (pidfd_getfd) both work, so the fd arrives in the consumer correctly; only the final driver call is wrong. The Fabric path is unaffected.

git blame points at e8db9e54 (2025-12-22, initial TENT import), so the FD path has been broken since it landed. There is no FD-mode test in the tree, which is why it was never caught.

Steps to reproduce

Two processes on one VMM-capable GPU, TENT with MC_ENABLE_MNNVL=1, handle_type=posix_fd, TCP/RDMA disabled.
P0 allocateLocalMemory + registerLocalMemory; C opens the segment and submits a READ.
C aborts:
```
CUDA error: cuMemImportFromShareableHandle ... 999
... Unable to relocate ... (LOG(FATAL))
```
With the single line changed to (void *)(uintptr_t)local_fd the READ completes with the right data. A standalone CUDA program shows the same: &fd → 999, (void *)(uintptr_t)fd → success.

Found while reproducing #4592 (the repro program there needs this fix to reach the mapping-cache bug).

Proposed fix

Pass (void *)(uintptr_t)local_fd, and add a GPU test that imports a POSIX-FD handle exported by a forked child. PR to follow.

Before submitting...

Ensure you searched for relevant issues and read the documentation

Originally created by @SongOf on GitHub (Oct 12, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4596 ## Bug Report **Version:** origin/main `b198a191` **Environment:** TENT MNNVL transport with `transports/nvlink/handle_type=posix_fd`. Observed on 2× NVIDIA L20, driver 580.126.09, CUDA 12.8, Ubuntu 22.04; not hardware specific. ### Description `MnnvlTransport::relocateSharedMemoryAddress` imports a peer buffer in POSIX-FD mode with https://github.com/kvcache-ai/Mooncake/blob/b198a191/mooncake-transfer-engine/tent/src/transport/mnnvl/mnnvl_transport.cpp#L818 ```cpp CHECK_CU(cuMemImportFromShareableHandle(&handle, &local_fd, handle_type_)); ``` For `CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR`, `cuMemImportFromShareableHandle` expects `osHandle` to be the fd value itself cast to `void *` (the export side, `cuMemExportToShareableHandle`, is the one that takes a pointer to an `int`). Passing `&local_fd` makes the driver treat a stack address as the fd and fail with `CUDA_ERROR_UNKNOWN` (999). `CHECK_CU` returns an error status and `submitTransferTasks` then hits `LOG(FATAL)`, aborting the process. Consequence: **MNNVL in POSIX-FD mode cannot import any peer buffer.** The export side (line 562) and `importFdFromProcess` (pidfd_getfd) both work, so the fd arrives in the consumer correctly; only the final driver call is wrong. The Fabric path is unaffected. `git blame` points at e8db9e54 (2025-12-22, initial TENT import), so the FD path has been broken since it landed. There is no FD-mode test in the tree, which is why it was never caught. ### Steps to reproduce 1. Two processes on one VMM-capable GPU, TENT with `MC_ENABLE_MNNVL=1`, `handle_type=posix_fd`, TCP/RDMA disabled. 2. P0 `allocateLocalMemory` + `registerLocalMemory`; C opens the segment and submits a READ. 3. C aborts: ``` CUDA error: cuMemImportFromShareableHandle ... 999 ... Unable to relocate ... (LOG(FATAL)) ``` With the single line changed to `(void *)(uintptr_t)local_fd` the READ completes with the right data. A standalone CUDA program shows the same: `&fd` → 999, `(void *)(uintptr_t)fd` → success. Found while reproducing #4592 (the repro program there needs this fix to reach the mapping-cache bug). ### Proposed fix Pass `(void *)(uintptr_t)local_fd`, and add a GPU test that imports a POSIX-FD handle exported by a forked child. PR to follow. ## Before submitting... - [x] Ensure you searched for relevant issues and read the documentation

bsakaguchi closed this issue

2026-10-29 21:23:35 +00:00

bsakaguchi commented

2026-10-29 21:23:36 +00:00

Author

Owner

@SongOf commented on GitHub (Oct 12, 2026):

Closing: this is small enough to go straight to a PR. Fix and regression test are in #4597; context is in the reproduction report on #4592.

@SongOf commented on GitHub (Oct 12, 2026): Closing: this is small enough to go straight to a PR. Fix and regression test are in #4597; context is in the reproduction report on #4592.

bsakaguchi commented

2026-10-29 21:23:36 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 12, 2026):

Thanks for opening this issue, @SongOf!

Field	Value
Issue	#4596
GitHub user ID	`46475785`
Reporter	@SongOf

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 12, 2026): Thanks for opening this issue, @SongOf! | Field | Value | |-------|-------| | **Issue** | #4596 | | **GitHub user ID** | `46475785` | | **Reporter** | @SongOf | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:24 +00:00

[PR #1173] [MERGED] [EP] Implement elastic scaling up #2050

bsakaguchi referenced this issue

2026-10-29 21:32:50 +00:00

[PR #1252] [MERGED] [Doc] update toc item of ep-backend #2105