kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #2722] [Bug] [TE] cudaEventRecord: invalid resource handle when NVLink/MNNVL transfer targets a non-current GPU #710

New issue

Closed

opened 2026-10-29 21:18:31 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 21:18:31 +00:00

Owner

Originally created by @staryxchen on GitHub (Jul 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2722

Originally assigned to: @xiangui33423 on GitHub.

Repro

Reported by @n-WN in the discussion on #2691:

Fresh field confirmation while validating #2714 on an 8×H100 node (CUDA 12.4, -DUSE_CUDA=ON): cross-GPU NVLink transfers (tebench --seg_type=VRAM --local_gpu_id=1 --target_gpu_id=0 --xport_type=nvlink) fail deterministically with NVLinkTransport: cudaEventRecord failed: invalid resource handle at nvlink_transport.cpp:259, on current main. Same-GPU transfers (--local_gpu_id=0 --target_gpu_id=0) work fine (~1.3 TB/s at 4 MB blocks), consistent with the device being inferred from the request pointer and the event/stream then living on the wrong device for the cross-GPU case.

The failure is specific to the case where the buffer's device differs from the calling thread's current CUDA device.

Root cause (from reading the code)

NVLinkTransport::startTransfer (nvlink_transport.cpp:225-233, mirrored in MnnvlTransport::startTransfer at mnnvl_transport.cpp:265-272):

cudaEvent_t event;
cudaEventCreateWithFlags(&event, cudaEventDisableTiming);   // (1)
cudaEventRecord(event, batch->async_stream.get());           // (2)

submitTransferTasks correctly resolves the buffer's device (say device 1) and calls getStreamFromPool(stream, 1).
CUDAStreamPool::DevicePool::acquire (cuda_stream_pool.cpp:70-82) does cudaSetDevice(1) → cudaStreamCreateWithFlags → cudaSetDevice(currentDevice) to restore the caller's device. So the stream lives on device 1, but the caller is back on device 0.
At (1), the event is created on device 0 (current device).
At (2), recording an event that lives on device 0 into a stream that lives on device 1 → cudaErrorInvalidResourceHandle.

Fix direction

Either:

(a) Wrap event creation + record in a cudaSetDevice(target_device) guard, restoring the caller's device afterward. Localized change in each startTransfer.
(b) Keep current device set to the stream's device across the whole startTransfer call (event, memcpy, event record, restore).

Both need to know the batch's target device — which is already computed in submitTransferTasks and can be plumbed to startTransfer via the SubBatch.

Same fix applies to both NVLinkTransport and MnnvlTransport.

Orthogonal to #2620 / #2691. #2691 only changes where the device id comes from (registration-time BufferDesc.location vs. submit-time cudaPointerGetAttributes); in this repro the old path already resolves the right device id. The mismatch is between event device and stream device, not in device inference.

Community fix welcome — happy to review.

Originally created by @staryxchen on GitHub (Jul 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2722 Originally assigned to: @xiangui33423 on GitHub. ## Repro Reported by @n-WN in the discussion on #2691: > Fresh field confirmation while validating #2714 on an 8×H100 node (CUDA 12.4, `-DUSE_CUDA=ON`): cross-GPU NVLink transfers (`tebench --seg_type=VRAM --local_gpu_id=1 --target_gpu_id=0 --xport_type=nvlink`) fail deterministically with `NVLinkTransport: cudaEventRecord failed: invalid resource handle` at `nvlink_transport.cpp:259`, on current main. Same-GPU transfers (`--local_gpu_id=0 --target_gpu_id=0`) work fine (~1.3 TB/s at 4 MB blocks), consistent with the device being inferred from the request pointer and the event/stream then living on the wrong device for the cross-GPU case. The failure is specific to the case where the buffer's device differs from the calling thread's current CUDA device. ## Root cause (from reading the code) `NVLinkTransport::startTransfer` (`nvlink_transport.cpp:225-233`, mirrored in `MnnvlTransport::startTransfer` at `mnnvl_transport.cpp:265-272`): ```cpp cudaEvent_t event; cudaEventCreateWithFlags(&event, cudaEventDisableTiming); // (1) cudaEventRecord(event, batch->async_stream.get()); // (2) ``` - `submitTransferTasks` correctly resolves the buffer's device (say device 1) and calls `getStreamFromPool(stream, 1)`. - `CUDAStreamPool::DevicePool::acquire` (`cuda_stream_pool.cpp:70-82`) does `cudaSetDevice(1)` → `cudaStreamCreateWithFlags` → **`cudaSetDevice(currentDevice)`** to restore the caller's device. So the stream lives on device 1, but the caller is back on device 0. - At (1), the event is created on device 0 (current device). - At (2), recording an event that lives on device 0 into a stream that lives on device 1 → `cudaErrorInvalidResourceHandle`. ## Fix direction Either: - **(a)** Wrap event creation + record in a `cudaSetDevice(target_device)` guard, restoring the caller's device afterward. Localized change in each `startTransfer`. - **(b)** Keep current device set to the stream's device across the whole `startTransfer` call (event, memcpy, event record, restore). Both need to know the batch's target device — which is already computed in `submitTransferTasks` and can be plumbed to `startTransfer` via the `SubBatch`. Same fix applies to both `NVLinkTransport` and `MnnvlTransport`. ## Not related to Orthogonal to #2620 / #2691. #2691 only changes *where* the device id comes from (registration-time `BufferDesc.location` vs. submit-time `cudaPointerGetAttributes`); in this repro the old path already resolves the right device id. The mismatch is between event device and stream device, not in device inference. Community fix welcome — happy to review.

bsakaguchi closed this issue

2026-10-29 21:18:31 +00:00

bsakaguchi commented

2026-10-29 21:18:32 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 2, 2026):

Thanks for opening this issue, @staryxchen!

Field	Value
Issue	#2722
GitHub user ID	`151037142`
Reporter	@staryxchen

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Jul 2, 2026): Thanks for opening this issue, @staryxchen! | Field | Value | |-------|-------| | **Issue** | #2722 | | **GitHub user ID** | `151037142` | | **Reporter** | @staryxchen | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:18:32 +00:00

Author

Owner

@xiangui33423 commented on GitHub (Jul 3, 2026):

Hi, I'd like to take this issue.

I understand the root cause as an event/stream device mismatch: the stream is created on the resolved target GPU, but the event is created on the caller's current CUDA device. I'll try to plumb the target device into startTransfer, guard event creation/recording with the correct CUDA device, restore the previous device afterward, and apply the fix to both NVLink and MNNVL.

Could you assign this to me?

@xiangui33423 commented on GitHub (Jul 3, 2026): Hi, I'd like to take this issue. I understand the root cause as an event/stream device mismatch: the stream is created on the resolved target GPU, but the event is created on the caller's current CUDA device. I'll try to plumb the target device into `startTransfer`, guard event creation/recording with the correct CUDA device, restore the previous device afterward, and apply the fix to both NVLink and MNNVL. Could you assign this to me?

bsakaguchi commented

2026-10-29 21:18:33 +00:00

Author

Owner

@staryxchen commented on GitHub (Jul 3, 2026):

Hi, I'd like to take this issue.

I understand the root cause as an event/stream device mismatch: the stream is created on the resolved target GPU, but the event is created on the caller's current CUDA device. I'll try to plumb the target device into startTransfer, guard event creation/recording with the correct CUDA device, restore the previous device afterward, and apply the fix to both NVLink and MNNVL.

Could you assign this to me?

OK, I've assigned this issue to you. Before you submit a PR, please make sure to verify on your local machine that the code compiles successfully and that the issue has been resolved. Thanks for your contribution in advance.

@staryxchen commented on GitHub (Jul 3, 2026): > Hi, I'd like to take this issue. > > I understand the root cause as an event/stream device mismatch: the stream is created on the resolved target GPU, but the event is created on the caller's current CUDA device. I'll try to plumb the target device into `startTransfer`, guard event creation/recording with the correct CUDA device, restore the previous device afterward, and apply the fix to both NVLink and MNNVL. > > Could you assign this to me? OK, I've assigned this issue to you. Before you submit a PR, please make sure to verify on your local machine that the code compiles successfully and that the issue has been resolved. Thanks for your contribution in advance.

bsakaguchi commented

2026-10-29 21:18:33 +00:00

Author

Owner

@xiangui33423 commented on GitHub (Jul 3, 2026):

Thanks for assigning this to me. I'll verify the build locally and test the reported NVLink/MNNVL cross-GPU case before submitting the PR.

@xiangui33423 commented on GitHub (Jul 3, 2026): Thanks for assigning this to me. I'll verify the build locally and test the reported NVLink/MNNVL cross-GPU case before submitting the PR.

bsakaguchi commented

2026-10-29 21:18:33 +00:00

Author

Owner

@n-WN commented on GitHub (Jul 6, 2026):

Field verification of #2754 on the original multi-GPU CUDA-enabled repro host: the exact command that failed deterministically before (tebench --seg_type=VRAM --local_gpu_id=1 --target_gpu_id=0 --xport_type=nvlink) now runs clean on latest main (f1f5e40) — no cudaEventRecord: invalid resource handle, cross-GPU reads sustain 140→372 GB/s from 256 KB to 4 MB blocks (vs. ~1.3 TB/s for the same-GPU path, consistent with NVLink vs. HBM). Thanks @xiangui33423 for the fix and @staryxchen for the fast triage.

@n-WN commented on GitHub (Jul 6, 2026): Field verification of #2754 on the original multi-GPU CUDA-enabled repro host: the exact command that failed deterministically before (`tebench --seg_type=VRAM --local_gpu_id=1 --target_gpu_id=0 --xport_type=nvlink`) now runs clean on latest main (f1f5e40) — no `cudaEventRecord: invalid resource handle`, cross-GPU reads sustain 140→372 GB/s from 256 KB to 4 MB blocks (vs. ~1.3 TB/s for the same-GPU path, consistent with NVLink vs. HBM). Thanks @xiangui33423 for the fix and @staryxchen for the fast triage.

bsakaguchi referenced this issue

2026-10-29 21:30:07 +00:00

[PR #710] [CLOSED] [WIP] Use VRAM as buffer #1761