[PR #683] [MERGED] bugfix(nvlink): Add explicit P2P access enablement and error handling for NvlinkTransport #1740

New issue

Closed

opened 2026-10-29 21:29:57 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:29:57 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/683
Author: @staryxchen
Created: 7/28/2025
Status: ✅ Merged
Merged: 8/1/2025
Merged by: @alogfans

Base: main ← Head: bugfix/nvlink

📝 Commits (4)

2680738 refactor(nvlink_transport): cache device count to avoid repeated queries
8919309 bugfix(nvlink): add explicit P2P access enablement and error handling
2b065b3 Added checkCudaErrorReturn function and updated enableP2PAccess to properly handle CUDA errors by returning false on failure.
2cf9dfd update something according to comment

📊 Changes

1 file changed (+97 additions, -9 deletions)

View changed files

📝 mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp (+97 -9)

📄 Description

Summary

This PR adds explicit peer-to-peer (P2P) access enablement and improved error handling to the NvlinkTransport class. The changes ensure that bidirectional P2P access is properly established between all GPU device pairs during transport initialization.

Changes

Added

checkCudaError() helper function for consistent CUDA error handling
enableP2PAccess() function to explicitly enable bidirectional P2P access between device pairs
Enhanced constructor logic to enable P2P access between all available GPU devices
Improved error logging with detailed CUDA error codes and messages

Modified

getNumDevices() now uses the new checkCudaError() helper for consistent error handling
NvlinkTransport constructor now iterates through all device pairs and enables P2P access
Added trace logging for P2P access enablement status

Technical Details

The implementation follows NVIDIA's recommended practices for P2P access:

Peer Access Query: Uses cudaDeviceCanAccessPeer() to check if devices support P2P access
Bidirectional Enablement: Enables P2P access in both directions (src→dst and dst→src)
Error Handling: Comprehensive error checking with detailed error messages

References

Impact

This fix resolves potential issues where P2P access might not be properly established, leading to runtime errors during GPU-to-GPU transfers. The explicit enablement ensures reliable P2P communication in multi-GPU environments.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/683 **Author:** [@staryxchen](https://github.com/staryxchen) **Created:** 7/28/2025 **Status:** ✅ Merged **Merged:** 8/1/2025 **Merged by:** [@alogfans](https://github.com/alogfans) **Base:** `main` ← **Head:** `bugfix/nvlink` --- ### 📝 Commits (4) - [`2680738`](https://github.com/kvcache-ai/Mooncake/commit/2680738828b2ff49ef35349e2393df24d379b058) refactor(nvlink_transport): cache device count to avoid repeated queries - [`8919309`](https://github.com/kvcache-ai/Mooncake/commit/891930911f534de71cfea8c7a9c686e73aebbc7b) bugfix(nvlink): add explicit P2P access enablement and error handling - [`2b065b3`](https://github.com/kvcache-ai/Mooncake/commit/2b065b3658d3982c25290370ae93e3ce6fd08fec) Added checkCudaErrorReturn function and updated enableP2PAccess to properly handle CUDA errors by returning false on failure. - [`2cf9dfd`](https://github.com/kvcache-ai/Mooncake/commit/2cf9dfd012199ae9e1493b4fe9f84fb51cbd830a) update something according to comment ### 📊 Changes **1 file changed** (+97 additions, -9 deletions) <details> <summary>View changed files</summary> 📝 `mooncake-transfer-engine/src/transport/nvlink_transport/nvlink_transport.cpp` (+97 -9) </details> ### 📄 Description ## Summary This PR adds explicit peer-to-peer (P2P) access enablement and improved error handling to the NvlinkTransport class. The changes ensure that bidirectional P2P access is properly established between all GPU device pairs during transport initialization. ## Changes ### Added - `checkCudaError()` helper function for consistent CUDA error handling - `enableP2PAccess()` function to explicitly enable bidirectional P2P access between device pairs - Enhanced constructor logic to enable P2P access between all available GPU devices - Improved error logging with detailed CUDA error codes and messages ### Modified - `getNumDevices()` now uses the new `checkCudaError()` helper for consistent error handling - `NvlinkTransport` constructor now iterates through all device pairs and enables P2P access - Added trace logging for P2P access enablement status ## Technical Details The implementation follows NVIDIA's recommended practices for P2P access: 1. **Peer Access Query**: Uses `cudaDeviceCanAccessPeer()` to check if devices support P2P access 2. **Bidirectional Enablement**: Enables P2P access in both directions (src→dst and dst→src) 4. **Error Handling**: Comprehensive error checking with detailed error messages ## References - [NVIDIA CUDA Samples - Simple P2P](https://github.com/NVIDIA/cuda-samples/blob/master/Samples/0_Introduction/simpleP2P/simpleP2P.cu#L93C1-L132C62) - [NVIDIA CUDA Runtime API - Peer Device Memory](https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__PEER.html) ## Impact This fix resolves potential issues where P2P access might not be properly established, leading to runtime errors during GPU-to-GPU transfers. The explicit enablement ensures reliable P2P communication in multi-GPU environments. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:29:57 +00:00