[PR #533] Add barrier between cudaIpcCloseMemHandle and cudaFree in Buffer::destroy #616

New issue

Open

opened 2026-10-27 21:04:35 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-27 21:04:35 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepEP/pull/533
Author: @yurekami
Created: 12/25/2025
Status: 🔄 Open

Base: main ← Head: fix/ipc-handle-close-barrier

📝 Commits (1)

3d2826e Add barrier between cudaIpcCloseMemHandle and cudaFree in Buffer::destroy

📊 Changes

1 file changed (+6 additions, -0 deletions)

View changed files

📝 csrc/deep_ep.cpp (+6 -0)

📄 Description

Summary

Add a barrier between closing remote IPC handles and freeing local buffer
Prevents undefined behavior per CUDA docs for cudaIpcOpenMemHandle
Ensures all ranks close remote handles before any rank frees its buffer

Background

According to CUDA documentation, it is undefined behavior to cudaFree an exported memory region before cudaIpcCloseMemHandle is called on all processes that opened the handle.

Without this fix, in a multi-GPU scenario:

Rank 1 finishes closing remote handles and proceeds to free its local buffer
Rank 2 is still trying to close the handle to Rank 1's buffer
This is UB since the underlying memory has been freed

Test plan

Verify Buffer::destroy completes without hangs on multi-GPU systems
Run stress tests with explicit buffer destruction

Fixes #497

🤖 Generated with Claude Code

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepEP/pull/533 **Author:** [@yurekami](https://github.com/yurekami) **Created:** 12/25/2025 **Status:** 🔄 Open **Base:** `main` ← **Head:** `fix/ipc-handle-close-barrier` --- ### 📝 Commits (1) - [`3d2826e`](https://github.com/deepseek-ai/DeepEP/commit/3d2826e286c6e87ef0c92ff1193a8a9b321e350f) Add barrier between cudaIpcCloseMemHandle and cudaFree in Buffer::destroy ### 📊 Changes **1 file changed** (+6 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `csrc/deep_ep.cpp` (+6 -0) </details> ### 📄 Description ## Summary - Add a barrier between closing remote IPC handles and freeing local buffer - Prevents undefined behavior per CUDA docs for cudaIpcOpenMemHandle - Ensures all ranks close remote handles before any rank frees its buffer ## Background According to CUDA documentation, it is undefined behavior to `cudaFree` an exported memory region before `cudaIpcCloseMemHandle` is called on all processes that opened the handle. Without this fix, in a multi-GPU scenario: 1. Rank 1 finishes closing remote handles and proceeds to free its local buffer 2. Rank 2 is still trying to close the handle to Rank 1's buffer 3. This is UB since the underlying memory has been freed ## Test plan - [ ] Verify Buffer::destroy completes without hangs on multi-GPU systems - [ ] Run stress tests with explicit buffer destruction Fixes #497 🤖 Generated with [Claude Code](https://claude.com/claude-code) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-27 21:04:35 +00:00

bsakaguchi referenced this issue

2026-10-27 21:05:01 +00:00

[PR #616] [MERGED] Fix DOCA build/path regressions from NIXL integration (#606) #665

bsakaguchi referenced this issue

2026-10-27 21:05:08 +00:00

[PR #625] [MERGED] Optimization of the standalone permute path #672