deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #30] When testing test_internode.py, what is the effect of setting NVSHMEM_DISABLE_P2P=1? Will NVLink be disabled? #28

New issue
Open
opened 2026-10-26 09:52:46 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 09:52:46 +00:00
Owner
Copy link

Originally created by @MoringKing on GitHub (Feb 28, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/30

Image
Originally created by @MoringKing on GitHub (Feb 28, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/30 <img width="544" alt="Image" src="https://github.com/user-attachments/assets/7d3f7f9f-8585-464f-b518-c7d7e8d9c483" />
bsakaguchi commented 2026-10-26 09:52:47 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Feb 28, 2025):

After setting NVSHMEM_DISABLE_P2P=1, you cannot use NVSHMEM for NVLink transfers. However, this is not an issue in our implementation, as our NVLink data transfers does not rely on NVSHMEM API. Instead, we directly utilize CUDA PTX instructions for NVLink data transfers.

<!-- gh-comment-id:2690296005 --> @sphish commented on GitHub (Feb 28, 2025): After setting NVSHMEM_DISABLE_P2P=1, you cannot use NVSHMEM for NVLink transfers. However, this is not an issue in our implementation, as our NVLink data transfers does not rely on NVSHMEM API. Instead, we directly utilize CUDA PTX instructions for NVLink data transfers.
bsakaguchi commented 2026-10-26 09:52:47 +00:00
Author
Owner
Copy link

@MoringKing commented on GitHub (Mar 1, 2025):

Thank you for your reply. Further, I want to confirm, what is the purpose of setting NVSHMEM_DISABLE_P2P=1 in the low_latency scenario? And what is the impact if NVSHMEM_DISABLE_P2P=1 is set in internode scenarios?

Image
<!-- gh-comment-id:2692020115 --> @MoringKing commented on GitHub (Mar 1, 2025): Thank you for your reply. Further, I want to confirm, what is the purpose of setting NVSHMEM_DISABLE_P2P=1 in the low_latency scenario? And what is the impact if NVSHMEM_DISABLE_P2P=1 is set in internode scenarios? <img width="885" alt="Image" src="https://github.com/user-attachments/assets/d90358ed-dffe-436d-977b-45d931a54806" />
bsakaguchi commented 2026-10-26 09:52:47 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Mar 3, 2025):

Enabling this environment variable also works, but we disable it to ensure we are not using NVLink through NVSHMEM.

<!-- gh-comment-id:2693873690 --> @sphish commented on GitHub (Mar 3, 2025): Enabling this environment variable also works, but we disable it to ensure we are not using NVLink through NVSHMEM.
bsakaguchi commented 2026-10-26 09:52:47 +00:00
Author
Owner
Copy link

@GitAlice123 commented on GitHub (Mar 11, 2025):

Enabling this environment variable also works, but we disable it to ensure we are not using NVLink through NVSHMEM.

#But when I set NVSHMEM_DISABLE_P2P to 0 and run low latency test on two nodes (16 GPUs), I get the following error:

Image

<!-- gh-comment-id:2713978216 --> @GitAlice123 commented on GitHub (Mar 11, 2025): > Enabling this environment variable also works, but we disable it to ensure we are not using NVLink through NVSHMEM. #But when I set NVSHMEM_DISABLE_P2P to 0 and run low latency test on two nodes (16 GPUs), I get the following error: ![Image](https://github.com/user-attachments/assets/b6a3397c-c8ec-43a4-ad6d-b47843e132c8)
bsakaguchi commented 2026-10-26 09:52:47 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Mar 17, 2025):

Enabling this environment variable also works, but we disable it to ensure we are not using NVLink through NVSHMEM.

#But when I set NVSHMEM_DISABLE_P2P to 0 and run low latency test on two nodes (16 GPUs), I get the following error:

Image

It appears that the program is failing during the bootstrap phase of NVSHMEM, which doesn't seem reasonable, but I'm not sure why this is happening.

<!-- gh-comment-id:2727842885 --> @sphish commented on GitHub (Mar 17, 2025): > > Enabling this environment variable also works, but we disable it to ensure we are not using NVLink through NVSHMEM. > > #But when I set NVSHMEM_DISABLE_P2P to 0 and run low latency test on two nodes (16 GPUs), I get the following error: > > ![Image](https://github.com/user-attachments/assets/b6a3397c-c8ec-43a4-ad6d-b47843e132c8) It appears that the program is failing during the bootstrap phase of NVSHMEM, which doesn't seem reasonable, but I'm not sure why this is happening.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#28
No description provided.