deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #491] Question about eager rdma #321

New issue

Closed

opened 2026-10-26 09:58:32 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 09:58:32 +00:00

Owner

Originally created by @Thunderbrook on GitHub (Nov 13, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/491

hello~ @Gaojiaqi I’d like to ask some questions about eager rdma.

what is the purpose of the flag "CU_POINTER_ATTRIBUTE_SYNC_MEMOPS". I understand that it only affects the asynchronous behavior of cudaMemcpy, is it right? In nvshmem, the static heap and qp control object uses this flag, but the default dynamic heap does not.
I see that the tag only uses 4 bytes out of the 16 bytes allocated. Why was a 16-byte space designed for it?
thx~

Originally created by @Thunderbrook on GitHub (Nov 13, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/491 hello~ @Gaojiaqi I’d like to ask some questions about eager rdma. 1. what is the purpose of the flag "CU_POINTER_ATTRIBUTE_SYNC_MEMOPS". I understand that it only affects the asynchronous behavior of cudaMemcpy, is it right? In nvshmem, the static heap and qp control object uses this flag, but the default dynamic heap does not. 2. I see that the tag only uses 4 bytes out of the 16 bytes allocated. Why was a 16-byte space designed for it? thx~

bsakaguchi closed this issue

2026-10-26 09:58:33 +00:00

bsakaguchi commented

2026-10-26 09:58:34 +00:00

Author

Owner

@Gaojiaqi commented on GitHub (Dec 7, 2025):

Hi @Thunderbrook,

After double check the setting, CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is indeed not enabled on dynamic heap during our internal pressure-testing, so it is not necessary (or potential data race still exist but with really low chance).
16-byte tag is for int4 alignments, it has the minimal impact to the current dispatch pipeline.

@Gaojiaqi commented on GitHub (Dec 7, 2025): Hi @Thunderbrook, 1. After double check the setting, CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is indeed not enabled on dynamic heap during our internal pressure-testing, so it is not necessary (or potential data race still exist but with really low chance). 2. 16-byte tag is for int4 alignments, it has the minimal impact to the current dispatch pipeline.

bsakaguchi commented

2026-10-26 09:58:35 +00:00

Author

Owner

@Thunderbrook commented on GitHub (Dec 17, 2025):

@Gaojiaqi thanks for reply~, but i wonder where exactly does the data race mentioned here occur? And how CU_POINTER_ATTRIBUTE_SYNC_MEMOPS solve the data race, thx~

@Thunderbrook commented on GitHub (Dec 17, 2025): @Gaojiaqi thanks for reply~, but i wonder where exactly does the data race mentioned here occur? And how CU_POINTER_ATTRIBUTE_SYNC_MEMOPS solve the data race, thx~

bsakaguchi commented

2026-10-26 09:58:35 +00:00

Author

Owner

@Gaojiaqi commented on GitHub (Jan 8, 2026):

Hey, the data race we meant was the DMA writing from the NIC to the GPU. When an RDMA packet arrives, for example with 4k MTU, the data is split into muliple writes from the NIC to the GPU memory. We were worried the mulitple writes were not in order, since the tag is only in the last write (at the tail of the 4096 bytes). We might experience correctness issue if reorder happens between the GPU and the NIC. That's why we disabled NVSHMEM_IB_ENABLE_RELAXED_ORDERING. CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is irrelevant this problem.

@Gaojiaqi commented on GitHub (Jan 8, 2026): Hey, the data race we meant was the DMA writing from the NIC to the GPU. When an RDMA packet arrives, for example with 4k MTU, the data is split into muliple writes from the NIC to the GPU memory. We were worried the mulitple writes were not in order, since the tag is only in the last write (at the tail of the 4096 bytes). We might experience correctness issue if reorder happens between the GPU and the NIC. That's why we disabled NVSHMEM_IB_ENABLE_RELAXED_ORDERING. CU_POINTER_ATTRIBUTE_SYNC_MEMOPS is irrelevant this problem.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#321

No description provided.

Rows
Columns