[GH-ISSUE #587] Why doesn't low-latency dispatch use system-level fence before issuing IBGDA writes? #360

New issue

Open

opened 2026-10-26 09:59:07 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:59:07 +00:00

Owner

Originally created by @hexclover on GitHub (Mar 2, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/587

(Clarification: https://github.com/deepseek-ai/DeepEP/issues/113 also inquires about the __threadfence in ibgda_post_send, but in this issue I am asking specifically about the consistency guarantee of the token data, not the WQE itself, so this is technically not a duplicate 😃)

To quote from internode_ll::dispatch, it first performs casting and writes FP8 data to the send buffer:

github.com/deepseek-ai/DeepEP@567632dd59/csrc/kernels/internode_ll.cu (L213-L252)

Then it issues IBGDA writes. nvshmemi_ibgda_put_nbi_warp is implemented such that the request is posted to NIC when slot_idx % 4 == 3.

github.com/deepseek-ai/DeepEP@567632dd59/csrc/kernels/internode_ll.cu (L267)

The problem is I don't see any use of system-level primitives here, to guarantee the NIC will read the freshly cast data from the send buffer when it composes IB packets. The closest thing I see is the __threadfence in ibgda_post_send, but it is not system-level (and I'm curious about that one, too).

Originally created by @hexclover on GitHub (Mar 2, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/587 (Clarification: https://github.com/deepseek-ai/DeepEP/issues/113 also inquires about the `__threadfence` in `ibgda_post_send`, but in this issue I am asking specifically about the consistency guarantee of the token data, not the WQE itself, so this is technically not a duplicate 😃) To quote from `internode_ll::dispatch`, it first performs casting and writes FP8 data to the send buffer: https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/csrc/kernels/internode_ll.cu#L213-L252 Then it issues IBGDA writes. `nvshmemi_ibgda_put_nbi_warp` is implemented such that the request is posted to NIC when `slot_idx % 4 == 3`. https://github.com/deepseek-ai/DeepEP/blob/567632dd59810d77b3cc05553df953cc0f779799/csrc/kernels/internode_ll.cu#L267 The problem is I don't see any use of system-level primitives here, to guarantee **the NIC will read the freshly cast data from the send buffer when it composes IB packets**. The closest thing I see is the `__threadfence` in `ibgda_post_send`, but it is not system-level (and I'm curious about that one, too).

bsakaguchi commented

2026-10-26 09:59:08 +00:00

Author

Owner

@hexclover commented on GitHub (Mar 2, 2026):

Grepping the original NVSHMEM code, I see both of __threadfence{,_system} are used in src/include/non_abi/device/pt-to-pt/ibgda_device.cuh, such as the examples below:

github.com/NVIDIA/nvshmem@fcd704cc46/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh (L222-L244)

github.com/NVIDIA/nvshmem@fcd704cc46/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh (L1629)

github.com/NVIDIA/nvshmem@fcd704cc46/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh (L2238-L2239)

Which one to choose from the two flavors of fences seems to depend on flags including nic_buf_on_gpumem and is_data_buf_in_sysmem. If those flags simply mean whether the buffer is located on host memory or GPU memory, then it seems NVSHMEM also considers a plain __threadfence is sufficient in these use cases. If this is the case, then this and #113 can probably be answered in the same way as https://github.com/deepseek-ai/DeepEP/issues/113#issuecomment-2785394086:

However, NVSHMEM implements it this way, and I believe that as an internal NVIDIA team, they have more insight into these details and can ensure the safety of this approach.

@hexclover commented on GitHub (Mar 2, 2026): Grepping the original NVSHMEM code, I see both of `__threadfence{,_system}` are used in `src/include/non_abi/device/pt-to-pt/ibgda_device.cuh`, such as the examples below: https://github.com/NVIDIA/nvshmem/blob/fcd704cc4604cdb728d81d952605109949d4fb95/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh#L222-L244 https://github.com/NVIDIA/nvshmem/blob/fcd704cc4604cdb728d81d952605109949d4fb95/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh#L1629 https://github.com/NVIDIA/nvshmem/blob/fcd704cc4604cdb728d81d952605109949d4fb95/src/include/non_abi/device/pt-to-pt/ibgda_device.cuh#L2238-L2239 Which one to choose from the two flavors of fences seems to depend on flags including `nic_buf_on_gpumem` and `is_data_buf_in_sysmem`. If those flags simply mean whether the buffer is located on host memory or GPU memory, then it seems NVSHMEM also considers a plain `__threadfence` is sufficient in these use cases. If this is the case, then this and #113 can probably be answered in the same way as https://github.com/deepseek-ai/DeepEP/issues/113#issuecomment-2785394086: > However, NVSHMEM implements it this way, and I believe that as an internal NVIDIA team, they have more insight into these details and can ensure the safety of this approach.

bsakaguchi referenced this issue

2026-10-27 21:03:45 +00:00

[PR #360] [MERGED] [fix] handle empty tensor in per_token_cast_back #535