[GH-ISSUE #212] [Bug] Possible Head/Tail Management and Memory Leak Problem for Normal Mode internode Dispatch Operation #171

New issue

Closed

opened 2026-10-26 09:55:42 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:55:42 +00:00

Owner

Originally created by @ZhiyiHu1999 on GitHub (Jun 16, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/212

Hi, in the normal internode diapatch, we have the kRDMANVLForwarder warp to update the forward_channel_head and forward_channel_retired , letting kNVLCoordinator know the forwarding progress.
github.com/deepseek-ai/DeepEP@21efbe9b48/csrc/kernels/internode.cu (L792)
github.com/deepseek-ai/DeepEP@21efbe9b48/csrc/kernels/internode.cu (L803)

However, I wonder if there is the possibility that, when the kRDMANVLForwarder performs the last update (updating the forward_channel_head and setting the forward_channel_retired to true), and before the kNVLCoordinator have read the newest forward_channel_head value, it read forward_channel_retired which is set to true. So, the newest forwarding head value may not be written to the sender side.

github.com/deepseek-ai/DeepEP@21efbe9b48/csrc/kernels/internode.cu (L826-L827)

Therefore, I want to know if there is the above head/tail management problem. if so, will it cause some problem, for example, memory leak when we perform the next dispatch which reuses the same buffer.

Thanks a lot for you help! And if you could confirm the problem, i am willing to submit a PR for it! Thanks again!

Originally created by @ZhiyiHu1999 on GitHub (Jun 16, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/212 Hi, in the normal internode diapatch, we have the `kRDMANVLForwarder` warp to update the `forward_channel_head` and `forward_channel_retired` , letting `kNVLCoordinator` know the forwarding progress. https://github.com/deepseek-ai/DeepEP/blob/21efbe9b4857ea4d8b0aa0fe8fc5eb673a6a1942/csrc/kernels/internode.cu#L792 https://github.com/deepseek-ai/DeepEP/blob/21efbe9b4857ea4d8b0aa0fe8fc5eb673a6a1942/csrc/kernels/internode.cu#L803 However, I wonder if there is the possibility that, when the `kRDMANVLForwarder` performs the last update (updating the `forward_channel_head` and setting the `forward_channel_retired` to true), and before the `kNVLCoordinator` have read the newest `forward_channel_head` value, it read `forward_channel_retired` which is set to true. So, the newest forwarding head value may not be written to the sender side. https://github.com/deepseek-ai/DeepEP/blob/21efbe9b4857ea4d8b0aa0fe8fc5eb673a6a1942/csrc/kernels/internode.cu#L826-L827 Therefore, I want to know if there is the above head/tail management problem. if so, will it cause some problem, for example, memory leak when we perform the next dispatch which reuses the same buffer. Thanks a lot for you help! And if you could confirm the problem, i am willing to submit a PR for it! Thanks again!

bsakaguchi closed this issue

2026-10-26 09:55:42 +00:00

bsakaguchi commented

2026-10-26 09:55:43 +00:00

Author

Owner

@sphish commented on GitHub (Jun 16, 2025):

I don't think there is an issue here. As shown in the code you referenced, kNVLCoordinator will compute the min_head. When a forwarder retires, it is effectively the same as setting its forward_channel_head to the maximum value.

@sphish commented on GitHub (Jun 16, 2025): I don't think there is an issue here. As shown in the code you referenced, `kNVLCoordinator` will compute the `min_head`. When a forwarder retires, it is effectively the same as setting its `forward_channel_head` to the maximum value.

bsakaguchi commented

2026-10-26 09:55:44 +00:00

Author

Owner

@LyricZhao commented on GitHub (Jun 16, 2025):

When a forwarder retires, it is effectively the same as setting its forward_channel_head to the maximum value.

"The maximum value" means the INT_MAX.

"Head" means the minimum (by index) slot that the receivers wait to consume. If a warp is retired, it means that the warp has already consumed all the required slots, so it is safe to release such slots.

@LyricZhao commented on GitHub (Jun 16, 2025): > When a forwarder retires, it is effectively the same as setting its forward_channel_head to the maximum value. "The maximum value" means the `INT_MAX`. **"Head" means the minimum (by index) slot that the receivers wait to consume.** If a warp is retired, it means that the warp has already consumed all the required slots, so it is safe to release such slots.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#171

No description provided.

Rows
Columns