[GH-ISSUE #426] cached_notify Slowdown When Offloading Activations to CPU #287

New issue

Closed

opened 2026-10-26 09:58:00 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:58:00 +00:00

Owner

Originally created by @yellow-banjo on GitHub (Sep 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/426

Hello, DeepEP team! Thank you for your great project!

I've encountered a significant slowdown in the cached_notify operation within internode_combine when offloading dispatch activations to the CPU.

My setup uses 8 H100 nodes with 8 GPUs each.

Here are the results from three runs:

Baseline: No activations are saved during the forward pass; all are recomputed during the backward pass.
Async CPU Offload: Dispatch activations are asynchronously offloaded to pinned CPU memory using a separate copy stream, and then reloaded to the GPU during the backward pass.
Sync CPU Offload: The tensors are offloaded to the CPU synchronously (not asynchronously).

The slowdown is noticeable even when offloading is done synchronously, and it occurs significantly before the internode_combine step.

Could you please advise on what might be causing this performance degradation?

Originally created by @yellow-banjo on GitHub (Sep 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/426 Hello, DeepEP team! Thank you for your great project! I've encountered a significant slowdown in the cached_notify operation within internode_combine when offloading dispatch activations to the CPU. My setup uses 8 H100 nodes with 8 GPUs each. Here are the results from three runs: 1) Baseline: No activations are saved during the forward pass; all are recomputed during the backward pass. 2) Async CPU Offload: Dispatch activations are asynchronously offloaded to pinned CPU memory using a separate copy stream, and then reloaded to the GPU during the backward pass. 3) Sync CPU Offload: The tensors are offloaded to the CPU synchronously (not asynchronously). The slowdown is noticeable even when offloading is done synchronously, and it occurs significantly before the internode_combine step. <img width="1671" height="879" alt="Image" src="https://github.com/user-attachments/assets/b6530477-1fea-48d6-82b6-1592d0818509" /> <img width="1669" height="887" alt="Image" src="https://github.com/user-attachments/assets/d7a6a2d7-a172-475e-9f13-c7dc62ad0d4f" /> <img width="1670" height="888" alt="Image" src="https://github.com/user-attachments/assets/878d8194-4113-4fc4-8724-b76fdde7ec62" /> Could you please advise on what might be causing this performance degradation?

bsakaguchi closed this issue

2026-10-26 09:58:00 +00:00

bsakaguchi commented

2026-10-26 09:58:01 +00:00

Author

Owner

@sphish commented on GitHub (Sep 24, 2025):

cached_notify contains a sync operation, so its execution time will include the gap between different ranks. I recommend checking the profiles of all ranks to see if the DtoH on some ranks is significantly slower than on others (in my experience, this is quite likely to happen).

@sphish commented on GitHub (Sep 24, 2025): `cached_notify` contains a `sync` operation, so its execution time will include the gap between different ranks. I recommend checking the profiles of all ranks to see if the `DtoH` on some ranks is significantly slower than on others (in my experience, this is quite likely to happen).

bsakaguchi commented

2026-10-26 09:58:01 +00:00

Author

Owner

@yellow-banjo commented on GitHub (Sep 24, 2025):

Thank you for your response!

In the case of synchronous offloading, the varying execution time of the DtoH copy could indeed affect the timing of cached_notify.

Could you please explain how the DtoH operation influences cached_notify even when the offloading is done asynchronously?

@yellow-banjo commented on GitHub (Sep 24, 2025): Thank you for your response! In the case of synchronous offloading, the varying execution time of the DtoH copy could indeed affect the timing of cached_notify. Could you please explain how the DtoH operation influences cached_notify even when the offloading is done asynchronously?

bsakaguchi commented

2026-10-26 09:58:02 +00:00

Author

Owner

@sphish commented on GitHub (Sep 26, 2025):

This will need to be evaluated alongside your profiling results. My guess is that the primary reason is that asynchronous DtoH transfers affect the performance of dispatch/combine, since they compete for PCIe bandwidth.

@sphish commented on GitHub (Sep 26, 2025): This will need to be evaluated alongside your profiling results. My guess is that the primary reason is that asynchronous DtoH transfers affect the performance of dispatch/combine, since they compete for PCIe bandwidth.

bsakaguchi commented

2026-10-26 09:58:02 +00:00

Author

Owner

@yellow-banjo commented on GitHub (Sep 26, 2025):

Thank you for your response!

@yellow-banjo commented on GitHub (Sep 26, 2025): Thank you for your response!

bsakaguchi referenced this issue

2026-10-27 21:03:19 +00:00

[PR #287] [MERGED] Use TMA to optimize internode combine. #504