[GH-ISSUE #462] Does deepEP support PCIe H800s？ #299

New issue

Open

opened 2026-10-26 09:58:11 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-26 09:58:11 +00:00

Owner

Originally created by @Weigaa on GitHub (Oct 20, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/462

I see the nvlink is a requirement in this repository.
But i wonder if deepep can be used on totally pcie 8-h800 server for single node training ?
It cannot run or just run slower?
Thank u for your greate work.

Originally created by @Weigaa on GitHub (Oct 20, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/462 I see the nvlink is a requirement in this repository. But i wonder if deepep can be used on totally pcie 8-h800 server for single node training ? It cannot run or just run slower? Thank u for your greate work.

bsakaguchi commented

2026-10-26 09:58:13 +00:00

Author

Owner

@sphish commented on GitHub (Oct 23, 2025):

We haven’t tested it on a pure PCIe system. Compared with NVLink, PCIe provides fewer guarantees for atomicity, so I think errors might occur.

@sphish commented on GitHub (Oct 23, 2025): We haven’t tested it on a pure PCIe system. Compared with NVLink, PCIe provides fewer guarantees for atomicity, so I think errors might occur.

bsakaguchi commented

2026-10-26 09:58:14 +00:00

Author

Owner

@Weigaa commented on GitHub (Oct 24, 2025):

@sphish Thank you for your response.
We have successfully install deepep low_latency mode and run with vllm 0.11.0.
And we use pure conda cuda env (12.8) to block sys cuda (13.0)
We only have one node with 8 pcie-connected h800s.
The most related env as followings:

libnvshmem3                             3.4.5
nvshmem4py                              0.1.2.36438984
deep-ep      1.2.1+73b6ea4

However, the running time result is not ideal because it seems higher than basic "allgather + reduce scatter" backend in vllm.
The result is running with 4 gpus in 1 node for olmoe-1b-7b post training in verl in gsm8k (5 steps result for bsz, end to end time):

deep_ep 313.459 tokens/s
allgather+reduce scatter 314.768 tokens/s

Not show obvious diffrences.
Thank u again for your help.

@Weigaa commented on GitHub (Oct 24, 2025): @sphish Thank you for your response. We have successfully install deepep low_latency mode and run with vllm 0.11.0. And we use pure conda cuda env (12.8) to block sys cuda (13.0) We only have one node with 8 pcie-connected h800s. The most related env as followings: ``` libnvshmem3 3.4.5 nvshmem4py 0.1.2.36438984 deep-ep 1.2.1+73b6ea4 ``` However, the running time result is not ideal because it seems higher than basic "allgather + reduce scatter" backend in vllm. The result is running with 4 gpus in 1 node for olmoe-1b-7b post training in verl in gsm8k (5 steps result for bsz, end to end time): ``` deep_ep 313.459 tokens/s allgather+reduce scatter 314.768 tokens/s ``` Not show obvious diffrences. Thank u again for your help.

bsakaguchi commented

2026-10-26 09:58:14 +00:00

Author

Owner

@sphish commented on GitHub (Oct 24, 2025):

NCCL has implemented many topology-aware optimizations for PCIe systems, whereas DeepEP has not been specifically optimized for PCIe. I believe this is the main reason.

@sphish commented on GitHub (Oct 24, 2025): NCCL has implemented many topology-aware optimizations for PCIe systems, whereas DeepEP has not been specifically optimized for PCIe. I believe this is the main reason.

bsakaguchi commented

2026-10-26 09:58:14 +00:00

Author

Owner

@Weigaa commented on GitHub (Oct 26, 2025):

Thank you @sphish , and I want to make sure if we use low_latency (ll) mode, If a GPU has multiple experts and it so happens that a certain token needs to be passed to k experts of this GPU, will this result in passing the token k times instead of just 1 time (like in normal mode)?

If so, I think there seems are some redundant communication overhead?

@Weigaa commented on GitHub (Oct 26, 2025): Thank you @sphish , and I want to make sure if we use low_latency (ll) mode, If a GPU has multiple experts and it so happens that a certain token needs to be passed to k experts of this GPU, will this result in passing the token k times instead of just 1 time (like in normal mode)? If so, I think there seems are some redundant communication overhead?

bsakaguchi referenced this issue

2026-10-27 21:03:23 +00:00

[PR #299] [MERGED] Optimize low latency combine send with TMA #508