deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #12] Benifits of FP8 cast and AR #10

New issue
Closed
opened 2026-10-26 09:52:21 +00:00 by bsakaguchi · 5 comments
bsakaguchi commented 2026-10-26 09:52:21 +00:00
Owner
Copy link

Originally created by @xiaoyao9933 on GitHub (Feb 25, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/12

Thank you very much for your open-source work! After reading the code, I have two questions:

1 Why is FP8 Cast only enabled in the low-latency mode? Wouldn't using FP8 communication in normal mode also achieve double bandwidth benefits?
2 The low-latency mode doesn't support packet forwarding, while normal mode requires forwarding. Does this imply your network topology resembles a multi-rail spine-leaf architecture? If forwarding isn't enabled on the rail plane, wouldn't hash polarization between leaf and spine nodes become severe, thus necessitating AR mode activation?

Originally created by @xiaoyao9933 on GitHub (Feb 25, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/12 Thank you very much for your open-source work! After reading the code, I have two questions: 1 Why is FP8 Cast only enabled in the low-latency mode? Wouldn't using FP8 communication in normal mode also achieve double bandwidth benefits? 2 The low-latency mode doesn't support packet forwarding, while normal mode requires forwarding. Does this imply your network topology resembles a multi-rail spine-leaf architecture? If forwarding isn't enabled on the rail plane, wouldn't hash polarization between leaf and spine nodes become severe, thus necessitating AR mode activation?
bsakaguchi commented 2026-10-26 09:52:22 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Feb 25, 2025):

  1. Normal mode dispatch supports FP8, the x input can be a tuple of FP8 tensor and scaling factors. But different with low-latency kernels, you should cast FP8 by yourself before launching dispatch.
  2. Sorry, we cannot tell much about our cluster network.
<!-- gh-comment-id:2681187390 --> @LyricZhao commented on GitHub (Feb 25, 2025): 1. Normal mode `dispatch` supports FP8, the `x` input can be a tuple of FP8 tensor and scaling factors. But different with low-latency kernels, you should cast FP8 by yourself before launching dispatch. 2. Sorry, we cannot tell much about our cluster network.
bsakaguchi commented 2026-10-26 09:52:22 +00:00
Author
Owner
Copy link

@xiaoyao9933 commented on GitHub (Feb 25, 2025):

Got it. Many thanks. @LyricZhao

<!-- gh-comment-id:2681209171 --> @xiaoyao9933 commented on GitHub (Feb 25, 2025): Got it. Many thanks. @LyricZhao
bsakaguchi commented 2026-10-26 09:52:22 +00:00
Author
Owner
Copy link

@Puellaquae commented on GitHub (Feb 25, 2025):

I'm a bit confused about this. Using FP8 can halve the amount of data transmitted, but is FP8 essential for low-latency mode, can we choose Bfloat16? Or was this decision specifically tailored to your model?

<!-- gh-comment-id:2681747992 --> @Puellaquae commented on GitHub (Feb 25, 2025): I'm a bit confused about this. Using FP8 can halve the amount of data transmitted, but is FP8 essential for low-latency mode, can we choose Bfloat16? Or was this decision specifically tailored to your model?
bsakaguchi commented 2026-10-26 09:52:22 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Feb 25, 2025):

I'm a bit confused about this. Using FP8 can halve the amount of data transmitted, but is FP8 essential for low-latency mode, can we choose Bfloat16? Or was this decision specifically tailored to your model?

Sorry, low-latency dispatch currently only supports FP8, but it is not very hard for supporting BF16 (maybe later).

<!-- gh-comment-id:2681793714 --> @LyricZhao commented on GitHub (Feb 25, 2025): > I'm a bit confused about this. Using FP8 can halve the amount of data transmitted, but is FP8 essential for low-latency mode, can we choose Bfloat16? Or was this decision specifically tailored to your model? Sorry, low-latency dispatch currently only supports FP8, but it is not very hard for supporting BF16 (maybe later).
bsakaguchi commented 2026-10-26 09:52:22 +00:00
Author
Owner
Copy link

@LyricZhao commented on GitHub (Mar 10, 2025):

@Puellaquae

Sorry, low-latency dispatch currently only supports FP8, but it is not very hard for supporting BF16 (maybe later).

Supported now, see github.com/deepseek-ai/DeepEP@ed7487c15e.

<!-- gh-comment-id:2709935401 --> @LyricZhao commented on GitHub (Mar 10, 2025): @Puellaquae > Sorry, low-latency dispatch currently only supports FP8, but it is not very hard for supporting BF16 (maybe later). Supported now, see https://github.com/deepseek-ai/DeepEP/commit/ed7487c15e219161f86656ed6604aa9877bf72bc.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#10
No description provided.