[GH-ISSUE #352] low latency test error with "dispatch_use_fp8 and round_scale and use_ue8m0" #245

New issue

Closed

opened 2026-10-26 09:57:16 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 09:57:16 +00:00

Owner

Originally created by @dongwang4096 on GitHub (Aug 5, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/352

In file DeepEP/tests/test_low_latency.py, I added the code (Line 79-83 ):

73 # Check received data
74 if current_x is not x_pure_rand:
75 recv_x = recv_x[:num_valid_tokens]
76 recv_x_amin = recv_x[:, :-128].amin(dim=-1)
77 recv_src_info = recv_src_info[:num_valid_tokens]
78 assert torch.equal(recv_x_amin, recv_x[:, :-128].amax(dim=-1))
79 if dispatch_use_fp8 and round_scale and use_ue8m0 :
80 recv_x_last = recv_x[:, -1]
81 if (recv_x_last < -1).any():
82 print(round_scale,use_ue8m0)
83 print(f"rank {rank}, num_times {num_times}, expert_id: {expert_id}, recv_x_last contains values less than -1!, nu m_valid_tokens: {num_valid_tokens}, recv_x_last: {recv_x_last}")

When running the test with dispatch_use_fp8=True, round_scale=True, and use_ue8m0=True on GB200, the condition (recv_x_last < -1).any() is triggered (Line 83 ). This indicates that invalid negative values are appearing in recv_x_last, which should not happen.

Command：python DeepEP/tests/test_low_latency.py --num-processes 4 --num-tokens 768

Originally created by @dongwang4096 on GitHub (Aug 5, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/352 In file DeepEP/tests/test_low_latency.py, I added the code (Line 79-83 ): 73 # Check received data 74 if current_x is not x_pure_rand: 75 recv_x = recv_x[:num_valid_tokens] 76 recv_x_amin = recv_x[:, :-128].amin(dim=-1) 77 recv_src_info = recv_src_info[:num_valid_tokens] 78 assert torch.equal(recv_x_amin, recv_x[:, :-128].amax(dim=-1)) **79 if dispatch_use_fp8 and round_scale and use_ue8m0 : 80 recv_x_last = recv_x[:, -1] 81 if (recv_x_last < -1).any(): 82 print(round_scale,use_ue8m0) 83 print(f"rank {rank}, num_times {num_times}, expert_id: {expert_id}, recv_x_last contains values less than -1!, nu m_valid_tokens: {num_valid_tokens}, recv_x_last: {recv_x_last}")** When running the test with dispatch_use_fp8=True, round_scale=True, and use_ue8m0=True on GB200, the condition (recv_x_last < -1).any() is triggered (Line 83 ). This indicates that invalid negative values are appearing in recv_x_last, which should not happen. Command：python DeepEP/tests/test_low_latency.py --num-processes 4 --num-tokens 768

bsakaguchi closed this issue

2026-10-26 09:57:17 +00:00

bsakaguchi commented

2026-10-26 09:57:18 +00:00

Author

Owner

@shifangx commented on GitHub (Aug 22, 2025):

It has be fixed after PR345
by changing from
x_scales = x_scales.view(dtype=torch.int8).to(torch.int) << 23
to
x_scales = x_scales.view(dtype=torch.uint8).to(torch.int) << 23

Root cause is that the function per_token_cast_back() in test case does not treat scale dtype correctly.

@shifangx commented on GitHub (Aug 22, 2025): It has be fixed after [PR345](https://github.com/deepseek-ai/DeepEP/pull/345) by changing from x_scales = x_scales.view(dtype=torch.int8).to(torch.int) << 23 to x_scales = x_scales.view(dtype=torch.uint8).to(torch.int) << 23 Root cause is that the function per_token_cast_back() in test case does not treat scale dtype correctly. <img width="994" height="304" alt="Image" src="https://github.com/user-attachments/assets/e32f8a67-dbff-480e-9edf-4e13f69258d8" />

bsakaguchi commented

2026-10-26 09:57:19 +00:00

Author

Owner

@polarstormx commented on GitHub (Sep 18, 2026):

This has been addressed in #345 (commit c5facf5), so closing as resolved.

@polarstormx commented on GitHub (Sep 18, 2026): This has been addressed in #345 (commit c5facf5), so closing as resolved.

bsakaguchi referenced this issue

2026-10-26 09:57:21 +00:00

[GH-ISSUE #361] H20's internode RDMA & NVL bandwidth drops to 50% after pr 245 #249

bsakaguchi referenced this issue

2026-10-26 10:01:16 +00:00

[PR #245] [MERGED] Add the transaction window data structure for RDMA senders #480