deepseek-ai/DeepEP

Fork 0

mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00

[GH-ISSUE #560] Hash Validation in test_low_latency.py #344

New issue

Closed

opened 2026-10-26 09:58:52 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:58:52 +00:00

Owner

Originally created by @Hizhaoyuan on GitHub (Jan 9, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/560

Question: Hash Validation in test_low_latency.py - Dependency on Receive Order?

Observation

In tests/test_low_latency.py , the test validates received tokens by hashing the entire received buffer for each expert:

if dispatch_use_fp8:
hash_value ^= hash_tensor(packed_recv_x[0][i, :num_valid_tokens])
hash_value ^= hash_tensor(packed_recv_x[1][i, :num_valid_tokens])
else:
hash_value ^= hash_tensor(packed_recv_x[i, :num_valid_tokens])

From reading csrc/kernels/internode_ll.cu , I noticed that the placement of received tokens uses an atomic operation:

recv_token_begin_idx = atomicAdd(packed_recv_count + local_expert_idx, num_recv_tokens);Based on this, my understanding is:

Each expert receives tokens from multiple ranks
The atomicAdd determines where each rank's tokens are placed in the buffer
The order depends on which rank's data arrives first (network timing, SM scheduling, etc.)
Therefore, even with identical inputs, tokens from different ranks could be arranged in different orders in packed_recv_x across multiple runs

Is my understanding correct? If so, wouldn't hashing the entire buffer be sensitive to this non-deterministic ordering?

Current

My understanding is incorrect, OR
The execution is deterministic enough in practice (stable GPU scheduling, stable network), OR
There's something I'm missing about the synchronization/ordering guarantees

Question

Could you clarify:

Is the receive order guaranteed to be deterministic in some way I'm not seeing?
Or does the test happen to work due to practical determinism in typical environments?

Suggested Solution (if my understanding is correct)

If the receive order is indeed non-deterministic, we could hash per-rank segments using packed_recv_layout_range:

for j in range(num_ranks):
begin_idx = (recv_layout_range[j] >> 32).item()
count = (recv_layout_range[j] & int_mask).item()
if count > 0:
if dispatch_use_fp8:
hash_value ^= hash_tensor(packed_recv_x[0][i, begin_idx:begin_idx + count])
hash_value ^= hash_tensor(packed_recv_x[1][i, begin_idx:begin_idx + count])
else:
hash_value ^= hash_tensor(packed_recv_x[i, begin_idx:begin_idx + count])This would:

Eliminate dependency on receive order
Validate correctness at per-rank granularity
Make tests more robust across different environments

However, I'd appreciate your confirmation on whether this concern is valid before proposing any changes.

Thank you for the excellent work on DeepEP!

Originally created by @Hizhaoyuan on GitHub (Jan 9, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/560 # Question: Hash Validation in test_low_latency.py - Dependency on Receive Order? ## Observation In `tests/test_low_latency.py` , the test validates received tokens by hashing the entire received buffer for each expert: if dispatch_use_fp8: hash_value ^= hash_tensor(packed_recv_x[0][i, :num_valid_tokens]) hash_value ^= hash_tensor(packed_recv_x[1][i, :num_valid_tokens]) else: hash_value ^= hash_tensor(packed_recv_x[i, :num_valid_tokens]) From reading `csrc/kernels/internode_ll.cu` , I noticed that the placement of received tokens uses an atomic operation: recv_token_begin_idx = atomicAdd(packed_recv_count + local_expert_idx, num_recv_tokens);Based on this, my understanding is: 1. Each expert receives tokens from multiple ranks 2. The `atomicAdd` determines where each rank's tokens are placed in the buffer 3. The order depends on which rank's data arrives first (network timing, SM scheduling, etc.) 4. Therefore, even with identical inputs, tokens from different ranks could be arranged in different orders in `packed_recv_x` across multiple runs **Is my understanding correct?** If so, wouldn't hashing the entire buffer be sensitive to this non-deterministic ordering? ## Current - My understanding is incorrect, OR - The execution is deterministic enough in practice (stable GPU scheduling, stable network), OR - There's something I'm missing about the synchronization/ordering guarantees ## Question Could you clarify: 1. Is the receive order guaranteed to be deterministic in some way I'm not seeing? 2. Or does the test happen to work due to practical determinism in typical environments? ## Suggested Solution (if my understanding is correct) If the receive order is indeed non-deterministic, we could hash per-rank segments using `packed_recv_layout_range`: for j in range(num_ranks): begin_idx = (recv_layout_range[j] >> 32).item() count = (recv_layout_range[j] & int_mask).item() if count > 0: if dispatch_use_fp8: hash_value ^= hash_tensor(packed_recv_x[0][i, begin_idx:begin_idx + count]) hash_value ^= hash_tensor(packed_recv_x[1][i, begin_idx:begin_idx + count]) else: hash_value ^= hash_tensor(packed_recv_x[i, begin_idx:begin_idx + count])This would: - Eliminate dependency on receive order - Validate correctness at per-rank granularity - Make tests more robust across different environments However, I'd appreciate your confirmation on whether this concern is valid before proposing any changes. Thank you for the excellent work on DeepEP!

bsakaguchi closed this issue

2026-10-26 09:58:52 +00:00

bsakaguchi commented

2026-10-26 09:58:54 +00:00

Author

Owner

@Hizhaoyuan commented on GitHub (Jan 9, 2026):

I've reviewed the hash_tensor implementation. The function uses sum() which is mathematically order-insensitive (commutative). Closing this issue as the implementation is correct by design.

@Hizhaoyuan commented on GitHub (Jan 9, 2026): I've reviewed the `hash_tensor` implementation. The function uses `sum()` which is mathematically order-insensitive (commutative). Closing this issue as the implementation is correct by design.

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepEP#344

No description provided.

Rows
Columns