[GH-ISSUE #518] [bug]Assert error run test_internode.py on EP128 #331

New issue

Open

opened 2026-10-26 09:58:43 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 09:58:43 +00:00

Owner

Originally created by @huakainomore on GitHub (Dec 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/518

when run normal kernel use EP128, this assert will fail

Assertion error  num_max_nvl_chunked_recv_tokens / num_rdma_ranks > std::max(num_max_rdma_chunked_send_tokens, num_max_nvl_chunked_send_tokens)'

because the num_max_nvl_chunked_send_tokens will be set to bigger than num_max_nvl_chunked_recv_tokens / num_rdma_ranks when tuning

for nvl_chunk_size in range(4, *45*, 4):  // test_internode.py

it seems should add nvl_buffer_size to fix it

rdma_buffer_size, nvl_buffer_size = 128, (720 if num_ranks in (24, 48, 96, 144, 160) else 512) 
-> rdma_buffer_size, nvl_buffer_size = 128, (720 if num_ranks in (24, 48, 96, 128,144, 160) else 512)

Originally created by @huakainomore on GitHub (Dec 11, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/518 when run normal kernel use EP128, this assert will fail ``` Assertion error num_max_nvl_chunked_recv_tokens / num_rdma_ranks > std::max(num_max_rdma_chunked_send_tokens, num_max_nvl_chunked_send_tokens)' ``` because the `num_max_nvl_chunked_send_tokens` will be set to bigger than `num_max_nvl_chunked_recv_tokens / num_rdma_ranks` when tuning ``` for nvl_chunk_size in range(4, *45*, 4): // test_internode.py ``` it seems should add nvl_buffer_size to fix it ``` rdma_buffer_size, nvl_buffer_size = 128, (720 if num_ranks in (24, 48, 96, 144, 160) else 512) -> rdma_buffer_size, nvl_buffer_size = 128, (720 if num_ranks in (24, 48, 96, 128,144, 160) else 512) ```

bsakaguchi referenced this issue

2026-10-27 21:03:38 +00:00

[PR #331] Support prefill with 2 GPUs #521