[GH-ISSUE #50] Why NUM_MAX_NVL_PEERS must be 8? #48

New issue

Open

opened 2026-10-26 09:53:12 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-26 09:53:12 +00:00

Owner

Originally created by @Huixxi on GitHub (Mar 6, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/50

In deep_ep.hpp, Can it be a smaller number? For example I only have a node with 2 H800 to run test_low_latency.py

Originally created by @Huixxi on GitHub (Mar 6, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/50 In deep_ep.hpp, Can it be a smaller number? For example I only have a node with 2 H800 to run test_low_latency.py

bsakaguchi commented

2026-10-26 09:53:13 +00:00

Author

Owner

@LyricZhao commented on GitHub (Mar 6, 2025):

If you only have one node, EP2 is supported for both intranode kernels (via NVLink) and low-latency kernels (via RDMA).

For multiple nodes with each less than 8 GPUs, you can change the NUM_MAX_NVL_PEERS macro into your settings to see whether the kernels work. We may later add a compilation macro for this. Thanks for feedback.

@LyricZhao commented on GitHub (Mar 6, 2025): If you only have one node, EP2 is supported for both intranode kernels (via NVLink) and low-latency kernels (via RDMA). For multiple nodes with each less than 8 GPUs, you can change the `NUM_MAX_NVL_PEERS` macro into your settings to see whether the kernels work. We may later add a compilation macro for this. Thanks for feedback.

bsakaguchi commented

2026-10-26 09:53:14 +00:00

Author

Owner

@alokprasad commented on GitHub (Nov 3, 2025):

@alokprasad commented on GitHub (Nov 3, 2025): Related issue https://github.com/deepseek-ai/DeepEP/issues/477

bsakaguchi commented

2026-10-26 09:53:14 +00:00

Author

Owner

@elvircrn commented on GitHub (Nov 5, 2025):

@LyricZhao Hitting this assert when switching NUM_MAX_NVL_PEERS to 4:

DeepEP/csrc/kernels/internode.cu(295): error: static assertion failed with "Invalid number of NVL peers"
                    static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers");
                    ^

DeepEP/csrc/kernels/internode.cu(507): error: static assertion failed with "Invalid number of NVL peers"
        static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers");

@elvircrn commented on GitHub (Nov 5, 2025): @LyricZhao Hitting this assert when switching NUM_MAX_NVL_PEERS to 4: ``` DeepEP/csrc/kernels/internode.cu(295): error: static assertion failed with "Invalid number of NVL peers" static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers"); ^ DeepEP/csrc/kernels/internode.cu(507): error: static assertion failed with "Invalid number of NVL peers" static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers"); ```

bsakaguchi commented

2026-10-26 09:53:14 +00:00

Author

Owner

@annali07 commented on GitHub (Nov 10, 2025):

Encountered the same issue.
And when I attempt to resolve by commenting out the assertions, I get a new error:

/miniconda3/envs/sglang/lib/python3.10/site-packages/deep_ep/buffer.py", line 135, in __init__ self.runtime.sync(device_ids, ipc_handles, root_unique_id) RuntimeError: Failed: CUDA error /home/annali/sglang/DeepEP/csrc/deep_ep.cpp:113 'invalid resource handle'

I'm on the latest branch of DeepEP with NUM_MAX_NVL_PEERS=2

@LyricZhao Hitting this assert when switching NUM_MAX_NVL_PEERS to 4:

DeepEP/csrc/kernels/internode.cu(295): error: static assertion failed with "Invalid number of NVL peers"
                    static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers");
                    ^

DeepEP/csrc/kernels/internode.cu(507): error: static assertion failed with "Invalid number of NVL peers"
        static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers");

@annali07 commented on GitHub (Nov 10, 2025): Encountered the same issue. And when I attempt to resolve by commenting out the assertions, I get a new error: `/miniconda3/envs/sglang/lib/python3.10/site-packages/deep_ep/buffer.py", line 135, in __init__ self.runtime.sync(device_ids, ipc_handles, root_unique_id) RuntimeError: Failed: CUDA error /home/annali/sglang/DeepEP/csrc/deep_ep.cpp:113 'invalid resource handle'` I'm on the latest branch of DeepEP with NUM_MAX_NVL_PEERS=2 > [@LyricZhao](https://github.com/LyricZhao) Hitting this assert when switching NUM_MAX_NVL_PEERS to 4: > > ``` > DeepEP/csrc/kernels/internode.cu(295): error: static assertion failed with "Invalid number of NVL peers" > static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers"); > ^ > > DeepEP/csrc/kernels/internode.cu(507): error: static assertion failed with "Invalid number of NVL peers" > static_assert(4 * sizeof(bool) == sizeof(uint64_t), "Invalid number of NVL peers"); > ```

bsakaguchi commented

2026-10-26 09:53:15 +00:00

Author

Owner

@jgcb00 commented on GitHub (Feb 2, 2026):

Same problem here, did anyone found a solution ?

@jgcb00 commented on GitHub (Feb 2, 2026): Same problem here, did anyone found a solution ?

bsakaguchi referenced this issue

2026-10-26 09:56:30 +00:00

[GH-ISSUE #270] A timeout error occurred when running test_internode.py on four devices. #200