[GH-ISSUE #401] Why is so much memory used when initializing nvshmem? test_intranode #273

New issue

Open

opened 2026-10-26 09:57:45 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:57:45 +00:00

Owner

Originally created by @wxk-cmd on GitHub (Sep 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/401

I created a large tensor and found that initializing nvshmem takes up a huge amount of memory.

    test_ll_compatibility, num_rdma_bytes = True,128
    if test_ll_compatibility:
        ll_num_tokens, ll_hidden, ll_num_experts, ll_num_topk = 16, 5120, 256, 9
        num_rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint(ll_num_tokens, ll_hidden, num_ranks, ll_num_experts)
        buffer = deep_ep.Buffer(group, int(2e9), num_rdma_bytes, low_latency_mode=test_ll_compatibility,
                            num_qps_per_rank=(ll_num_experts // num_ranks if test_ll_compatibility else 1), explicitly_destroy=True)
    tensor = torch.empty(138*2**30,device = "cuda",dtype = torch.int8)
    torch.manual_seed(rank)

Originally created by @wxk-cmd on GitHub (Sep 11, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/401 <img width="1093" height="111" alt="Image" src="https://github.com/user-attachments/assets/ff810de7-4bb0-4860-a054-1596227434fa" /> I created a large tensor and found that initializing nvshmem takes up a huge amount of memory. ``` test_ll_compatibility, num_rdma_bytes = True,128 if test_ll_compatibility: ll_num_tokens, ll_hidden, ll_num_experts, ll_num_topk = 16, 5120, 256, 9 num_rdma_bytes = deep_ep.Buffer.get_low_latency_rdma_size_hint(ll_num_tokens, ll_hidden, num_ranks, ll_num_experts) buffer = deep_ep.Buffer(group, int(2e9), num_rdma_bytes, low_latency_mode=test_ll_compatibility, num_qps_per_rank=(ll_num_experts // num_ranks if test_ll_compatibility else 1), explicitly_destroy=True) tensor = torch.empty(138*2**30,device = "cuda",dtype = torch.int8) torch.manual_seed(rank) ```

bsakaguchi commented

2026-10-26 09:57:47 +00:00

Author

Owner

@sphish commented on GitHub (Sep 24, 2025):

The CUDA and Torch contexts also consume some GPU memory, so your current testing method isn’t accurate. You can initialize the Torch and CUDA contexts beforehand, then print the GPU memory usage before and after creating DeepEP to get an exact difference.

@sphish commented on GitHub (Sep 24, 2025): The CUDA and Torch contexts also consume some GPU memory, so your current testing method isn’t accurate. You can initialize the Torch and CUDA contexts beforehand, then print the GPU memory usage before and after creating DeepEP to get an exact difference.

bsakaguchi referenced this issue

2026-10-27 21:03:10 +00:00

[PR #273] [MERGED] use cli arg for num_processes #494