deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #404] Request for CUDA Graph Integration Example with low_latency_dispatch in MoE Workflow #274

New issue
Closed
opened 2026-10-26 09:57:48 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-26 09:57:48 +00:00
Owner
Copy link

Originally created by @tarjen on GitHub (Sep 14, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/404

Description

I'm trying to use CUDA Graph with low_latency_dispatch but getting incorrect results. The README mentions buffer status restoration for CUDA Graph replay, but I need clarification on:

How to properly initialize the buffers for CUDA Graph?

Could you provide a complete working example?

Current Issues
Tensor lifetime management with from_blob

Handling async operations during graph capture

Buffer restoration process during replay

Specific Questions

How should I initialize the buffers for CUDA Graph compatibility?

Can you provide a complete CUDA Graph integration example with low_latency_dispatch?

Pseudo-code Example

# Pre-allocate input buffers  
input_ptr = pre_allocated_memory  
topk_idx_ptr = pre_allocated_indices  

# Start graph capture  
graph.begin_capture()  

# Create tensors via from_blob  
input_tensor = torch.from_blob(input_ptr, [token_count, hidden_dim], dtype=torch.bfloat16)  
topk_idx_tensor = torch.from_blob(topk_idx_ptr, [token_count, topk], dtype=torch.int64)  

# Call low latency dispatch  
recv_states, expert_count, handle, event, hook = low_latency_dispatch(  
    input_tensor, topk_idx_tensor, max_tokens_per_rank, num_experts  
)  

# Perform expert computations (GEMM) and result combination  
# ...  

# End capture  
graph.end_capture()

Specific Issues

  1. The results after graph replay are incorrect, suggesting potential issues with:
    • Tensor lifetime management using from_blob
    • Asynchronous operations during graph capture
    • Buffer restoration during graph replay

Request

Could you provide:

  1. A complete example showing proper CUDA Graph integration with low_latency_dispatch?
  2. Any limitations or constraints when using this approach?

Thank you for your support!

Originally created by @tarjen on GitHub (Sep 14, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/404 ### Description I'm trying to use CUDA Graph with low_latency_dispatch but getting incorrect results. The README mentions buffer status restoration for CUDA Graph replay, but I need clarification on: How to properly initialize the buffers for CUDA Graph? Could you provide a complete working example? Current Issues Tensor lifetime management with from_blob Handling async operations during graph capture Buffer restoration process during replay ### Specific Questions How should I initialize the buffers for CUDA Graph compatibility? Can you provide a complete CUDA Graph integration example with low_latency_dispatch? ### Pseudo-code Example ```python # Pre-allocate input buffers input_ptr = pre_allocated_memory topk_idx_ptr = pre_allocated_indices # Start graph capture graph.begin_capture() # Create tensors via from_blob input_tensor = torch.from_blob(input_ptr, [token_count, hidden_dim], dtype=torch.bfloat16) topk_idx_tensor = torch.from_blob(topk_idx_ptr, [token_count, topk], dtype=torch.int64) # Call low latency dispatch recv_states, expert_count, handle, event, hook = low_latency_dispatch( input_tensor, topk_idx_tensor, max_tokens_per_rank, num_experts ) # Perform expert computations (GEMM) and result combination # ... # End capture graph.end_capture() ``` ### Specific Issues 1. The results after graph replay are incorrect, suggesting potential issues with: - Tensor lifetime management using `from_blob` - Asynchronous operations during graph capture - Buffer restoration during graph replay ### Request Could you provide: 1. A complete example showing proper CUDA Graph integration with `low_latency_dispatch`? 2. Any limitations or constraints when using this approach? Thank you for your support!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#274
No description provided.