[GH-ISSUE #103] python test_low_latency.py blocked in H20 #96

Open
opened 2026-10-26 09:54:11 +00:00 by bsakaguchi · 1 comment
Owner

Originally created by @cll24 on GitHub (Mar 31, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/103

By analyzing the code, we find the code is blocked in the instantiation of buffer

Image

Image

Image

We got the following log when setting "NVSHMEM_DEBUG=INFO"

Image

What does the Error "cudaErrorInvalidValue 1 cudaErrorInvalidSymbol 13 cudaErrorInvalidMemcpyDirection 21 cudaErrorNoKernelImageForDevice 209" mean? Is it the root cause of process blocking.

Here is the configuration of NVSHMEM
Image

Originally created by @cll24 on GitHub (Mar 31, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/103 By analyzing the code, we find the code is blocked in the instantiation of buffer ![Image](https://github.com/user-attachments/assets/126aa226-c40f-483c-9f15-1eac4c96aead) ![Image](https://github.com/user-attachments/assets/ba99c771-ebd9-4474-ab1f-d166b3057df1) ![Image](https://github.com/user-attachments/assets/49ef44f1-e71a-4a6c-8ae4-83574dab9569) We got the following log when setting "NVSHMEM_DEBUG=INFO" ![Image](https://github.com/user-attachments/assets/5972f4ea-4173-41ba-b98e-a23e78fc1ad2) What does the Error "cudaErrorInvalidValue 1 cudaErrorInvalidSymbol 13 cudaErrorInvalidMemcpyDirection 21 cudaErrorNoKernelImageForDevice 209" mean? Is it the root cause of process blocking. Here is the configuration of NVSHMEM ![Image](https://github.com/user-attachments/assets/a9745ff6-de58-47e8-afbf-c8604a5d1f37)
Author
Owner

@sphish commented on GitHub (Apr 1, 2025):

The log line cudaErrorInvalidValue 1 cudaErrorInvalidSymbol 13 cudaErrorInvalidMemcpyDirection 21 cudaErrorNoKernelImageForDevice 209 is a normal log in the NVSHMEM.

I suggest you use gdb to print the backtrace when the test processes hang.

<!-- gh-comment-id:2767825500 --> @sphish commented on GitHub (Apr 1, 2025): The log line `cudaErrorInvalidValue 1 cudaErrorInvalidSymbol 13 cudaErrorInvalidMemcpyDirection 21 cudaErrorNoKernelImageForDevice 209` is a normal log in the NVSHMEM. I suggest you use gdb to print the backtrace when the test processes hang.
Sign in to join this conversation.
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#96
No description provided.