deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #656] [DeepEPv2] test_agrs.py failed due to CUDA_ERROR_NOT_SUPPORTED, operation not supported #386

New issue
Closed
opened 2026-10-26 09:59:51 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-26 09:59:51 +00:00
Owner
Copy link

Originally created by @elevenxiang on GitHub (Jun 1, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/656

Hi,

we tested the test_agrs.py and found it will be failed on H20:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 87, in _wrap
fn(i, *args)
File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/workspace/DeepEP/tests/elastic/test_agrs.py", line 125, in test
buffer.destroy_agrs_session()
File "/usr/local/lib/python3.12/dist-packages/deep_ep/buffers/elastic.py", line 503, in destroy_agrs_session
self.runtime.destroy_agrs_session()
RuntimeError: CUDA driver exception (/workspace/DeepEP/csrc/kernels/backend/cuda_driver.cu:51): 801 (CUDA_ERROR_NOT_SUPPORTED, operation not supported)

github.com/deepseek-ai/DeepEP@d4f41e4e93/csrc/kernels/backend/cuda_driver.cu (L51)

Originally created by @elevenxiang on GitHub (Jun 1, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/656 Hi, we tested the test_agrs.py and found it will be failed on H20: -- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 87, in _wrap fn(i, *args) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/workspace/DeepEP/tests/elastic/test_agrs.py", line 125, in test buffer.destroy_agrs_session() File "/usr/local/lib/python3.12/dist-packages/deep_ep/buffers/elastic.py", line 503, in destroy_agrs_session self.runtime.destroy_agrs_session() RuntimeError: CUDA driver exception (/workspace/DeepEP/csrc/kernels/backend/cuda_driver.cu:51): 801 (CUDA_ERROR_NOT_SUPPORTED, operation not supported) https://github.com/deepseek-ai/DeepEP/blob/d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb/csrc/kernels/backend/cuda_driver.cu#L51
bsakaguchi commented 2026-10-26 09:59:53 +00:00
Author
Owner
Copy link

@MARD1NO commented on GitHub (Jul 13, 2026):

Hi,

we tested the test_agrs.py and found it will be failed on H20:

-- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 87, in _wrap fn(i, *args) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/workspace/DeepEP/tests/elastic/test_agrs.py", line 125, in test buffer.destroy_agrs_session() File "/usr/local/lib/python3.12/dist-packages/deep_ep/buffers/elastic.py", line 503, in destroy_agrs_session self.runtime.destroy_agrs_session() RuntimeError: CUDA driver exception (/workspace/DeepEP/csrc/kernels/backend/cuda_driver.cu:51): 801 (CUDA_ERROR_NOT_SUPPORTED, operation not supported)

DeepEP/csrc/kernels/backend/cuda_driver.cu

Line 51 in d4f41e4

CUDA_DRIVER_CHECK(lazy_cuStreamBatchMemOp(stream, ops.size(), ops.data(), 0));

What is your cuda version?

<!-- gh-comment-id:4955175475 --> @MARD1NO commented on GitHub (Jul 13, 2026): > Hi, > > we tested the test_agrs.py and found it will be failed on H20: > > -- Process 0 terminated with the following error: Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/spawn.py", line 87, in _wrap fn(i, *args) File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/workspace/DeepEP/tests/elastic/test_agrs.py", line 125, in test buffer.destroy_agrs_session() File "/usr/local/lib/python3.12/dist-packages/deep_ep/buffers/elastic.py", line 503, in destroy_agrs_session self.runtime.destroy_agrs_session() RuntimeError: CUDA driver exception (/workspace/DeepEP/csrc/kernels/backend/cuda_driver.cu:51): 801 (CUDA_ERROR_NOT_SUPPORTED, operation not supported) > > [DeepEP/csrc/kernels/backend/cuda_driver.cu](https://github.com/deepseek-ai/DeepEP/blob/d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb/csrc/kernels/backend/cuda_driver.cu#L51) > > Line 51 in [d4f41e4](/deepseek-ai/DeepEP/commit/d4f41e4e93602a15e95f55f6ee8df8f1aaa0e4bb) > > CUDA_DRIVER_CHECK(lazy_cuStreamBatchMemOp(stream, ops.size(), ops.data(), 0)); What is your cuda version?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#386
No description provided.