[GH-ISSUE #942] [Bug] CUDA error: operation not permitted when stream is capturing #671

New issue

Closed

opened 2026-10-29 14:42:40 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:42:40 +00:00

Owner

Originally created by @sz-xuejin on GitHub (Mar 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/942

检查清单

1. 我已经搜索过相关问题，但未能获得预期的帮助
2. 该问题在最新版本中尚未修复
3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

ktransformers main分支代码，commit 32a91c78c1
Version: 0.2.3.post2+torch26fancy

修改ktransformers/optimize/optimize_rules下的Qwen2-57B-A14B-Instruct.yaml配置，将

- match:
    name: "^model\\.layers\\..*\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    # device: "cpu"   # which devices to load this module when initializing
    kwargs:
      prefill_device: "cuda"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op:  "KExpertsCPU"
      out_device: "cuda"
  recursive: False # don't recursively inject submodules of this module

改为

- match:
   name: "^model\\.layers\\.([0-1])\\.mlp\\.experts$"
  replace:
   class: ktransformers.operators.experts.KTransformersExperts  
   kwargs:
     prefill_device: "cuda:0"
     prefill_op: "KExpertsTorch"
     generate_device: "cuda:0"
     generate_op:  "KExpertsTorch"
     out_device: "cuda:0"
  recursive: False

- match:
   name: "^model\\.layers\\.([2-9]|[1][0-9]|[2][0-7])\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     
    kwargs:
     prefill_device: "cuda:0"
     prefill_op: "KExpertsTorch"
     generate_device: "cpu"
     generate_op:  "KExpertsCPU"
     out_device: "cuda:0"
  recursive: False

尝试将部分专家加载到GPU上，运行local_chat加载模型阶段正常，但输入提问后运行报错，错误如下：
Traceback (most recent call last):
File "ktransformers/ktransformers/ktransformers/util/cuda_graph_runner.py", line 41, in capture
logits=model(inputs_embeds=inputs_embeds,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 1414, in forward
outputs = self.model(
^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/operators/models.py", line 363, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 936, in forward
hidden_states = self.mlp(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 655, in forward
self.moe_kexperts(
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 675, in moe_kexperts
outs = self.experts(x, topk_ids, topk_weight)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 595, in forward
return self.generate_experts.forward(input_tensor, expert_ids, weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 515, in forward
idx, top_x = torch.where(expert_mask[expert_idx])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: operation not permitted when stream is capturing
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "ktransformers/ktransformers/ktransformers/local_chat.py", line 185, in
fire.Fire(local_chat)
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
generated = prefill_and_generate(
^^^^^^^^^^^^^^^^^^^^^
File "ktransformers/ktransformers/ktransformers/util/utils.py", line 247, in prefill_and_generate
cuda_graph_runner.capture(model, next_token.unsqueeze(0), position_ids, cache_position, past_key_values, torch_device, return_dict=False, use_cache=True)
File "ktransformers/ktransformers/ktransformers/util/cuda_graph_runner.py", line 40, in capture
with torch.cuda.graph(self.graph, stream = capture_stream):
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/cuda/graphs.py", line 186, in exit
self.cuda_graph.capture_end()
File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/cuda/graphs.py", line 84, in capture_end
super().capture_end()
RuntimeError: CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
请问是KT本身不支持这样做还是没正确配置Qwen2-57B-A14B-Instruct.yaml

复现步骤

运行命令：
python -m ktransformers.local_chat --model_path Qwen/Qwen2-57B-A14B --gguf_path /ktransformers/Qwen2-57B-A14B --optimize_config_path ./ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml --cpu_infer 124 --max_new_tokens 1000
使用模型：Qwen2-57B-A14B

环境信息

操作系统：Linux Ubuntu22.04
CPU: Intel(R) Xeon(R) Gold 6430，2个物理cpu，均为32核64线程，总128线程，2个numa节点每个节点512GB内存，总内存为1TB
GPU: Nvidia 4090 D，显存24GB

Originally created by @sz-xuejin on GitHub (Mar 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/942 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [ ] 2. 该问题在最新版本中尚未修复 - [ ] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [ ] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [ ] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 ktransformers main分支代码，commit 32a91c78c179e37fa34302505bb287a3cc64db0c Version: 0.2.3.post2+torch26fancy 修改ktransformers/optimize/optimize_rules下的Qwen2-57B-A14B-Instruct.yaml配置，将 ``` - match: name: "^model\\.layers\\..*\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism # device: "cpu" # which devices to load this module when initializing kwargs: prefill_device: "cuda" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda" recursive: False # don't recursively inject submodules of this module ``` 改为 ``` - match: name: "^model\\.layers\\.([0-1])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts kwargs: prefill_device: "cuda:0" prefill_op: "KExpertsTorch" generate_device: "cuda:0" generate_op: "KExpertsTorch" out_device: "cuda:0" recursive: False - match: name: "^model\\.layers\\.([2-9]|[1][0-9]|[2][0-7])\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts kwargs: prefill_device: "cuda:0" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda:0" recursive: False ``` 尝试将部分专家加载到GPU上，运行local_chat加载模型阶段正常，但输入提问后运行报错，错误如下： Traceback (most recent call last): File "ktransformers/ktransformers/ktransformers/util/cuda_graph_runner.py", line 41, in capture logits=model(inputs_embeds=inputs_embeds, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 1414, in forward outputs = self.model( ^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/operators/models.py", line 363, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/models/modeling_qwen2_moe.py", line 936, in forward hidden_states = self.mlp(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 655, in forward self.moe_kexperts( File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 675, in moe_kexperts outs = self.experts(x, topk_ids, topk_weight) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 595, in forward return self.generate_experts.forward(input_tensor, expert_ids, weights) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/operators/experts.py", line 515, in forward idx, top_x = torch.where(expert_mask[expert_idx]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: operation not permitted when stream is capturing CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "ktransformers/ktransformers/ktransformers/local_chat.py", line 185, in <module> fire.Fire(local_chat) File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/local_chat.py", line 179, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "ktransformers/ktransformers/ktransformers/util/utils.py", line 247, in prefill_and_generate cuda_graph_runner.capture(model, next_token.unsqueeze(0), position_ids, cache_position, past_key_values, torch_device, return_dict=False, use_cache=True) File "ktransformers/ktransformers/ktransformers/util/cuda_graph_runner.py", line 40, in capture with torch.cuda.graph(self.graph, stream = capture_stream): File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/cuda/graphs.py", line 186, in __exit__ self.cuda_graph.capture_end() File "/opt/conda/envs/ktransformers/lib/python3.11/site-packages/torch/cuda/graphs.py", line 84, in capture_end super().capture_end() RuntimeError: CUDA error: operation failed due to a previous error during capture CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 请问是KT本身不支持这样做还是没正确配置Qwen2-57B-A14B-Instruct.yaml ### 复现步骤运行命令： python -m ktransformers.local_chat --model_path Qwen/Qwen2-57B-A14B --gguf_path /ktransformers/Qwen2-57B-A14B --optimize_config_path ./ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml --cpu_infer 124 --max_new_tokens 1000 使用模型：Qwen2-57B-A14B ### 环境信息操作系统：Linux Ubuntu22.04 CPU: Intel(R) Xeon(R) Gold 6430，2个物理cpu，均为32核64线程，总128线程，2个numa节点每个节点512GB内存，总内存为1TB GPU: Nvidia 4090 D，显存24GB

bsakaguchi closed this issue

2026-10-29 14:42:40 +00:00

bsakaguchi commented

2026-10-29 14:42:41 +00:00

Author

Owner

@sz-xuejin commented on GitHub (Mar 21, 2025):

把util/utils.py的if use_cuda_graph and ( (warm_uped == True and int(i) == 1) or (warm_uped == False and int(i) == 2) ): 改为 if use_cuda_graph is True and ( (warm_uped == True and int(i) == 1) or (warm_uped == False and int(i) == 2) ):且不启用cuda graph，命令行改为
python -m ktransformers.local_chat --model_path Qwen/Qwen2-57B-A14B --gguf_path /ktransformers/Qwen2-57B-A14B --optimize_config_path ./ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml --cpu_infer 124 --max_new_tokens 1000 --use_cuda_graph false即可运行，但decode的性能没有提升。

@sz-xuejin commented on GitHub (Mar 21, 2025): 把util/utils.py的`if use_cuda_graph and ( (warm_uped == True and int(i) == 1) or (warm_uped == False and int(i) == 2) ):` 改为 `if use_cuda_graph is True and ( (warm_uped == True and int(i) == 1) or (warm_uped == False and int(i) == 2) ):`且不启用cuda graph，命令行改为 `python -m ktransformers.local_chat --model_path Qwen/Qwen2-57B-A14B --gguf_path /ktransformers/Qwen2-57B-A14B --optimize_config_path ./ktransformers/optimize/optimize_rules/Qwen2-57B-A14B-Instruct.yaml --cpu_infer 124 --max_new_tokens 1000 --use_cuda_graph false`即可运行，但decode的性能没有提升。