[GH-ISSUE #579] 在运行main.py时报错：CUDA error: no kernel image is available for execution on the deviceCUDA kernel errors might be asynchronously reported at some other ApI call, #426

New issue

Closed

opened 2026-10-29 14:39:28 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:39:28 +00:00

Owner

Originally created by @winni0 on GitHub (Feb 21, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/579

在A100上的容器内推理时，执行命令：python3 /deepseek/ktransformers/ktransformers/server/main.py --model_path /deepseek/DeepSeek-V3 --gguf_path /deepseek/DeepSeek-V3-Q4_K_M/ --cpu_infer 65 --max_new_tokens 1000 --host 0.0.0.0 --port 10002
容器内环境：cuda11.8，torch2.4.1，cuda最高可兼容版本为12.2，容器外cuda没装驱动。
报错如下：self.generate linear.load(w=w)File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/linear.py", line 215, in loadw ref, marlin q w, marlin s, g idx, sort indices, marlin quantize(File "/opt/conda/lib/python3. 10/site-packages/ktrans formers/ktransformers_ext/operators/custom marlin/quantize/utils/w ref, q_w,s,g idx, rand perm = quantize _weights(w, num bits, group size,File "/opt/conda/lib/python3.10/site-packages/ktrans formers/ktransformers ext/operators/custom marlin/quantize/utils/= w.reshape((group size,-1)RuntimeError: CUDA error: no kernel image is available for execution on the deviceCUDA kernel errors might be asynchronously reported at some other ApI call, so the stacktrace below might be incorrect.For debugging consider passing CUDA LAUNCH BLOCKING=1.Compile with TORCH USE CUDA DSA` to enable device-side assertions.
在H100上使用同样的容器，进行推理时没有报错，请各位大佬解答

Originally created by @winni0 on GitHub (Feb 21, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/579 在A100上的容器内推理时，执行命令：python3 /deepseek/ktransformers/ktransformers/server/main.py --model_path /deepseek/DeepSeek-V3 --gguf_path /deepseek/DeepSeek-V3-Q4_K_M/ --cpu_infer 65 --max_new_tokens 1000 --host 0.0.0.0 --port 10002 容器内环境：cuda11.8，torch2.4.1，cuda最高可兼容版本为12.2，容器外cuda没装驱动。报错如下：self.generate linear.load(w=w)File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/linear.py", line 215, in loadw ref, marlin q w, marlin s, g idx, sort indices, marlin quantize(File "/opt/conda/lib/python3. 10/site-packages/ktrans formers/ktransformers_ext/operators/custom marlin/quantize/utils/w ref, q_w,s,g idx, rand perm = quantize _weights(w, num bits, group size,File "/opt/conda/lib/python3.10/site-packages/ktrans formers/ktransformers ext/operators/custom marlin/quantize/utils/= w.reshape((group size,-1)RuntimeError: CUDA error: no kernel image is available for execution on the deviceCUDA kernel errors might be asynchronously reported at some other ApI call, so the stacktrace below might be incorrect.For debugging consider passing CUDA LAUNCH BLOCKING=1.Compile with TORCH USE CUDA DSA` to enable device-side assertions. 在H100上使用同样的容器，进行推理时没有报错，请各位大佬解答

bsakaguchi

2026-10-29 14:39:28 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:39:29 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！