mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #756] CUDA error: no kernel image is available for execution on the device #549
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#549
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Liskarm on GitHub (Mar 1, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/756
Hello, I am encountering a RuntimeError:
CUDA error: no kernel image is available for execution on the devicewhile trying to run my script using the following command:python ./ktransformers/local_chat.py --model_path /workspace/ktransformers/kt0.2.2config --gguf_path /workspace/ktggufpath --prompt_file /workspace/ktpromptpath --cpu_infer 65 --max_new_tokens 1000My system configuration is as follows:
Intel Xeon Platinum 8171M (dual socket)
12-channel 1TB DDR4 memory
Here's the detailed traceback of the error:
Injecting model.layers.60 as default Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.q_a_layernorm as default Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.kv_a_layernorm as default Injecting model.layers.60.self_attn.kv_b_proj as default Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbeddingV3 Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate Injecting model.layers.60.mlp.shared_experts as default Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear Injecting model.layers.60.mlp.shared_experts.act_fn as default Injecting model.layers.60.input_layernorm as default Injecting model.layers.60.post_attention_layernorm as default Injecting model.norm as default Injecting lm_head as ktransformers.operators.linear . KTransformersLinear Traceback (most recent call last): File "/workspace/ktransformers/./ktransformers/local_chat.py", line 183, in <module> fire.Fire(local_chat) File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( File "/opt/conda/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/workspace/ktransformers/./ktransformers/local_chat.py", line 110, in local_chat optimize_and_load_gguf(model, optimize_config_path, gguf_path, config) File "/workspace/ktransformers/./ktransformers/optimize/optimize.py", line 130, in optimize_and_load_gguf load_weights(module.lm_head, gguf_loader, "lm_head.") File "/workspace/ktransformers/./ktransformers/util/utils.py", line 111, in load_weights module.load() File "/workspace/ktransformers/./ktransformers/operators/linear.py", line 522, in load self.generate_linear.load(w=w) File "/workspace/ktransformers/./ktransformers/operators/linear.py", line 279, in load w = self.load_weight(device=device) File "/workspace/ktransformers/./ktransformers/operators/linear.py", line 97, in load_weight tensors = self.load_multi(key, ["weight"], device=device) File "/workspace/ktransformers/./ktransformers/operators/linear.py", line 107, in load_multi tensors[k] = self.gguf_loader.load_gguf_tensor(key + "." + k, device=device) File "/workspace/ktransformers/./ktransformers/util/custom_gguf.py", line 376, in load_gguf_tensor values[blocks_begin : blocks_end] = cur_values RuntimeError: CUDA error: no kernel image is available for execution on the device Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.bash: syntax error near unexpected token
newline'I have already ensured that my cuda and pytorch installations allright. Could someone please help me resolve this issue? Any suggestions or solutions would be greatly appreciated.
@Liskarm commented on GitHub (Mar 1, 2025):
The operating system version I am using is Ubuntu 22.04 LTS
@qiyuxinlin commented on GitHub (Mar 2, 2025):
Can you see your currently available GPU devices via torch?
@Liskarm commented on GitHub (Mar 4, 2025):
yes,i will include the results I have reviewed below.
@qiyuxinlin commented on GitHub (Mar 4, 2025):
@Atream
@ErvinXie commented on GitHub (Dec 11, 2025):
This issue has been automatically closed due to inactivity for 60 days.
If you believe this issue is still relevant, please feel free to reopen it with additional information or context.
Thank you for your contribution to KTransformers!
此 issue 因超过 60 天无活动而被自动关闭。
如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。
感谢您对 KTransformers 的贡献！