[GH-ISSUE #44] CUDA error: No kernel image is available for execution on the device #23

New issue

Closed

opened 2026-10-29 14:32:44 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:32:44 +00:00

Owner

Originally created by @Forsworns on GitHub (Aug 20, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/44

I tried the DeepSeek-V2-Lite in your container image approachingai/ktransformers:0.1.1, which is deployed on an Nvidia T4 GPU with driver 535.161.8. And I got following errors, is T4 not sufficient to run it, or should I change my driver version?

Originally created by @Forsworns on GitHub (Aug 20, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/44 I tried the DeepSeek-V2-Lite in your container image `approachingai/ktransformers:0.1.1`, which is deployed on an Nvidia T4 GPU with driver 535.161.8. And I got following errors, is T4 not sufficient to run it, or should I change my driver version? ![image](https://github.com/user-attachments/assets/623c2d8f-f5a8-44fd-8612-ecde2024f4df)

bsakaguchi closed this issue

2026-10-29 14:32:45 +00:00

bsakaguchi commented

2026-10-29 14:32:45 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 20, 2024):

Hi, thank u for your interest.
For your problem, Your GPU Compute Capability is 7.5, which our precompiled docker only support 8.0 and higher. If you want to run ktransformers, you may follow the readme Or you can download source code and compile: section.
One more thing is that our default linear operator(marlin) only support Nvidia's Ampere and higher architecture. For T4, which is Turing, you may need modify /ktransformer/optimize/optimize_rules/DeepSeek-V2-Chat.yaml and change linear backend to KLinearTorch .

The modified linear replace rule will like:

- match:
    name: "^model\\.layers\\.(?!.*self_attn).*$"  # regular expression 
    class: torch.nn.Linear  # only match modules matching name and class simultaneously
  replace:
    class: ktransformers.operators.linear.KTransformersLinear  # optimized Kernel on quantized data types
    kwargs:
      generate_device: "cuda"
      generate_op: "KLinearTorch"

@Azure-Tang commented on GitHub (Aug 20, 2024): Hi, thank u for your interest. For your problem, Your GPU Compute Capability is 7.5, which our precompiled docker only support 8.0 and higher. If you want to run ktransformers, you may follow the readme `Or you can download source code and compile:` section. One more thing is that our default linear operator(marlin) only support Nvidia's `Ampere` and higher architecture. For T4, which is `Turing`, you may need modify `/ktransformer/optimize/optimize_rules/DeepSeek-V2-Chat.yaml` and change linear backend to `KLinearTorch` . The modified linear replace rule will like: ```yaml - match: name: "^model\\.layers\\.(?!.*self_attn).*$" # regular expression class: torch.nn.Linear # only match modules matching name and class simultaneously replace: class: ktransformers.operators.linear.KTransformersLinear # optimized Kernel on quantized data types kwargs: generate_device: "cuda" generate_op: "KLinearTorch" ```

bsakaguchi commented

2026-10-29 14:32:46 +00:00

Author

Owner

@Forsworns commented on GitHub (Aug 22, 2024):

Thanks! I have run the original container image successfully with A10 cards :)

@Forsworns commented on GitHub (Aug 22, 2024): Thanks! I have run the original container image successfully with A10 cards :)

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871

bsakaguchi referenced this issue

2026-10-29 14:45:39 +00:00

[GH-ISSUE #1288] [Bug] #876