[GH-ISSUE #10] [model support] Requesting support for Gemma 2 #6

New issue

Closed

opened 2026-10-29 14:32:34 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:32:34 +00:00

Owner

Originally created by @sand-bit on GitHub (Jul 28, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/10

Requesting support for Gemma 2 27B/9B.

Sliding window attention, used every other layer
Logit soft capping (aka soft capping, so no flash attention support right now)
Possible tokenizer bugs in transformer implementation https://github.com/ggerganov/llama.cpp/issues/8240#issuecomment-2212494531 (See that issue for other discussions on tokenizer bugs. Maybe okay in Google's official Keras implementation?)

Originally created by @sand-bit on GitHub (Jul 28, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/10 Requesting support for Gemma 2 27B/9B. 1. Sliding window attention, used every other layer 2. Logit soft capping (aka soft capping, so no flash attention support right now) 3. Possible tokenizer bugs in transformer implementation https://github.com/ggerganov/llama.cpp/issues/8240#issuecomment-2212494531 (See that issue for other discussions on tokenizer bugs. Maybe okay in Google's official Keras implementation?)

bsakaguchi closed this issue

2026-10-29 14:32:34 +00:00

bsakaguchi commented

2026-10-29 14:32:35 +00:00

Author

Owner

@james0zan commented on GitHub (Jul 29, 2024):

As it is a dense model, we may currently not be able to run Gemma 2 faster than its official support. Thus, Gemma's support may not be available in the coming days.

@james0zan commented on GitHub (Jul 29, 2024): As it is a dense model, we may currently not be able to run Gemma 2 faster than its official support. Thus, Gemma's support may not be available in the coming days.

bsakaguchi referenced this issue

2026-10-29 14:37:20 +00:00

[GH-ISSUE #420] deepseek r1 using 4090 nvidia running, meet segmentation fault(core dumped) problem #288

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859