[GH-ISSUE #13] About 1M ctx models #8

New issue

Closed

opened 2026-10-29 14:32:36 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:32:36 +00:00

Owner

Originally created by @choyakawa on GitHub (Jul 29, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/13

I saw elsewhere that you mentioned optimization for 1M ctx models (glm-4-9b-chat-1m/internlm2_5-7b-chat-1m) is on the roadmap and will be released in a future version. And I'm interested to know what kind of improvements you're anticipating achieving.

Originally created by @choyakawa on GitHub (Jul 29, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/13 I saw elsewhere that you mentioned optimization for 1M ctx models (glm-4-9b-chat-1m/internlm2_5-7b-chat-1m) is on the roadmap and will be released in a future version. And I'm interested to know what kind of improvements you're anticipating achieving.

bsakaguchi closed this issue

2026-10-29 14:32:36 +00:00

bsakaguchi commented

2026-10-29 14:32:37 +00:00

Author

Owner

@james0zan commented on GitHub (Jul 30, 2024):

We intend to take advantage from the sparsity of attention operator. There are many great work study on this topic and we are evaluating and integrating some of them into the KTransformers framework.

@james0zan commented on GitHub (Jul 30, 2024): We intend to take advantage from the sparsity of attention operator. There are many great work study on this topic and we are evaluating and integrating some of them into the KTransformers framework.

bsakaguchi referenced this issue

2026-10-29 14:37:20 +00:00

[GH-ISSUE #420] deepseek r1 using 4090 nvidia running, meet segmentation fault(core dumped) problem #288

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859