[GH-ISSUE #374] 使用volta和turing架构显卡运行的可能办法 #248

New issue

Closed

opened 2026-10-29 14:36:33 +00:00 by bsakaguchi · 23 comments

bsakaguchi commented 2026-10-29 14:36:33 +00:00

Owner

Copy link

Originally created by @20000419 on GitHub (Feb 16, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/374

首先，git下载最新的源码。
然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch；
然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。
然后，安装flash-attn，不会用到，但是不安装会报错。
然后，修改local_chat.py，将里面所有的flash_attention_2换成eager；
然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整；
然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制）
最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

Originally created by @20000419 on GitHub (Feb 16, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/374 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

bsakaguchi closed this issue 2026-10-29 14:36:33 +00:00

bsakaguchi commented 2026-10-29 14:36:34 +00:00

Author

Owner

Copy link

@20000419 commented on GitHub (Feb 16, 2025):

欢迎大家补充更多细节。

<!-- gh-comment-id:2661438667 --> @20000419 commented on GitHub (Feb 16, 2025): 欢迎大家补充更多细节。

bsakaguchi commented 2026-10-29 14:36:34 +00:00

Author

Owner

Copy link

@ShinoharaHare commented on GitHub (Feb 16, 2025):

在 Linux 上 KDeepseekV2Attention 還是會使用 FA2
github.com/kvcache-ai/ktransformers@65d73ea3f8/ktransformers/operators/attention.py (L310-L316)
要強制使用 forward_windows 才行
github.com/kvcache-ai/ktransformers@65d73ea3f8/ktransformers/operators/attention.py (L393-L425)

<!-- gh-comment-id:2661559961 --> @ShinoharaHare commented on GitHub (Feb 16, 2025): 在 Linux 上 `KDeepseekV2Attention` 還是會使用 FA2 https://github.com/kvcache-ai/ktransformers/blob/65d73ea3f862bd33712db2c5ce81e745c24265b1/ktransformers/operators/attention.py#L310-L316 要強制使用 `forward_windows` 才行 https://github.com/kvcache-ai/ktransformers/blob/65d73ea3f862bd33712db2c5ce81e745c24265b1/ktransformers/operators/attention.py#L393-L425

bsakaguchi commented 2026-10-29 14:36:34 +00:00

Author

Owner

Copy link

@20000419 commented on GitHub (Feb 17, 2025):

在 Linux 上 KDeepseekV2Attention 還是會使用 FA2

ktransformers/ktransformers/operators/attention.py

Lines 310 to 316 in 65d73ea

attn_output = flash_attn_func(
query_states,
key_states,
value_states_padded,
softmax_scale=self.softmax_scale,
causal=True,
)

要強制使用 forward_windows 才行
ktransformers/ktransformers/operators/attention.py

Lines 393 to 425 in 65d73ea

def forward(
self,
hidden_states: torch.Tensor,
attention_mask: Optional[torch.Tensor] = None,
position_ids: Optional[torch.LongTensor] = None,
past_key_value: Optional[Cache] = None,
output_attentions: bool = False,
use_cache: bool = False,
cache_position: Optional[torch.LongTensor] = None,
**kwargs,
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
if os.name == 'nt':
return self.forward_windows(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)
else:
return self.forward_linux(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)

请问您现在跑通了吗？在解决了这个报错之后。

<!-- gh-comment-id:2661772170 --> @20000419 commented on GitHub (Feb 17, 2025): > 在 Linux 上 `KDeepseekV2Attention` 還是會使用 FA2 > > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/65d73ea3f862bd33712db2c5ce81e745c24265b1/ktransformers/operators/attention.py#L310-L316) > > Lines 310 to 316 in [65d73ea](/kvcache-ai/ktransformers/commit/65d73ea3f862bd33712db2c5ce81e745c24265b1) > > attn_output = flash_attn_func( > query_states, > key_states, > value_states_padded, > softmax_scale=self.softmax_scale, > causal=True, > ) > > 要強制使用 `forward_windows` 才行 > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/65d73ea3f862bd33712db2c5ce81e745c24265b1/ktransformers/operators/attention.py#L393-L425) > > Lines 393 to 425 in [65d73ea](/kvcache-ai/ktransformers/commit/65d73ea3f862bd33712db2c5ce81e745c24265b1) > > def forward( > self, > hidden_states: torch.Tensor, > attention_mask: Optional[torch.Tensor] = None, > position_ids: Optional[torch.LongTensor] = None, > past_key_value: Optional[Cache] = None, > output_attentions: bool = False, > use_cache: bool = False, > cache_position: Optional[torch.LongTensor] = None, > **kwargs, > ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: > if os.name == 'nt': > return self.forward_windows( > hidden_states, > attention_mask, > position_ids, > past_key_value, > output_attentions, > use_cache, > cache_position, > **kwargs, > ) > else: > return self.forward_linux( > hidden_states, > attention_mask, > position_ids, > past_key_value, > output_attentions, > use_cache, > cache_position, > **kwargs, > ) 请问您现在跑通了吗？在解决了这个报错之后。

bsakaguchi commented 2026-10-29 14:36:34 +00:00

Author

Owner

Copy link

@ShinoharaHare commented on GitHub (Feb 17, 2025):

是的，有成功運行

<!-- gh-comment-id:2661919943 --> @ShinoharaHare commented on GitHub (Feb 17, 2025): 是的，有成功運行

bsakaguchi commented 2026-10-29 14:36:34 +00:00

Author

Owner

Copy link

@1986smalltiger commented on GitHub (Feb 17, 2025):

这个是怎么修改的？还有指导？ @ShinoharaHare

<!-- gh-comment-id:2662430442 --> @1986smalltiger commented on GitHub (Feb 17, 2025): 这个是怎么修改的？还有指导？ @ShinoharaHare

bsakaguchi commented 2026-10-29 14:36:35 +00:00

Author

Owner

Copy link

@20000419 commented on GitHub (Feb 18, 2025):

这个是怎么修改的？还有指导？ @ShinoharaHare
修改 if os.name == 'nt':
return self.forward_windows(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)
else:
return self.forward_linux(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)
为
if os.name == 'nt':
return self.forward_windows(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)
else:
return self.forward_windows(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)
就可以，这个代码在ktransformers/ktransformers/operators/attention.py中

<!-- gh-comment-id:2664331762 --> @20000419 commented on GitHub (Feb 18, 2025): > 这个是怎么修改的？还有指导？ [@ShinoharaHare](https://github.com/ShinoharaHare) 修改 if os.name == 'nt': return self.forward_windows( hidden_states, attention_mask, position_ids, past_key_value, output_attentions, use_cache, cache_position, **kwargs, ) else: return self.forward_linux( hidden_states, attention_mask, position_ids, past_key_value, output_attentions, use_cache, cache_position, **kwargs, ) 为 if os.name == 'nt': return self.forward_windows( hidden_states, attention_mask, position_ids, past_key_value, output_attentions, use_cache, cache_position, **kwargs, ) else: return self.forward_windows( hidden_states, attention_mask, position_ids, past_key_value, output_attentions, use_cache, cache_position, **kwargs, ) 就可以，这个代码在[ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/65d73ea3f862bd33712db2c5ce81e745c24265b1/ktransformers/operators/attention.py#L393-L425)中

bsakaguchi commented 2026-10-29 14:36:35 +00:00

Author

Owner

Copy link

@tonyeatsm commented on GitHub (Feb 18, 2025):

是的，有成功運行

感谢，你的修改方案+ “20000419” 的方案 可以在v100 的四张卡上运行， 但是太慢了48 Core的CPU已经打满了，但是4张GPU 利用率跳跃 只利用15%
Chat: nihao
Nihao! (你好!) That means "hello" in Chinese. How can I assist you today? 😊
prompt eval count: 5 token(s)
prompt eval duration: 5.843478202819824s
prompt eval rate: 0.8556547704049283 tokens/s
eval count: 25 token(s)
eval duration: 16.465613842010498s
eval rate: 1.5183156996075544 tokens/s

<!-- gh-comment-id:2664656606 --> @tonyeatsm commented on GitHub (Feb 18, 2025): > 是的，有成功運行 感谢，你的修改方案+ “[20000419](https://github.com/20000419)” 的方案 可以在v100 的四张卡上运行， 但是太慢了48 Core的CPU已经打满了，但是4张GPU 利用率跳跃 只利用15% Chat: nihao Nihao! (你好!) That means "hello" in Chinese. How can I assist you today? 😊 prompt eval count: 5 token(s) prompt eval duration: 5.843478202819824s prompt eval rate: 0.8556547704049283 tokens/s eval count: 25 token(s) eval duration: 16.465613842010498s eval rate: 1.5183156996075544 tokens/s

bsakaguchi commented 2026-10-29 14:36:35 +00:00

Author

Owner

Copy link

@bluezd commented on GitHub (Feb 18, 2025):

@tonyeatsm 能分享下具体是怎么改的吗？
我修改后报如下错误：

Chat: hi
Traceback (most recent call last):
  File "/root/ktransformers/ktransformers/operators/base_operator.py", line 42, in __getattr__
    return object.__getattribute__(self, name) # if this attr belongs to BaseInjectedModule
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'KDeepseekV2Attention' object has no attribute 'forward_linux'. Did you mean: 'forward_chunck'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/ktransformers/ktransformers/operators/base_operator.py", line 47, in __getattr__
    return nn.Module.__getattr__(self, "orig_module").__getattr__(name) # if this attr belongs to orig_module
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__
    raise AttributeError(
AttributeError: 'DeepseekV3Attention' object has no attribute 'forward_linux'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/ktransformers/ktransformers/local_chat.py", line 187, in <module>
    fire.Fire(local_chat)
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/local_chat.py", line 181, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/util/utils.py", line 156, in prefill_and_generate
    logits = model(
             ^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/operators/models.py", line 722, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        if os.name == 'nt':
            return self.forward_windows(
                hidden_states,
                attention_mask,
                position_ids,
                past_key_value,
                output_attentions,
                use_cache,
                cache_position,
                **kwargs,
            )
        else:
            return self.forward_linux(
                hidden_states,
                attention_mask,
                position_ids,
                past_key_value,
                output_attentions,
                use_cache,
                cache_position,
                **kwargs,
            )

class KLlamaAttention(BaseInjectedModule):
    """Multi-headed attention from 'Attention Is All You Need' paper"""

    def __init__(self,
                 key: str,
                 gguf_loader : GGUFLoader,
                 config: PretrainedConfig,
                 orig_module: nn.Module,
                 prefill_device: str = "cuda",
                 generate_device: str = "cuda",
                 **kwargs):
        BaseInjectedModule.__init__(self, key, gguf_loader, config, orig_module, prefill_device, generate_device, **kwargs)
        self.orig_module.__init__(orig_module.config,
            orig_module.layer_idx)
    def apply_rotary_pos_emb(self, q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
        """Applies Rotary Position Embedding to the query and key tensors.

        Args:
            q (`torch.Tensor`): The query tensor.
            k (`torch.Tensor`): The key tensor.
"ktransformers/operators/attention.py" 710L, 34335B                                                                                                                 581,1         83%

    def __init__(self,
                 key: str,
                 gguf_loader : GGUFLoader,
                 config: PretrainedConfig,
                 orig_module: nn.Module,
                 prefill_device: str = "cuda",
                 generate_device: str = "cuda",
                 **kwargs):
        BaseInjectedModule.__init__(self, key, gguf_loader, config, orig_module, prefill_device, generate_device, **kwargs)
        self.orig_module.__init__(orig_module.config,
            orig_module.layer_idx)
    def apply_rotary_pos_emb(self, q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
        """Applies Rotary Position Embedding to the query and key tensors.

        Args:
            q (`torch.Tensor`): The query tensor.
            k (`torch.Tensor`): The key tensor.
            cos (`torch.Tensor`): The cosine part of the rotary embedding.
            sin (`torch.Tensor`): The sine part of the rotary embedding.
            position_ids (`torch.Tensor`, *optional*):
                Deprecated and unused.
            unsqueeze_dim (`int`, *optional*, defaults to 1):
    def __init__(self,
                 key: str,
                 gguf_loader : GGUFLoader,
                 config: PretrainedConfig,
                 orig_module: nn.Module,
                 prefill_device: str = "cuda",
                 generate_device: str = "cuda",
                 **kwargs):
        BaseInjectedModule.__init__(self, key, gguf_loader, config, orig_module, prefill_device, generate_device, **kwargs)
        self.orig_module.__init__(orig_module.config,
            orig_module.layer_idx)
    def apply_rotary_pos_emb(self, q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
        """Applies Rotary Position Embedding to the query and key tensors.

        Args:
            q (`torch.Tensor`): The query tensor.
            k (`torch.Tensor`): The key tensor.
            cos (`torch.Tensor`): The cosine part of the rotary embedding.
            sin (`torch.Tensor`): The sine part of the rotary embedding.
            position_ids (`torch.Tensor`, *optional*):
                Deprecated and unused.
            unsqueeze_dim (`int`, *optional*, defaults to 1):
                The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
                sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
                that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
                k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
                cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
                the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
        Returns:
            `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
        """
        cos = cos.unsqueeze(unsqueeze_dim)
        sin = sin.unsqueeze(unsqueeze_dim)
        q_embed = (q * cos) + (rotate_half(q) * sin)
        k_embed = (k * cos) + (rotate_half(k) * sin)
        return q_embed, k_embed

    def forward(
        self,
        hidden_states: torch.Tensor,
        attention_mask: Optional[torch.Tensor] = None,
        position_ids: Optional[torch.LongTensor] = None,
        past_key_value: Optional[Cache] = None,
search hit BOTTOM, continuing at TOP
                self.attn_mask[:, :, :, :cur_idx] = 0
                chunk_mask = torch.narrow(self.attn_mask, 2, 0, min(self.chunck_size, q_len-cur_idx))

            cur_output, _, _ = self.forward_chunck(
                            hidden_states[:, cur_idx:min(cur_idx + self.chunck_size, q_len), ...],
                            chunk_mask,
                            position_ids[:, cur_idx:min(cur_idx + self.chunck_size, q_len)],
                            past_key_value,
                            output_attentions,
                            use_cache,
                            cache_position[cur_idx:min(cur_idx + self.chunck_size, q_len)],
                            **kwargs
                        )
  File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/operators/attention.py", line 572, in forward
    return self.forward_linux(
           ^^^^^^^^^^^^^^^^^^
  File "/root/ktransformers/ktransformers/operators/base_operator.py", line 49, in __getattr__
    return super(nn.Module, nn.Module.__getattr__(self, "orig_module")).__getattribute__(name) # if this attr belongs to orig_module but not in nn.Module.__dict__
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'DeepseekV3Attention' object has no attribute 'forward_linux'

<!-- gh-comment-id:2664928368 --> @bluezd commented on GitHub (Feb 18, 2025): @tonyeatsm 能分享下具体是怎么改的吗？ 我修改后报如下错误： ``` Chat: hi Traceback (most recent call last): File "/root/ktransformers/ktransformers/operators/base_operator.py", line 42, in __getattr__ return object.__getattribute__(self, name) # if this attr belongs to BaseInjectedModule ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'KDeepseekV2Attention' object has no attribute 'forward_linux'. Did you mean: 'forward_chunck'? During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/root/ktransformers/ktransformers/operators/base_operator.py", line 47, in __getattr__ return nn.Module.__getattr__(self, "orig_module").__getattr__(name) # if this attr belongs to orig_module ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__ raise AttributeError( AttributeError: 'DeepseekV3Attention' object has no attribute 'forward_linux' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/ktransformers/ktransformers/local_chat.py", line 187, in <module> fire.Fire(local_chat) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/local_chat.py", line 181, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/util/utils.py", line 156, in prefill_and_generate logits = model( ^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward outputs = self.model( ^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/operators/models.py", line 722, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ if os.name == 'nt': return self.forward_windows( hidden_states, attention_mask, position_ids, past_key_value, output_attentions, use_cache, cache_position, **kwargs, ) else: return self.forward_linux( hidden_states, attention_mask, position_ids, past_key_value, output_attentions, use_cache, cache_position, **kwargs, ) class KLlamaAttention(BaseInjectedModule): """Multi-headed attention from 'Attention Is All You Need' paper""" def __init__(self, key: str, gguf_loader : GGUFLoader, config: PretrainedConfig, orig_module: nn.Module, prefill_device: str = "cuda", generate_device: str = "cuda", **kwargs): BaseInjectedModule.__init__(self, key, gguf_loader, config, orig_module, prefill_device, generate_device, **kwargs) self.orig_module.__init__(orig_module.config, orig_module.layer_idx) def apply_rotary_pos_emb(self, q, k, cos, sin, position_ids=None, unsqueeze_dim=1): """Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. "ktransformers/operators/attention.py" 710L, 34335B 581,1 83% def __init__(self, key: str, gguf_loader : GGUFLoader, config: PretrainedConfig, orig_module: nn.Module, prefill_device: str = "cuda", generate_device: str = "cuda", **kwargs): BaseInjectedModule.__init__(self, key, gguf_loader, config, orig_module, prefill_device, generate_device, **kwargs) self.orig_module.__init__(orig_module.config, orig_module.layer_idx) def apply_rotary_pos_emb(self, q, k, cos, sin, position_ids=None, unsqueeze_dim=1): """Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): def __init__(self, key: str, gguf_loader : GGUFLoader, config: PretrainedConfig, orig_module: nn.Module, prefill_device: str = "cuda", generate_device: str = "cuda", **kwargs): BaseInjectedModule.__init__(self, key, gguf_loader, config, orig_module, prefill_device, generate_device, **kwargs) self.orig_module.__init__(orig_module.config, orig_module.layer_idx) def apply_rotary_pos_emb(self, q, k, cos, sin, position_ids=None, unsqueeze_dim=1): """Applies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. """ cos = cos.unsqueeze(unsqueeze_dim) sin = sin.unsqueeze(unsqueeze_dim) q_embed = (q * cos) + (rotate_half(q) * sin) k_embed = (k * cos) + (rotate_half(k) * sin) return q_embed, k_embed def forward( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_value: Optional[Cache] = None, search hit BOTTOM, continuing at TOP self.attn_mask[:, :, :, :cur_idx] = 0 chunk_mask = torch.narrow(self.attn_mask, 2, 0, min(self.chunck_size, q_len-cur_idx)) cur_output, _, _ = self.forward_chunck( hidden_states[:, cur_idx:min(cur_idx + self.chunck_size, q_len), ...], chunk_mask, position_ids[:, cur_idx:min(cur_idx + self.chunck_size, q_len)], past_key_value, output_attentions, use_cache, cache_position[cur_idx:min(cur_idx + self.chunck_size, q_len)], **kwargs ) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/operators/attention.py", line 572, in forward return self.forward_linux( ^^^^^^^^^^^^^^^^^^ File "/root/ktransformers/ktransformers/operators/base_operator.py", line 49, in __getattr__ return super(nn.Module, nn.Module.__getattr__(self, "orig_module")).__getattribute__(name) # if this attr belongs to orig_module but not in nn.Module.__dict__ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'DeepseekV3Attention' object has no attribute 'forward_linux' ```

bsakaguchi commented 2026-10-29 14:36:35 +00:00

Author

Owner

Copy link

@bluezd commented on GitHub (Feb 18, 2025):

main 分支中并没有 forward_linux()这个方法 https://github.com/kvcache-ai/ktransformers/blob/main/ktransformers/operators/attention.py#L549-L593

<!-- gh-comment-id:2664956577 --> @bluezd commented on GitHub (Feb 18, 2025): main 分支中并没有 `forward_linux()`这个方法 https://github.com/kvcache-ai/ktransformers/blob/main/ktransformers/operators/attention.py#L549-L593

bsakaguchi commented 2026-10-29 14:36:35 +00:00

Author

Owner

Copy link

@ShinoharaHare commented on GitHub (Feb 19, 2025):

分享另一種改法，會比強制使用 forward_windows 快 1.5~2.0 倍

github.com/kvcache-ai/ktransformers@89f8218a2a/ktransformers/operators/attention.py (L274-L279)

+ original_dtype = query_states.dtype
+ target_dtype = torch.half
+ query_states = query_states.to(target_dtype)
+ compressed_kv_with_k_pe = compressed_kv_with_k_pe.to(target_dtype)
+ compressed_kv = compressed_kv.to(target_dtype)
+ attn_output = attn_output.to(target_dtype)

decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output,
                             page_table,
                             position_ids.squeeze(0).to(torch.int32)+1, attn_logits,
                             4, #num_kv_splits # follow vLLM, fix it TODO
                             self.softmax_scale,
                             past_key_value.page_size)

+ attn_output = attn_output.to(original_dtype)

github.com/kvcache-ai/ktransformers@89f8218a2a/ktransformers/operators/attention.py (L320-L326)

- attn_output = flash_attn_func( 
-     query_states, 
-     key_states, 
-     value_states_padded, 
-     softmax_scale=self.softmax_scale, 
-     causal=True, 
- )
+ attn_output = F.scaled_dot_product_attention(
+     query_states.transpose(1, 2),
+     key_states.transpose(1, 2),
+     value_states_padded.transpose(1, 2),
+     scale=self.softmax_scale,
+     is_causal=True
+ ).transpose(1, 2)

<!-- gh-comment-id:2667520838 --> @ShinoharaHare commented on GitHub (Feb 19, 2025): 分享另一種改法，會比強制使用 `forward_windows` 快 1.5~2.0 倍 https://github.com/kvcache-ai/ktransformers/blob/89f8218a2ab7ff82fa54dbfe30df741c574317fc/ktransformers/operators/attention.py#L274-L279 ```diff + original_dtype = query_states.dtype + target_dtype = torch.half + query_states = query_states.to(target_dtype) + compressed_kv_with_k_pe = compressed_kv_with_k_pe.to(target_dtype) + compressed_kv = compressed_kv.to(target_dtype) + attn_output = attn_output.to(target_dtype) decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output, page_table, position_ids.squeeze(0).to(torch.int32)+1, attn_logits, 4, #num_kv_splits # follow vLLM, fix it TODO self.softmax_scale, past_key_value.page_size) + attn_output = attn_output.to(original_dtype) ``` https://github.com/kvcache-ai/ktransformers/blob/89f8218a2ab7ff82fa54dbfe30df741c574317fc/ktransformers/operators/attention.py#L320-L326 ```diff - attn_output = flash_attn_func( - query_states, - key_states, - value_states_padded, - softmax_scale=self.softmax_scale, - causal=True, - ) + attn_output = F.scaled_dot_product_attention( + query_states.transpose(1, 2), + key_states.transpose(1, 2), + value_states_padded.transpose(1, 2), + scale=self.softmax_scale, + is_causal=True + ).transpose(1, 2) ```

bsakaguchi commented 2026-10-29 14:36:35 +00:00

Author

Owner

Copy link

@chaucerling commented on GitHub (Feb 19, 2025):

按 https://github.com/kvcache-ai/ktransformers/issues/374#issuecomment-2667520838 的修改，有提升，不知道是不是带宽瓶颈了

<!-- gh-comment-id:2667895017 --> @chaucerling commented on GitHub (Feb 19, 2025):   按 https://github.com/kvcache-ai/ktransformers/issues/374#issuecomment-2667520838 的修改，有提升，不知道是不是带宽瓶颈了

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@Dielianss commented on GitHub (Feb 21, 2025):

在 Linux 上 KDeepseekV2Attention 還是會使用 FA2
ktransformers/ktransformers/operators/attention.py
Lines 310 to 316 in 65d73ea
attn_output = flash_attn_func(
query_states,
key_states,
value_states_padded,
softmax_scale=self.softmax_scale,
causal=True,
)
要強制使用 forward_windows 才行
ktransformers/ktransformers/operators/attention.py
Lines 393 to 425 in 65d73ea
def forward(
self,
hidden_states: torch.Tensor,
attention_mask: Optional[torch.Tensor] = None,
position_ids: Optional[torch.LongTensor] = None,
past_key_value: Optional[Cache] = None,
output_attentions: bool = False,
use_cache: bool = False,
cache_position: Optional[torch.LongTensor] = None,
**kwargs,
) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
if os.name == 'nt':
return self.forward_windows(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)
else:
return self.forward_linux(
hidden_states,
attention_mask,
position_ids,
past_key_value,
output_attentions,
use_cache,
cache_position,
**kwargs,
)

请问您现在跑通了吗？在解决了这个报错之后。

是的，V100+Cuda 11.8可以成功运行

<!-- gh-comment-id:2674196805 --> @Dielianss commented on GitHub (Feb 21, 2025): > > 在 Linux 上 `KDeepseekV2Attention` 還是會使用 FA2 > > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/65d73ea3f862bd33712db2c5ce81e745c24265b1/ktransformers/operators/attention.py#L310-L316) > > Lines 310 to 316 in [65d73ea](/kvcache-ai/ktransformers/commit/65d73ea3f862bd33712db2c5ce81e745c24265b1) > > attn_output = flash_attn_func( > > query_states, > > key_states, > > value_states_padded, > > softmax_scale=self.softmax_scale, > > causal=True, > > ) > > 要強制使用 `forward_windows` 才行 > > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/65d73ea3f862bd33712db2c5ce81e745c24265b1/ktransformers/operators/attention.py#L393-L425) > > Lines 393 to 425 in [65d73ea](/kvcache-ai/ktransformers/commit/65d73ea3f862bd33712db2c5ce81e745c24265b1) > > def forward( > > self, > > hidden_states: torch.Tensor, > > attention_mask: Optional[torch.Tensor] = None, > > position_ids: Optional[torch.LongTensor] = None, > > past_key_value: Optional[Cache] = None, > > output_attentions: bool = False, > > use_cache: bool = False, > > cache_position: Optional[torch.LongTensor] = None, > > **kwargs, > > ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: > > if os.name == 'nt': > > return self.forward_windows( > > hidden_states, > > attention_mask, > > position_ids, > > past_key_value, > > output_attentions, > > use_cache, > > cache_position, > > **kwargs, > > ) > > else: > > return self.forward_linux( > > hidden_states, > > attention_mask, > > position_ids, > > past_key_value, > > output_attentions, > > use_cache, > > cache_position, > > **kwargs, > > ) > > 请问您现在跑通了吗？在解决了这个报错之后。 是的，V100+Cuda 11.8可以成功运行

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@JimiZhou commented on GitHub (Feb 21, 2025):

使用8卡T4可以跑起来了，有一处报错需要额外修改：
报错信息：
triton.runtime.autotuner.0utofResources: out of resource: shared memory, Required: 89088, Hardware limit:65536. Reducing block sizes or'num stages may help.`
解决方法：
在triton_attention.py内把num_stages改成1

<!-- gh-comment-id:2674746155 --> @JimiZhou commented on GitHub (Feb 21, 2025): 使用8卡T4可以跑起来了，有一处报错需要额外修改： 报错信息： `triton.runtime.autotuner.0utofResources: out of resource: shared memory, Required: 89088, Hardware limit:65536. Reducing block sizes or'num stages` may help.` 解决方法： 在triton_attention.py内把num_stages改成1

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@youde2000 commented on GitHub (Feb 22, 2025):

为什么我跑出来的结果都是be~~~~或者in [[ be~~~~这种？

<!-- gh-comment-id:2675973695 --> @youde2000 commented on GitHub (Feb 22, 2025): 为什么我跑出来的结果都是be~~~~或者in [[ be~~~~这种？

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@jcheunglin commented on GitHub (Feb 24, 2025):

4*t4的机器改完可以运行，q2kl版本，回答有模有样

<!-- gh-comment-id:2677863857 --> @jcheunglin commented on GitHub (Feb 24, 2025): 4*t4的机器改完可以运行，q2kl版本，回答有模有样 

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@bones7456 commented on GitHub (Feb 26, 2025):

分享另一種改法，會比強制使用 forward_windows 快 1.5~2.0 倍

ktransformers/ktransformers/operators/attention.py

Lines 274 to 279 in 89f8218

decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output,
page_table,
position_ids.squeeze(0).to(torch.int32)+1, attn_logits,
4, #num_kv_splits # follow vLLM, fix it TODO
self.softmax_scale,
past_key_value.page_size)

• original_dtype = query_states.dtype
• target_dtype = torch.half
• query_states = query_states.to(target_dtype)
• compressed_kv_with_k_pe = compressed_kv_with_k_pe.to(target_dtype)
• compressed_kv = compressed_kv.to(target_dtype)
• attn_output = attn_output.to(target_dtype)

decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output,
page_table,
position_ids.squeeze(0).to(torch.int32)+1, attn_logits,
4, #num_kv_splits # follow vLLM, fix it TODO
self.softmax_scale,
past_key_value.page_size)

• attn_output = attn_output.to(original_dtype)
  ktransformers/ktransformers/operators/attention.py

Lines 320 to 326 in 89f8218

attn_output = flash_attn_func(
query_states,
key_states,
value_states_padded,
softmax_scale=self.softmax_scale,
causal=True,
)

• attn_output = flash_attn_func(

• query_states, 
  

• key_states, 
  

• value_states_padded, 
  

• softmax_scale=self.softmax_scale, 
  

• causal=True, 
  

• )

• attn_output = F.scaled_dot_product_attention(

• query_states.transpose(1, 2),
  

• key_states.transpose(1, 2),
  

• value_states_padded.transpose(1, 2),
  

• scale=self.softmax_scale,
  

• is_causal=True
  

• ).transpose(1, 2)

我这边4块V100，用这个办法解决了问题，跑起来了，谢谢🙏

<!-- gh-comment-id:2684458831 --> @bones7456 commented on GitHub (Feb 26, 2025): > 分享另一種改法，會比強制使用 `forward_windows` 快 1.5~2.0 倍 > > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/89f8218a2ab7ff82fa54dbfe30df741c574317fc/ktransformers/operators/attention.py#L274-L279) > > Lines 274 to 279 in [89f8218](/kvcache-ai/ktransformers/commit/89f8218a2ab7ff82fa54dbfe30df741c574317fc) > > decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output, > page_table, > position_ids.squeeze(0).to(torch.int32)+1, attn_logits, > 4, #num_kv_splits # follow vLLM, fix it TODO > self.softmax_scale, > past_key_value.page_size) > + original_dtype = query_states.dtype > + target_dtype = torch.half > + query_states = query_states.to(target_dtype) > + compressed_kv_with_k_pe = compressed_kv_with_k_pe.to(target_dtype) > + compressed_kv = compressed_kv.to(target_dtype) > + attn_output = attn_output.to(target_dtype) > > decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output, > page_table, > position_ids.squeeze(0).to(torch.int32)+1, attn_logits, > 4, #num_kv_splits # follow vLLM, fix it TODO > self.softmax_scale, > past_key_value.page_size) > > + attn_output = attn_output.to(original_dtype) > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/89f8218a2ab7ff82fa54dbfe30df741c574317fc/ktransformers/operators/attention.py#L320-L326) > > Lines 320 to 326 in [89f8218](/kvcache-ai/ktransformers/commit/89f8218a2ab7ff82fa54dbfe30df741c574317fc) > > attn_output = flash_attn_func( > query_states, > key_states, > value_states_padded, > softmax_scale=self.softmax_scale, > causal=True, > ) > - attn_output = flash_attn_func( > - query_states, > - key_states, > - value_states_padded, > - softmax_scale=self.softmax_scale, > - causal=True, > - ) > + attn_output = F.scaled_dot_product_attention( > + query_states.transpose(1, 2), > + key_states.transpose(1, 2), > + value_states_padded.transpose(1, 2), > + scale=self.softmax_scale, > + is_causal=True > + ).transpose(1, 2) 我这边4块V100，用这个办法解决了问题，跑起来了，谢谢🙏

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@hubuwch commented on GitHub (Mar 20, 2025):

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的T4的单卡16G显存就爆了~~

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的单卡T4 16G显存就爆了~~

<!-- gh-comment-id:2739446096 --> @hubuwch commented on GitHub (Mar 20, 2025): > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 按照这个修改，我的T4的单卡16G显存就爆了~~ > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 按照这个修改，我的单卡T4 16G显存就爆了~~

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@jeffrey4l commented on GitHub (Mar 26, 2025):

4*t4的机器改完可以运行，q2kl版本，回答有模有样

@jcheunglin @JimiZhou how many T4 GPU memory used when running Q2? thanks.

<!-- gh-comment-id:2754402198 --> @jeffrey4l commented on GitHub (Mar 26, 2025): > 4*t4的机器改完可以运行，q2kl版本，回答有模有样 > >  @jcheunglin @JimiZhou how many T4 GPU memory used when running Q2? thanks.

bsakaguchi commented 2026-10-29 14:36:36 +00:00

Author

Owner

Copy link

@MaomaoMAo-17 commented on GitHub (Mar 31, 2025):

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的T4的单卡16G显存就爆了~~

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的单卡T4 16G显存就爆了~~

p5000出现同样的问题 爆显存 是我哪里没改对吗。。。
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 0 has a total capacity of 15.88 GiB of which 195.88 MiB is free. Including non-PyTorch memory, this process has 15.68 GiB memory in use. Of the allocated memory 15.53 GiB is allocated by PyTorch, and 37.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True已启用

<!-- gh-comment-id:2765130138 --> @MaomaoMAo-17 commented on GitHub (Mar 31, 2025): > > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 > > 按照这个修改，我的T4的单卡16G显存就爆了~~ > > > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 > > 按照这个修改，我的单卡T4 16G显存就爆了~~ p5000出现同样的问题 爆显存 是我哪里没改对吗。。。 torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 0 has a total capacity of 15.88 GiB of which 195.88 MiB is free. Including non-PyTorch memory, this process has 15.68 GiB memory in use. Of the allocated memory 15.53 GiB is allocated by PyTorch, and 37.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True已启用

bsakaguchi commented 2026-10-29 14:36:37 +00:00

Author

Owner

Copy link

@20000419 commented on GitHub (Apr 2, 2025):

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的T4的单卡16G显存就爆了~~

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的单卡T4 16G显存就爆了~~

p5000出现同样的问题 爆显存 是我哪里没改对吗。。。 torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 0 has a total capacity of 15.88 GiB of which 195.88 MiB is free. Including non-PyTorch memory, this process has 15.68 GiB memory in use. Of the allocated memory 15.53 GiB is allocated by PyTorch, and 37.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True已启用

应该不是没改对，是因为Volta和Turing架构不支持marlin算子，这会导致实际需求的显存翻倍，我是双卡2080ti各占用17个g才跑起来，T4的话最好是四卡。。。

<!-- gh-comment-id:2771187526 --> @20000419 commented on GitHub (Apr 2, 2025): > > > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 > > > > > > 按照这个修改，我的T4的单卡16G显存就爆了~~ > > > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 > > > > > > 按照这个修改，我的单卡T4 16G显存就爆了~~ > > p5000出现同样的问题 爆显存 是我哪里没改对吗。。。 torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 0 has a total capacity of 15.88 GiB of which 195.88 MiB is free. Including non-PyTorch memory, this process has 15.68 GiB memory in use. Of the allocated memory 15.53 GiB is allocated by PyTorch, and 37.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) > > PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True已启用 应该不是没改对，是因为Volta和Turing架构不支持marlin算子，这会导致实际需求的显存翻倍，我是双卡2080ti各占用17个g才跑起来，T4的话最好是四卡。。。

bsakaguchi commented 2026-10-29 14:36:37 +00:00

Author

Owner

Copy link

@morphishk commented on GitHub (Apr 6, 2025):

我使用两张2080ti，128G内存跑1.58bit的R1，能跑但是内存没有任何占用，128G内存是空的只能跑0.44token，加到160可以跑3.5token，但是内存还是没有任何占用，问问大佬知道这事什么情况嘛？

<!-- gh-comment-id:2781498177 --> @morphishk commented on GitHub (Apr 6, 2025): 我使用两张2080ti，128G内存跑1.58bit的R1，能跑但是内存没有任何占用，128G内存是空的只能跑0.44token，加到160可以跑3.5token，但是内存还是没有任何占用，问问大佬知道这事什么情况嘛？

bsakaguchi commented 2026-10-29 14:36:37 +00:00

Author

Owner

Copy link

@MaomaoMAo-17 commented on GitHub (Apr 8, 2025):

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的T4的单卡16G显存就爆了~~

首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。

按照这个修改，我的单卡T4 16G显存就爆了~~

p5000出现同样的问题 爆显存 是我哪里没改对吗。。。 torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 0 has a total capacity of 15.88 GiB of which 195.88 MiB is free. Including non-PyTorch memory, this process has 15.68 GiB memory in use. Of the allocated memory 15.53 GiB is allocated by PyTorch, and 37.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True已启用

应该不是没改对，是因为Volta和Turing架构不支持marlin算子，这会导致实际需求的显存翻倍，我是双卡2080ti各占用17个g才跑起来，T4的话最好是四卡。。。

是的 已经换卡了

<!-- gh-comment-id:2785164626 --> @MaomaoMAo-17 commented on GitHub (Apr 8, 2025): > > > > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 > > > > > > > > > 按照这个修改，我的T4的单卡16G显存就爆了~~ > > > > 首先，git下载最新的源码。 然后，修改源码中的DeepSeek-V3-Chat-multi-gpu-4.yaml，和其他所有相关的yaml文件，将里面所有的KLinearMarlin换成KLinearTorch； 然后，你需要从ktransformer的源码进行编译，直到在本机编译成功。 然后，安装flash-attn，不会用到，但是不安装会报错。 然后，修改local_chat.py，将里面所有的flash_attention_2换成eager； 然后，运行locai_chat.py,务必按照官方教程的命令来，根据本机参数调整； 然后，在运行过程中检查内存占用，通过top命令观察其调用情况，单路cpu上的内存容量一定要大于模型的完整大小。（多路也只是复制） 最后，确认模型完整加载到内存，特定权重层完整加载到显存后，在chat界面尝试输入内容，观察是否还有报错出现。 > > > > > > > > > 按照这个修改，我的单卡T4 16G显存就爆了~~ > > > > > > p5000出现同样的问题 爆显存 是我哪里没改对吗。。。 torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 0 has a total capacity of 15.88 GiB of which 195.88 MiB is free. Including non-PyTorch memory, this process has 15.68 GiB memory in use. Of the allocated memory 15.53 GiB is allocated by PyTorch, and 37.10 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) > > PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True已启用 > > 应该不是没改对，是因为Volta和Turing架构不支持marlin算子，这会导致实际需求的显存翻倍，我是双卡2080ti各占用17个g才跑起来，T4的话最好是四卡。。。 是的 已经换卡了

bsakaguchi commented 2026-10-29 14:36:37 +00:00

Author

Owner

Copy link

@jercylew commented on GitHub (Nov 9, 2025):

分享另一種改法，會比強制使用 forward_windows 快 1.5~2.0 倍
ktransformers/ktransformers/operators/attention.py
Lines 274 to 279 in 89f8218
decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output,
page_table,
position_ids.squeeze(0).to(torch.int32)+1, attn_logits,
4, #num_kv_splits # follow vLLM, fix it TODO
self.softmax_scale,
past_key_value.page_size)

• original_dtype = query_states.dtype
• target_dtype = torch.half
• query_states = query_states.to(target_dtype)
• compressed_kv_with_k_pe = compressed_kv_with_k_pe.to(target_dtype)
• compressed_kv = compressed_kv.to(target_dtype)
• attn_output = attn_output.to(target_dtype)

decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output,
page_table,
position_ids.squeeze(0).to(torch.int32)+1, attn_logits,
4, #num_kv_splits # follow vLLM, fix it TODO
self.softmax_scale,
past_key_value.page_size)

• attn_output = attn_output.to(original_dtype)
  ktransformers/ktransformers/operators/attention.py

Lines 320 to 326 in 89f8218
attn_output = flash_attn_func(
query_states,
key_states,
value_states_padded,
softmax_scale=self.softmax_scale,
causal=True,
)

• attn_output = flash_attn_func(

• query_states, 
  

• key_states, 
  

• value_states_padded, 
  

• softmax_scale=self.softmax_scale, 
  

• causal=True, 
  

• )

• attn_output = F.scaled_dot_product_attention(

• query_states.transpose(1, 2),
  

• key_states.transpose(1, 2),
  

• value_states_padded.transpose(1, 2),
  

• scale=self.softmax_scale,
  

• is_causal=True
  

• ).transpose(1, 2)

我这边4块V100，用这个办法解决了问题，跑起来了，谢谢🙏

我这里也是V100 x 16, 试了两天没跑起来，请问你装的是哪个版本的CUDA和哪个版本的PyTorch?

<!-- gh-comment-id:3508384125 --> @jercylew commented on GitHub (Nov 9, 2025): > > 分享另一種改法，會比強制使用 `forward_windows` 快 1.5~2.0 倍 > > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/89f8218a2ab7ff82fa54dbfe30df741c574317fc/ktransformers/operators/attention.py#L274-L279) > > Lines 274 to 279 in [89f8218](/kvcache-ai/ktransformers/commit/89f8218a2ab7ff82fa54dbfe30df741c574317fc) > > decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output, > > page_table, > > position_ids.squeeze(0).to(torch.int32)+1, attn_logits, > > 4, #num_kv_splits # follow vLLM, fix it TODO > > self.softmax_scale, > > past_key_value.page_size) > > > > * original_dtype = query_states.dtype > > * target_dtype = torch.half > > * query_states = query_states.to(target_dtype) > > * compressed_kv_with_k_pe = compressed_kv_with_k_pe.to(target_dtype) > > * compressed_kv = compressed_kv.to(target_dtype) > > * attn_output = attn_output.to(target_dtype) > > > > decode_attention_fwd_grouped(query_states, compressed_kv_with_k_pe, compressed_kv, attn_output, > > page_table, > > position_ids.squeeze(0).to(torch.int32)+1, attn_logits, > > 4, #num_kv_splits # follow vLLM, fix it TODO > > self.softmax_scale, > > past_key_value.page_size) > > > > * attn_output = attn_output.to(original_dtype) > > [ktransformers/ktransformers/operators/attention.py](https://github.com/kvcache-ai/ktransformers/blob/89f8218a2ab7ff82fa54dbfe30df741c574317fc/ktransformers/operators/attention.py#L320-L326) > > > > Lines 320 to 326 in [89f8218](/kvcache-ai/ktransformers/commit/89f8218a2ab7ff82fa54dbfe30df741c574317fc) > > attn_output = flash_attn_func( > > query_states, > > key_states, > > value_states_padded, > > softmax_scale=self.softmax_scale, > > causal=True, > > ) > > > > * attn_output = flash_attn_func( > > * ``` > > query_states, > > ``` > > * ``` > > key_states, > > ``` > > * ``` > > value_states_padded, > > ``` > > * ``` > > softmax_scale=self.softmax_scale, > > ``` > > * ``` > > causal=True, > > ``` > > * ) > > > > > > * attn_output = F.scaled_dot_product_attention( > > * ``` > > query_states.transpose(1, 2), > > ``` > > * ``` > > key_states.transpose(1, 2), > > ``` > > * ``` > > value_states_padded.transpose(1, 2), > > ``` > > * ``` > > scale=self.softmax_scale, > > ``` > > * ``` > > is_causal=True > > ``` > > * ).transpose(1, 2) > > 我这边4块V100，用这个办法解决了问题，跑起来了，谢谢🙏 我这里也是V100 x 16, 试了两天没跑起来，请问你装的是哪个版本的CUDA和哪个版本的PyTorch?

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#248

No description provided.