[GH-ISSUE #1259] [Bug] "ValueError: too many values to unpack" when trying to run Qwen3-30B-A3B with v0.3 ktransformers #850

New issue

Open

opened 2026-10-29 14:45:22 +00:00 by bsakaguchi · 7 comments

bsakaguchi commented 2026-10-29 14:45:22 +00:00

Owner

Copy link

Originally created by @zondie17 on GitHub (May 6, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1259

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
• 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

First run I got AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'? like https://github.com/kvcache-ai/ktransformers/issues/1238, so I tried the fix @vickiegpt provided and @createthis provided, then got another error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/local_chat.py", line 187, in <module>
    fire.Fire(local_chat)
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/local_chat.py", line 181, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/util/utils.py", line 225, in prefill_and_generate
    logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/util/utils.py", line 182, in chunk_prefill
    logits = model(
             ^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/models/modeling_qwen3_moe.py", line 1172, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/operators/models.py", line 363, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/models/modeling_qwen3_moe.py", line 383, in forward
    hidden_states = self.input_layernorm(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/operators/layernorm.py", line 144, in forward
    bsz, hidden_size = x.shape
    ^^^^^^^^^^^^^^^^
ValueError: too many values to unpack (expected 2)

Reproduction

Qwen3: https://huggingface.co/lmstudio-community/Qwen3-30B-A3B-GGUF/resolve/main/Qwen3-30B-A3B-Q4_K_M.gguf
ktransformers version: 753049
Modifications made are as follows:

diff --git a/ktransformers/local_chat.py b/ktransformers/local_chat.py
index 928de48..973c476 100644
--- a/ktransformers/local_chat.py
+++ b/ktransformers/local_chat.py
@@ -25,6 +25,7 @@ import fire
 from ktransformers.optimize.optimize import optimize_and_load_gguf
 from ktransformers.models.modeling_deepseek import DeepseekV2ForCausalLM
 from ktransformers.models.modeling_qwen2_moe import Qwen2MoeForCausalLM
+from ktransformers.models.modeling_qwen3_moe import Qwen3MoeForCausalLM
 from ktransformers.models.modeling_deepseek_v3 import DeepseekV3ForCausalLM
 from ktransformers.models.modeling_llama import LlamaForCausalLM
 from ktransformers.models.modeling_mixtral import MixtralForCausalLM
@@ -37,6 +38,7 @@ custom_models = {
     "DeepseekV2ForCausalLM": DeepseekV2ForCausalLM,
     "DeepseekV3ForCausalLM": DeepseekV3ForCausalLM,
     "Qwen2MoeForCausalLM": Qwen2MoeForCausalLM,
+    "Qwen3MoeForCausalLM": Qwen3MoeForCausalLM,
     "LlamaForCausalLM": LlamaForCausalLM,
     "MixtralForCausalLM": MixtralForCausalLM,
 }
diff --git a/ktransformers/models/modeling_qwen3_moe.py b/ktransformers/models/modeling_qwen3_moe.py
index 175f88c..0bec253 100644
--- a/ktransformers/models/modeling_qwen3_moe.py
+++ b/ktransformers/models/modeling_qwen3_moe.py
@@ -1049,6 +1049,46 @@ class Qwen3MoeForCausalLM(Qwen3MoePreTrainedModel, GenerationMixin):
         # Initialize weights and apply final processing
         self.post_init()
 
+    def _get_logits_warper(self, generation_config, device=None):
+        from transformers.generation.logits_process import (
+            LogitsProcessorList,
+            TemperatureLogitsWarper,
+            TopPLogitsWarper,
+            TopKLogitsWarper,
+            MinPLogitsWarper,
+            TypicalLogitsWarper,
+        )
+        
+        # Initialize with an empty processor list
+        warpers = LogitsProcessorList()
+        
+        # Add temperature warper if applicable
+        temperature = generation_config.temperature
+        if temperature is not None and temperature != 1.0:
+            warpers.append(TemperatureLogitsWarper(temperature))
+            
+        # Add top_p warper if applicable
+        top_p = generation_config.top_p
+        if top_p is not None and top_p < 1.0:
+            warpers.append(TopPLogitsWarper(top_p))
+        
+        # Add top_k warper if applicable
+        top_k = generation_config.top_k
+        if top_k is not None and top_k != 0:
+            warpers.append(TopKLogitsWarper(top_k))
+            
+        # Add min_p warper if applicable
+        min_p = getattr(generation_config, "min_p", None)
+        if min_p is not None and min_p > 0.0:
+            warpers.append(MinPLogitsWarper(min_p))
+            
+        # Add typical_p warper if applicable
+        typical_p = getattr(generation_config, "typical_p", None)
+        if typical_p is not None and 0 < typical_p < 1.0:
+            warpers.append(TypicalLogitsWarper(typical_p))
+            
+        return warpers
+
     def get_input_embeddings(self):
         return self.model.embed_tokens

python -m ktransformers.local_chat --architectures Qwen3MoeForCausalLM --model_path Qwen/Qwen3-30B-A3B --gguf_path ./Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --backend_type balance_serce

Environment

OS: Ubuntu 22.04.1
Hardware: AMD EPYC 7V13, A100 80GB
Software: python 3.11, pytorch 2.7.0, cuda 12.8, cmake 3.29.6

Originally created by @zondie17 on GitHub (May 6, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1259 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug First run I got `AttributeError: 'Qwen3MoeForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'?` like https://github.com/kvcache-ai/ktransformers/issues/1238, so I tried the fix @vickiegpt provided and @createthis provided, then got another error: ``` Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/local_chat.py", line 187, in <module> fire.Fire(local_chat) File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/local_chat.py", line 181, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/util/utils.py", line 225, in prefill_and_generate logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/util/utils.py", line 182, in chunk_prefill logits = model( ^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/models/modeling_qwen3_moe.py", line 1172, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/operators/models.py", line 363, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/models/modeling_qwen3_moe.py", line 383, in forward hidden_states = self.input_layernorm(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/v-zhaoxiang/zhangyiwei/kTransformers/ktransformers/ktransformers/operators/layernorm.py", line 144, in forward bsz, hidden_size = x.shape ^^^^^^^^^^^^^^^^ ValueError: too many values to unpack (expected 2) ``` ### Reproduction Qwen3: https://huggingface.co/lmstudio-community/Qwen3-30B-A3B-GGUF/resolve/main/Qwen3-30B-A3B-Q4_K_M.gguf ktransformers version: `753049` Modifications made are as follows: ``` diff --git a/ktransformers/local_chat.py b/ktransformers/local_chat.py index 928de48..973c476 100644 --- a/ktransformers/local_chat.py +++ b/ktransformers/local_chat.py @@ -25,6 +25,7 @@ import fire from ktransformers.optimize.optimize import optimize_and_load_gguf from ktransformers.models.modeling_deepseek import DeepseekV2ForCausalLM from ktransformers.models.modeling_qwen2_moe import Qwen2MoeForCausalLM +from ktransformers.models.modeling_qwen3_moe import Qwen3MoeForCausalLM from ktransformers.models.modeling_deepseek_v3 import DeepseekV3ForCausalLM from ktransformers.models.modeling_llama import LlamaForCausalLM from ktransformers.models.modeling_mixtral import MixtralForCausalLM @@ -37,6 +38,7 @@ custom_models = { "DeepseekV2ForCausalLM": DeepseekV2ForCausalLM, "DeepseekV3ForCausalLM": DeepseekV3ForCausalLM, "Qwen2MoeForCausalLM": Qwen2MoeForCausalLM, + "Qwen3MoeForCausalLM": Qwen3MoeForCausalLM, "LlamaForCausalLM": LlamaForCausalLM, "MixtralForCausalLM": MixtralForCausalLM, } diff --git a/ktransformers/models/modeling_qwen3_moe.py b/ktransformers/models/modeling_qwen3_moe.py index 175f88c..0bec253 100644 --- a/ktransformers/models/modeling_qwen3_moe.py +++ b/ktransformers/models/modeling_qwen3_moe.py @@ -1049,6 +1049,46 @@ class Qwen3MoeForCausalLM(Qwen3MoePreTrainedModel, GenerationMixin): # Initialize weights and apply final processing self.post_init() + def _get_logits_warper(self, generation_config, device=None): + from transformers.generation.logits_process import ( + LogitsProcessorList, + TemperatureLogitsWarper, + TopPLogitsWarper, + TopKLogitsWarper, + MinPLogitsWarper, + TypicalLogitsWarper, + ) + + # Initialize with an empty processor list + warpers = LogitsProcessorList() + + # Add temperature warper if applicable + temperature = generation_config.temperature + if temperature is not None and temperature != 1.0: + warpers.append(TemperatureLogitsWarper(temperature)) + + # Add top_p warper if applicable + top_p = generation_config.top_p + if top_p is not None and top_p < 1.0: + warpers.append(TopPLogitsWarper(top_p)) + + # Add top_k warper if applicable + top_k = generation_config.top_k + if top_k is not None and top_k != 0: + warpers.append(TopKLogitsWarper(top_k)) + + # Add min_p warper if applicable + min_p = getattr(generation_config, "min_p", None) + if min_p is not None and min_p > 0.0: + warpers.append(MinPLogitsWarper(min_p)) + + # Add typical_p warper if applicable + typical_p = getattr(generation_config, "typical_p", None) + if typical_p is not None and 0 < typical_p < 1.0: + warpers.append(TypicalLogitsWarper(typical_p)) + + return warpers + def get_input_embeddings(self): return self.model.embed_tokens ``` ``` python -m ktransformers.local_chat --architectures Qwen3MoeForCausalLM --model_path Qwen/Qwen3-30B-A3B --gguf_path ./Qwen3-30B-A3B-GGUF/Qwen3-30B-A3B-Q4_K_M.gguf --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --backend_type balance_serce ``` ### Environment OS: Ubuntu 22.04.1 Hardware: AMD EPYC 7V13, A100 80GB Software: python 3.11, pytorch 2.7.0, cuda 12.8, cmake 3.29.6

bsakaguchi commented 2026-10-29 14:45:22 +00:00

Author

Owner

Copy link

@createthis commented on GitHub (May 6, 2025):

I'm not a member of the ktransformers team, just a user, but I'm pretty sure Qwen3 was never intended to work with the ktransformers backend. It works with the balance_serve backend though.

<!-- gh-comment-id:2854575287 --> @createthis commented on GitHub (May 6, 2025): I'm not a member of the ktransformers team, just a user, but I'm pretty sure Qwen3 was never intended to work with the ktransformers backend. It works with the balance_serve backend though.

bsakaguchi commented 2026-10-29 14:45:22 +00:00

Author

Owner

Copy link

@zondie17 commented on GitHub (May 8, 2025):

I'm not a member of the ktransformers team, just a user, but I'm pretty sure Qwen3 was never intended to work with the ktransformers backend. It works with the balance_serve backend though.

Thanks, but after making the above modifications and reinstalling ktransformers, I still encountered the same ValueError when using the balance_serve backend. Have you ever encountered this issue?

<!-- gh-comment-id:2861416486 --> @zondie17 commented on GitHub (May 8, 2025): > I'm not a member of the ktransformers team, just a user, but I'm pretty sure Qwen3 was never intended to work with the ktransformers backend. It works with the balance_serve backend though. Thanks, but after making the above modifications and reinstalling ktransformers, I still encountered the same ValueError when using the balance_serve backend. Have you ever encountered this issue?

bsakaguchi commented 2026-10-29 14:45:23 +00:00

Author

Owner

Copy link

@createthis commented on GitHub (May 8, 2025):

@zondie17 have you tried reverting your patch? It shouldn't be necessary for balance_serve. I patched ktransformers/models/modeling_deepseek_v3.py to run deepseek-v3 under the ktransformers backend, but I didn't have to patch ktransformers/models/modeling_qwen3_moe.py.

It's been a while since I was using that version, so I'm going off memory and my patch notes.

<!-- gh-comment-id:2861440523 --> @createthis commented on GitHub (May 8, 2025): @zondie17 have you tried reverting your patch? It shouldn't be necessary for `balance_serve`. I patched `ktransformers/models/modeling_deepseek_v3.py` to run deepseek-v3 under the `ktransformers` backend, but I didn't have to patch `ktransformers/models/modeling_qwen3_moe.py`. It's been a while since I was using that version, so I'm going off memory and my patch notes.

bsakaguchi commented 2026-10-29 14:45:23 +00:00

Author

Owner

Copy link

@zondie17 commented on GitHub (May 8, 2025):

@zondie17 have you tried reverting your patch? It shouldn't be necessary for balance_serve. I patched ktransformers/models/modeling_deepseek_v3.py to run deepseek-v3 under the ktransformers backend, but I didn't have to patch ktransformers/models/modeling_qwen3_moe.py.

It's been a while since I was using that version, so I'm going off memory and my patch notes.

Thanks, may I ask which version you are using now?

<!-- gh-comment-id:2861476491 --> @zondie17 commented on GitHub (May 8, 2025): > [@zondie17](https://github.com/zondie17) have you tried reverting your patch? It shouldn't be necessary for `balance_serve`. I patched `ktransformers/models/modeling_deepseek_v3.py` to run deepseek-v3 under the `ktransformers` backend, but I didn't have to patch `ktransformers/models/modeling_qwen3_moe.py`. > > It's been a while since I was using that version, so I'm going off memory and my patch notes. Thanks, may I ask which version you are using now?

bsakaguchi commented 2026-10-29 14:45:23 +00:00

Author

Owner

Copy link

@createthis commented on GitHub (May 8, 2025):

Thanks, may I ask which version you are using now?

I'm not using Qwen3. I tried it with balance_serve, and it was ok, but it lacked long context, which I need for agentic purposes. I also tried it with llama.cpp and long context. It didn't work well though. It just kept looping on a file I told it to read.

I'm currently running Deepseek-V3-0324:671b-Q4_K_M with ktransformers using the ktransformers backend. I have it checked out from main at commit 8dc1ab right now, which is somewhere between 0.2.4-post1 and 0.3.

It's not perfect. It crashes a lot. But it gives me long context and it's pretty fast on my machine and it follows instructions really well, so I like it.

<!-- gh-comment-id:2861520830 --> @createthis commented on GitHub (May 8, 2025): > Thanks, may I ask which version you are using now? I'm not using Qwen3. I tried it with `balance_serve`, and it was ok, but it lacked long context, which I need for agentic purposes. I also tried it with `llama.cpp` and long context. It didn't work well though. It just kept looping on a file I told it to read. I'm currently running Deepseek-V3-0324:671b-Q4_K_M with ktransformers using the `ktransformers` backend. I have it checked out from `main` at commit `8dc1ab` right now, which is somewhere between `0.2.4-post1` and `0.3`. It's not perfect. It crashes a lot. But it gives me long context and it's pretty fast on my machine and it follows instructions really well, so I like it.

bsakaguchi commented 2026-10-29 14:45:23 +00:00

Author

Owner

Copy link

@visuOwO commented on GitHub (Aug 15, 2025):

It‘s probably that you are using KQwen3MoeRMSNorm in the optimiaze rules. I think there are some differents in the interfaces for different methods, you can try modify it with DeepseekV3RMSNormTorch in the optimize rules yaml file.

<!-- gh-comment-id:3190617966 --> @visuOwO commented on GitHub (Aug 15, 2025): It‘s probably that you are using KQwen3MoeRMSNorm in the optimiaze rules. I think there are some differents in the interfaces for different methods, you can try modify it with DeepseekV3RMSNormTorch in the optimize rules yaml file.

bsakaguchi commented 2026-10-29 14:45:23 +00:00

Author

Owner

Copy link

@PPXGS commented on GitHub (Aug 29, 2025):

我的ktransformers版本是v0.3.1，不论加不加--backend_type balance_serce，也有ValueError: too many values to unpack (expected 2)的报错。后续你用这个local_chat方式成功运行qwen3-moe模型了吗？

<!-- gh-comment-id:3236377570 --> @PPXGS commented on GitHub (Aug 29, 2025): 我的ktransformers版本是v0.3.1，不论加不加--backend_type balance_serce，也有ValueError: too many values to unpack (expected 2)的报错。后续你用这个local_chat方式成功运行qwen3-moe模型了吗？

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#850

No description provided.