mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1235] [Bug] NotImplementedError: ggml_type 3 not implemented for unsloth/Qwen3-235B-A22B-128K-GGUF-Q4_1 #840
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#840
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @344303947 on GitHub (Apr 29, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1235
检查清单
问题描述
Injecting model.layers.93.input_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.layers.93.post_attention_layernorm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.norm as ktransformers.operators.layernorm . KQwen3MoeRMSNorm
Injecting model.rotary_emb as default
Injecting cache as default
Injecting lm_head as ktransformers.operators.linear . KTransformersLinear
loading token_embd.weight to cpu
loading token_embd.weight with CPU
Process SpawnProcess-1:
Traceback (most recent call last):
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 277, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 181, in init
optimize_and_load_gguf(self.model, optimize_config_path, gguf_path, config)
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf
load_weights(module, gguf_loader)
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/util/utils.py", line 127, in load_weights
module.load()
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/util/utils.py", line 125, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/util/utils.py", line 123, in load_weights
load_cur_state_dict(module, gguf_loader, prefix)
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/util/utils.py", line 113, in load_cur_state_dict
weights = load_dequantized_tensor(translated_key, device=device).to(dtype=target_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/sean/miniconda3/envs/kt3/lib/python3.11/site-packages/ktransformers/util/custom_gguf.py", line 351, in load_gguf_tensor
raise NotImplementedError(f"ggml_type {ggml_type} not implemented")
NotImplementedError: ggml_type 3 not implemented
GGML_TYPES = {
"F32": 0,
"F16": 1,
"Q4_0": 2,
"Q5_0": 6,
"Q8_0": 8,
"Q2_K": 10,
"Q3_K": 11,
"Q4_K": 12,
"Q5_K": 13,
"Q6_K": 14,
"IQ4_XS": 23,
"BF16": 30,
}
复现步骤
python ktransformers/server/main.py --architectures Qwen3MoeForCausalLM
--port 6688
--model_path /media/mydisk/models/Qwen3-235B-A22B
--gguf_path /media/mydisk/models/Qwen3-235B-A22B-128K-GGUF/Q4_1
--optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml
--model_name Qwen3Moe
--backend_type balance_serve
环境信息
3.0
unsloth/Qwen3-235B-A22B-128K-GGUF-Q4_1
@chandratan03 commented on GitHub (Apr 30, 2025):
Any update on this?
@CodeZ-Hao commented on GitHub (Apr 30, 2025):
I been told that this version can only fit IQ4_XS and Q4_KM, and I had passed in Q4_KM.
@344303947 commented on GitHub (May 1, 2025):