[GH-ISSUE #2113] MiniMax M2.7: AttributeError: 'LlamafileMoEWrapper' object has no attribute 'submit_write_weight_scale_to_buffer' #1281

New issue

Open

opened 2026-10-29 14:50:50 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented 2026-10-29 14:50:50 +00:00

Owner

Copy link

Originally created by @mrgaolei on GitHub (Jul 31, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2113

Reminder

• I have read the above rules and searched the existing issues.

System Info

AMD 7k62 with avx2
DDR4 256G
4090 48G *2

Ubuntu 24.04
miniconda
kt 0.6.4

Reproduction

/home/aigao/miniconda3/envs/kt-kernel/bin/python3 -m sglang.launch_server   --host 0.0.0.0 --port 30000   --model /home/aigao/models/MiniMax-M2.7   --kt-weight-path /home/aigao/models/MiniMax-M2.7-GGUF/UD-Q4_K_M   --kt-cpuinfer 76   --kt-threadpool-count 1   --kt-num-gpu-experts 8   --kt-method LLAMAFILE   --kt-gpu-prefill-token-threshold 500   --kt-enable-dynamic-expert-update   --attention-backend flashinfer   --trust-remote-code   --mem-fraction-static 0.9   --chunked-prefill-size 32768   --max-running-requests 1   --max-total-tokens 32768   --watchdog-timeout 3000   --enable-mixed-chunk   --tensor-parallel-size 2   --enable-p2p-check   --disable-shared-experts-fusion

  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 998, in forward
    return self.forward_impl(hidden_states, topk_output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 1017, in forward_impl
    combine_input = self.run_moe_core(
                    ^^^^^^^^^^^^^^^^^^
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 1038, in run_moe_core
    return self.quant_method.apply(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2959, in apply
    ctx = self._build_full_context(layer)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 3319, in _build_full_context
    _SHARED_FULL_CONTEXT.load(
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1678, in load
    self._prepare_weight_fp8(wrapper, None, gpu_experts_mask,
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1181, in _prepare_weight_fp8
    submit_write_expert(cpu_expert_ids[0], 0)
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1171, in submit_write_expert
    wrapper.submit_write_weight_scale_to_buffer(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LlamafileMoEWrapper' object has no attribute 'submit_write_weight_scale_to_buffer'

[2026-07-31 14:22:09] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
已杀死```


### Others

看起来是sglang没有针对纯avx2进行优化。请问我要切换到sglang的那个版本可以兼容纯avx2的CPU？

Originally created by @mrgaolei on GitHub (Jul 31, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2113 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info AMD 7k62 with avx2 DDR4 256G 4090 48G *2 Ubuntu 24.04 miniconda kt 0.6.4 ### Reproduction ```text /home/aigao/miniconda3/envs/kt-kernel/bin/python3 -m sglang.launch_server --host 0.0.0.0 --port 30000 --model /home/aigao/models/MiniMax-M2.7 --kt-weight-path /home/aigao/models/MiniMax-M2.7-GGUF/UD-Q4_K_M --kt-cpuinfer 76 --kt-threadpool-count 1 --kt-num-gpu-experts 8 --kt-method LLAMAFILE --kt-gpu-prefill-token-threshold 500 --kt-enable-dynamic-expert-update --attention-backend flashinfer --trust-remote-code --mem-fraction-static 0.9 --chunked-prefill-size 32768 --max-running-requests 1 --max-total-tokens 32768 --watchdog-timeout 3000 --enable-mixed-chunk --tensor-parallel-size 2 --enable-p2p-check --disable-shared-experts-fusion ``` ```text File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 998, in forward return self.forward_impl(hidden_states, topk_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 1017, in forward_impl combine_input = self.run_moe_core( ^^^^^^^^^^^^^^^^^^ File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 1038, in run_moe_core return self.quant_method.apply( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2959, in apply ctx = self._build_full_context(layer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 3319, in _build_full_context _SHARED_FULL_CONTEXT.load( File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1678, in load self._prepare_weight_fp8(wrapper, None, gpu_experts_mask, File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1181, in _prepare_weight_fp8 submit_write_expert(cpu_expert_ids[0], 0) File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1171, in submit_write_expert wrapper.submit_write_weight_scale_to_buffer( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'LlamafileMoEWrapper' object has no attribute 'submit_write_weight_scale_to_buffer' [2026-07-31 14:22:09] SIGQUIT received. signum=None, frame=None. It usually means one child failed. 已杀死``` ### Others 看起来是sglang没有针对纯avx2进行优化。请问我要切换到sglang的那个版本可以兼容纯avx2的CPU？

bsakaguchi commented 2026-10-29 14:50:50 +00:00

Author

Owner

Copy link

@ebarkhordar commented on GitHub (Jul 31, 2026):

@mrgaolei sorry, that was written for the maintainers. Here it is in plain terms.

Nothing is wrong with your setup. The crash is a gap in the code. The option you turned on needs a function that exists on only one of the three CPU backends, and the backend you are running does not have it. The commit history shows that function being added to one backend and never added to the other two, so it looks like something that was never built rather than something deliberately left out. No maintainer has labelled it a bug or a known limitation yet, so I cannot tell you which of the two it will be called.

It is also not really about AVX2. Choosing the AMX backend on an AMX machine hits the same crash, so different hardware, or a different sglang version, would not help.

Whether AVX2 can run MiniMax or DeepSeek V4-Flash from GGUF at all, I cannot answer. I have no machine like yours and no copy of those weights, so I never ran either model. That part is a question for the maintainers.

One thing to try: drop the --kt-gpu-prefill-token-threshold 500 flag. That flag is what switches on the code path that crashes, and it is off by default. I have not tested it, so treat it as a suggestion rather than a fix.

The rest below is the detail for the maintainers.

---

Tree: ktransformers a8062bf with the third_party/sglang submodule at its pinned 1e098a7, which is the checkout your line numbers land in: kt_ep_wrapper.py:1171 is wrapper.submit_write_weight_scale_to_buffer( and :1678 is self._prepare_weight_fp8(wrapper, None, gpu_experts_mask,. None of it was run end to end here (no AVX2-only host with 2x4090 and no M2.7 weights), so everything below is read from source rather than run.

The method exists on exactly one wrapper. Parsing the class bodies with ast, own methods plus everything reachable through the bases:

class	has submit_write_weight_scale_to_buffer
NativeMoEWrapper (kt-kernel/python/utils/amx.py:549)	yes
AMXMoEWrapper (kt-kernel/python/utils/amx.py:248)	no
LlamafileMoEWrapper (kt-kernel/python/utils/llamafile.py:21)	no
BaseMoEWrapper (kt-kernel/python/experts_base.py:227)	no

None of those modules defines __getattr__ or sets the attribute dynamically, so attribute resolution is what the class bodies say.

KTMoEWrapper picks the backend at kt-kernel/python/experts.py:340-354: AMXINT4/AMXINT8 give AMXMoEWrapper, LLAMAFILE gives LlamafileMoEWrapper, and only RAWINT4, FP8, BF16, FP8_PERCHANNEL, GPTQ_INT4, SYCL_GPTQ_INT4, MXFP4, MXFP8 give NativeMoEWrapper.

So this reads as a --kt-method limitation rather than an AVX2 one, which matters for your question at the end. --kt-method AMXINT4 on an AMX-capable CPU would hit the same AttributeError on this path, because AMXMoEWrapper is missing the same two methods. Changing sglang version alone is unlikely to move it.

It is also not specific to the fp8 branch you landed in. SharedFullContext.load() dispatches at kt_ep_wrapper.py:1662-1688, and all four concrete preparers call submit_write_weight_scale_to_buffer: _prepare_weight_int4 at :1004, _prepare_weight_fp8 at :1171, _prepare_weight_fp8_channel at :1419, _prepare_weight_bf16 at :1571. The mxfp4 and mxfp8 branches delegate into _prepare_weight_fp8 at :1247 and :1284. Every branch of that method needs the Native-only API.

Possible workaround, untested here. The entry gate is the flag you set: self.gpu_prefill_token_threshold = kt_config.gpu_prefill_token_threshold or 0 (:2514), and _build_full_context is reached only when self.gpu_prefill_token_threshold > 0 and num_tokens >= self.gpu_prefill_token_threshold (:2955-2956). server_args.kt_gpu_prefill_token_threshold defaults to None (server_args.py:732), so dropping --kt-gpu-prefill-token-threshold 500 should keep you off this path entirely. I have not run it, so treat it as a suggestion.

History, in case it helps decide whether this is an oversight or an intended limit. On an unshallowed clone, git log -G 'submit_write_weight_scale_to_buffer' across the whole repo returns exactly one commit: fcf8882 (#1656, "Add avx-based kimi-k2 support"). It added both methods to utils/amx.py and touched experts.py, and it did not touch utils/llamafile.py. Nothing since removes them from anywhere. That reads as an API built for the Native path and never extended to the other two wrappers, rather than one deliberately withheld from them, though a maintainer would know better than the log does.

If that reading is right, rejecting a non-Native --kt-method combined with --kt-gpu-prefill-token-threshold > 0 at startup would turn this into an immediate message instead of a crash partway through weight loading. Whether that guard or implementing the two methods on the other wrappers is the better fix is your call.

<!-- gh-comment-id:5146208615 --> @ebarkhordar commented on GitHub (Jul 31, 2026): @mrgaolei sorry, that was written for the maintainers. Here it is in plain terms. Nothing is wrong with your setup. The crash is a gap in the code. The option you turned on needs a function that exists on only one of the three CPU backends, and the backend you are running does not have it. The commit history shows that function being added to one backend and never added to the other two, so it looks like something that was never built rather than something deliberately left out. No maintainer has labelled it a bug or a known limitation yet, so I cannot tell you which of the two it will be called. It is also not really about AVX2. Choosing the AMX backend on an AMX machine hits the same crash, so different hardware, or a different sglang version, would not help. Whether AVX2 can run MiniMax or DeepSeek V4-Flash from GGUF at all, I cannot answer. I have no machine like yours and no copy of those weights, so I never ran either model. That part is a question for the maintainers. One thing to try: drop the `--kt-gpu-prefill-token-threshold 500` flag. That flag is what switches on the code path that crashes, and it is off by default. I have not tested it, so treat it as a suggestion rather than a fix. The rest below is the detail for the maintainers. --- Tree: ktransformers `a8062bf` with the `third_party/sglang` submodule at its pinned `1e098a7`, which is the checkout your line numbers land in: `kt_ep_wrapper.py:1171` is `wrapper.submit_write_weight_scale_to_buffer(` and `:1678` is `self._prepare_weight_fp8(wrapper, None, gpu_experts_mask,`. None of it was run end to end here (no AVX2-only host with 2x4090 and no M2.7 weights), so everything below is read from source rather than run. **The method exists on exactly one wrapper.** Parsing the class bodies with `ast`, own methods plus everything reachable through the bases: | class | has `submit_write_weight_scale_to_buffer` | |---|---| | `NativeMoEWrapper` (`kt-kernel/python/utils/amx.py:549`) | yes | | `AMXMoEWrapper` (`kt-kernel/python/utils/amx.py:248`) | no | | `LlamafileMoEWrapper` (`kt-kernel/python/utils/llamafile.py:21`) | no | | `BaseMoEWrapper` (`kt-kernel/python/experts_base.py:227`) | no | None of those modules defines `__getattr__` or sets the attribute dynamically, so attribute resolution is what the class bodies say. `KTMoEWrapper` picks the backend at `kt-kernel/python/experts.py:340-354`: `AMXINT4`/`AMXINT8` give `AMXMoEWrapper`, `LLAMAFILE` gives `LlamafileMoEWrapper`, and only `RAWINT4`, `FP8`, `BF16`, `FP8_PERCHANNEL`, `GPTQ_INT4`, `SYCL_GPTQ_INT4`, `MXFP4`, `MXFP8` give `NativeMoEWrapper`. **So this reads as a `--kt-method` limitation rather than an AVX2 one**, which matters for your question at the end. `--kt-method AMXINT4` on an AMX-capable CPU would hit the same `AttributeError` on this path, because `AMXMoEWrapper` is missing the same two methods. Changing sglang version alone is unlikely to move it. **It is also not specific to the fp8 branch you landed in.** `SharedFullContext.load()` dispatches at `kt_ep_wrapper.py:1662-1688`, and all four concrete preparers call `submit_write_weight_scale_to_buffer`: `_prepare_weight_int4` at `:1004`, `_prepare_weight_fp8` at `:1171`, `_prepare_weight_fp8_channel` at `:1419`, `_prepare_weight_bf16` at `:1571`. The mxfp4 and mxfp8 branches delegate into `_prepare_weight_fp8` at `:1247` and `:1284`. Every branch of that method needs the Native-only API. **Possible workaround, untested here.** The entry gate is the flag you set: `self.gpu_prefill_token_threshold = kt_config.gpu_prefill_token_threshold or 0` (`:2514`), and `_build_full_context` is reached only when `self.gpu_prefill_token_threshold > 0 and num_tokens >= self.gpu_prefill_token_threshold` (`:2955-2956`). `server_args.kt_gpu_prefill_token_threshold` defaults to `None` (`server_args.py:732`), so dropping `--kt-gpu-prefill-token-threshold 500` should keep you off this path entirely. I have not run it, so treat it as a suggestion. **History, in case it helps decide whether this is an oversight or an intended limit.** On an unshallowed clone, `git log -G 'submit_write_weight_scale_to_buffer'` across the whole repo returns exactly one commit: `fcf8882` (#1656, "Add avx-based kimi-k2 support"). It added both methods to `utils/amx.py` and touched `experts.py`, and it did not touch `utils/llamafile.py`. Nothing since removes them from anywhere. That reads as an API built for the Native path and never extended to the other two wrappers, rather than one deliberately withheld from them, though a maintainer would know better than the log does. If that reading is right, rejecting a non-Native `--kt-method` combined with `--kt-gpu-prefill-token-threshold > 0` at startup would turn this into an immediate message instead of a crash partway through weight loading. Whether that guard or implementing the two methods on the other wrappers is the better fix is your call.

bsakaguchi commented 2026-10-29 14:50:50 +00:00

Author

Owner

Copy link

@mrgaolei commented on GitHub (Aug 1, 2026):

I could not reproduce this end to end (no AVX2-only host with 2x4090 and no M2.7 weights here), so everything below is read from source rather than run. Tree: ktransformers a8062bf with the third_party/sglang submodule at its pinned 1e098a7, which is the checkout your line numbers land in: kt_ep_wrapper.py:1171 is wrapper.submit_write_weight_scale_to_buffer( and :1678 is self._prepare_weight_fp8(wrapper, None, gpu_experts_mask,.

The method exists on exactly one wrapper. Parsing the class bodies with ast, own methods plus everything reachable through the bases:

class has submit_write_weight_scale_to_buffer
NativeMoEWrapper (kt-kernel/python/utils/amx.py:549) yes
AMXMoEWrapper (kt-kernel/python/utils/amx.py:248) no
LlamafileMoEWrapper (kt-kernel/python/utils/llamafile.py:21) no
BaseMoEWrapper (kt-kernel/python/experts_base.py:227) no
None of those modules defines __getattr__ or sets the attribute dynamically, so attribute resolution is what the class bodies say.

KTMoEWrapper picks the backend at kt-kernel/python/experts.py:340-354: AMXINT4/AMXINT8 give AMXMoEWrapper, LLAMAFILE gives LlamafileMoEWrapper, and only RAWINT4, FP8, BF16, FP8_PERCHANNEL, GPTQ_INT4, SYCL_GPTQ_INT4, MXFP4, MXFP8 give NativeMoEWrapper.

So this reads as a --kt-method limitation rather than an AVX2 one, which matters for your question at the end. --kt-method AMXINT4 on an AMX-capable CPU would hit the same AttributeError on this path, because AMXMoEWrapper is missing the same two methods. Changing sglang version alone is unlikely to move it.

It is also not specific to the fp8 branch you landed in. SharedFullContext.load() dispatches at kt_ep_wrapper.py:1662-1688, and all four concrete preparers call submit_write_weight_scale_to_buffer: _prepare_weight_int4 at :1004, _prepare_weight_fp8 at :1171, _prepare_weight_fp8_channel at :1419, _prepare_weight_bf16 at :1571. The mxfp4 and mxfp8 branches delegate into _prepare_weight_fp8 at :1247 and :1284. Every branch of that method needs the Native-only API.

Possible workaround, untested here. The entry gate is the flag you set: self.gpu_prefill_token_threshold = kt_config.gpu_prefill_token_threshold or 0 (:2514), and _build_full_context is reached only when self.gpu_prefill_token_threshold > 0 and num_tokens >= self.gpu_prefill_token_threshold (:2955-2956). server_args.kt_gpu_prefill_token_threshold defaults to None (server_args.py:732), so dropping --kt-gpu-prefill-token-threshold 500 should keep you off this path entirely. I have not run it, so treat it as a suggestion.

History, in case it helps decide whether this is an oversight or an intended limit. On an unshallowed clone, git log -G 'submit_write_weight_scale_to_buffer' across the whole repo returns exactly one commit: fcf8882 (#1656, "Add avx-based kimi-k2 support"). It added both methods to utils/amx.py and touched experts.py, and it did not touch utils/llamafile.py. Nothing since removes them from anywhere. That reads as an API built for the Native path and never extended to the other two wrappers, rather than one deliberately withheld from them, though a maintainer would know better than the log does.

If that reading is right, rejecting a non-Native --kt-method combined with --kt-gpu-prefill-token-threshold > 0 at startup would turn this into an immediate message instead of a crash partway through weight loading. Whether that guard or implementing the two methods on the other wrappers is the better fix is your call.

I apologize, your reply was too technical for me to understand. I would like to ask if avx2 can run MiniMax and DeepSeek V4-Flash via gguf? Is this a bug, or a known unsupported feature?

<!-- gh-comment-id:5150781100 --> @mrgaolei commented on GitHub (Aug 1, 2026): > I could not reproduce this end to end (no AVX2-only host with 2x4090 and no M2.7 weights here), so everything below is read from source rather than run. Tree: ktransformers `a8062bf` with the `third_party/sglang` submodule at its pinned `1e098a7`, which is the checkout your line numbers land in: `kt_ep_wrapper.py:1171` is `wrapper.submit_write_weight_scale_to_buffer(` and `:1678` is `self._prepare_weight_fp8(wrapper, None, gpu_experts_mask,`. > > **The method exists on exactly one wrapper.** Parsing the class bodies with `ast`, own methods plus everything reachable through the bases: > > class has `submit_write_weight_scale_to_buffer` > `NativeMoEWrapper` (`kt-kernel/python/utils/amx.py:549`) yes > `AMXMoEWrapper` (`kt-kernel/python/utils/amx.py:248`) no > `LlamafileMoEWrapper` (`kt-kernel/python/utils/llamafile.py:21`) no > `BaseMoEWrapper` (`kt-kernel/python/experts_base.py:227`) no > None of those modules defines `__getattr__` or sets the attribute dynamically, so attribute resolution is what the class bodies say. > > `KTMoEWrapper` picks the backend at `kt-kernel/python/experts.py:340-354`: `AMXINT4`/`AMXINT8` give `AMXMoEWrapper`, `LLAMAFILE` gives `LlamafileMoEWrapper`, and only `RAWINT4`, `FP8`, `BF16`, `FP8_PERCHANNEL`, `GPTQ_INT4`, `SYCL_GPTQ_INT4`, `MXFP4`, `MXFP8` give `NativeMoEWrapper`. > > **So this reads as a `--kt-method` limitation rather than an AVX2 one**, which matters for your question at the end. `--kt-method AMXINT4` on an AMX-capable CPU would hit the same `AttributeError` on this path, because `AMXMoEWrapper` is missing the same two methods. Changing sglang version alone is unlikely to move it. > > **It is also not specific to the fp8 branch you landed in.** `SharedFullContext.load()` dispatches at `kt_ep_wrapper.py:1662-1688`, and all four concrete preparers call `submit_write_weight_scale_to_buffer`: `_prepare_weight_int4` at `:1004`, `_prepare_weight_fp8` at `:1171`, `_prepare_weight_fp8_channel` at `:1419`, `_prepare_weight_bf16` at `:1571`. The mxfp4 and mxfp8 branches delegate into `_prepare_weight_fp8` at `:1247` and `:1284`. Every branch of that method needs the Native-only API. > > **Possible workaround, untested here.** The entry gate is the flag you set: `self.gpu_prefill_token_threshold = kt_config.gpu_prefill_token_threshold or 0` (`:2514`), and `_build_full_context` is reached only when `self.gpu_prefill_token_threshold > 0 and num_tokens >= self.gpu_prefill_token_threshold` (`:2955-2956`). `server_args.kt_gpu_prefill_token_threshold` defaults to `None` (`server_args.py:732`), so dropping `--kt-gpu-prefill-token-threshold 500` should keep you off this path entirely. I have not run it, so treat it as a suggestion. > > **History, in case it helps decide whether this is an oversight or an intended limit.** On an unshallowed clone, `git log -G 'submit_write_weight_scale_to_buffer'` across the whole repo returns exactly one commit: `fcf8882` ([#1656](https://github.com/kvcache-ai/ktransformers/pull/1656), "Add avx-based kimi-k2 support"). It added both methods to `utils/amx.py` and touched `experts.py`, and it did not touch `utils/llamafile.py`. Nothing since removes them from anywhere. That reads as an API built for the Native path and never extended to the other two wrappers, rather than one deliberately withheld from them, though a maintainer would know better than the log does. > > If that reading is right, rejecting a non-Native `--kt-method` combined with `--kt-gpu-prefill-token-threshold > 0` at startup would turn this into an immediate message instead of a crash partway through weight loading. Whether that guard or implementing the two methods on the other wrappers is the better fix is your call. I apologize, your reply was too technical for me to understand. I would like to ask if avx2 can run MiniMax and DeepSeek V4-Flash via gguf? Is this a bug, or a known unsupported feature?

bsakaguchi commented 2026-10-29 14:50:51 +00:00

Author

Owner

Copy link

@youyoulyz commented on GitHub (Aug 5, 2026):

We reproduced this end-to-end and have findings that may help close it.

Environment: 2× RTX 3080 20GB, Qwen3-30B-A3B, --kt-method LLAMAFILE (Q4_K_M GGUF via --kt-weight-path), kt 0.6.4.

Reproduction: the first request with ≥ --kt-gpu-prefill-token-threshold tokens crashes the scheduler with the same AttributeError in _prepare_weight_bf16 → submit_write_expert → wrapper.submit_write_weight_scale_to_buffer. The crash reproduces with or without --kt-enable-dynamic-expert-update — the threshold flag alone is sufficient; the dynamic-update flag only gates the promote step after the fallback compute.

Still present on latest main: verified end-to-end on ktransformers e03cf1c + sglang submodule d164eea75 (2026-08-05). The capability gate _full_gpu_fallback_supported still checks only layer attributes (w13_weight / w13_weight_packed / _v4_tk_path), never the wrapper's capability, so a bf16/LLAMAFILE layer passes the gate and hits the missing method.

On PR #2111: we applied the BaseMoEWrapper move locally and re-ran. The error changes from AttributeError to NotImplementedError: write_weight_scale_to_buffer_task and the scheduler still SIGQUITs. The compiled kt_kernel_ext.moe.MOE in the wheel does not expose write_weight_scale_to_buffer_task for the LLAMAFILE backend (hasattr(MOE, "write_weight_scale_to_buffer_task") is False), so relocating the Python method is not sufficient for LLAMAFILE — the fix needs either the C++-side task on the llamafile backend or an explicit startup-time gate rejecting non-Native --kt-method combined with --kt-gpu-prefill-token-threshold > 0.

Happy to share the full traceback set and the minimal repro script if useful.

<!-- gh-comment-id:5188302906 --> @youyoulyz commented on GitHub (Aug 5, 2026): We reproduced this end-to-end and have findings that may help close it. **Environment**: 2× RTX 3080 20GB, Qwen3-30B-A3B, `--kt-method LLAMAFILE` (Q4_K_M GGUF via `--kt-weight-path`), kt 0.6.4. **Reproduction**: the first request with ≥ `--kt-gpu-prefill-token-threshold` tokens crashes the scheduler with the same `AttributeError` in `_prepare_weight_bf16` → `submit_write_expert` → `wrapper.submit_write_weight_scale_to_buffer`. The crash reproduces **with or without** `--kt-enable-dynamic-expert-update` — the threshold flag alone is sufficient; the dynamic-update flag only gates the promote step *after* the fallback compute. **Still present on latest main**: verified end-to-end on ktransformers `e03cf1c` + sglang submodule `d164eea75` (2026-08-05). The capability gate `_full_gpu_fallback_supported` still checks only layer attributes (`w13_weight` / `w13_weight_packed` / `_v4_tk_path`), never the wrapper's capability, so a bf16/LLAMAFILE layer passes the gate and hits the missing method. **On PR #2111**: we applied the BaseMoEWrapper move locally and re-ran. The error changes from `AttributeError` to `NotImplementedError: write_weight_scale_to_buffer_task` and the scheduler still SIGQUITs. The compiled `kt_kernel_ext.moe.MOE` in the wheel does not expose `write_weight_scale_to_buffer_task` for the LLAMAFILE backend (`hasattr(MOE, "write_weight_scale_to_buffer_task")` is False), so relocating the Python method is not sufficient for LLAMAFILE — the fix needs either the C++-side task on the llamafile backend or an explicit startup-time gate rejecting non-Native `--kt-method` combined with `--kt-gpu-prefill-token-threshold > 0`. Happy to share the full traceback set and the minimal repro script if useful.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#1281

No description provided.