[GH-ISSUE #1714] Cannot find any model weights /workspace/sglang/python/sglang/srt/model_loader/loader.py #1094

New issue

Open

opened 2026-10-29 14:48:25 +00:00 by bsakaguchi · 10 comments

bsakaguchi commented 2026-10-29 14:48:25 +00:00

Owner

Copy link

Originally created by @withyou971 on GitHub (Dec 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1714

Originally assigned to: @SkqLiao on GitHub.

Reminder

• I have read the above rules and searched the existing issues.

System Info

[2025-12-14 07:09:52] Scheduler hit an exception: Traceback (most recent call last):
File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2674, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 311, in init
self.tp_worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in init
self._model_runner = ModelRunner(
^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 323, in init
self.initialize(min_per_gpu_memory)
File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 404, in initialize
self.load_model()
File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 758, in load_model
self.model = get_model(
^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/init.py", line 28, in get_model
return loader.load_model(
^^^^^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 600, in load_model
self.load_weights_and_postprocess(
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 608, in load_weights_and_postprocess
model.load_weights(weights)
File "/workspace/sglang/python/sglang/srt/models/qwen3_moe.py", line 812, in load_weights
for name, loaded_weight in weights:
^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 498, in _get_all_weights
yield from self._get_weights_iterator(primary_weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 447, in _get_weights_iterator
hf_folder, hf_weights_files, use_safetensors = self._prepare_weights(
^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 436, in _prepare_weights
raise RuntimeError(
RuntimeError: Cannot find any model weights with /mnt/model/Qwen3-30B-A3B ['*.safetensors', '*.bin'] []

Reproduction

approachingai/ktransformers                              v0.4.3-cu128
wsl ubuntu 2204

Others

No response

Originally created by @withyou971 on GitHub (Dec 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1714 Originally assigned to: @SkqLiao on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info [2025-12-14 07:09:52] Scheduler hit an exception: Traceback (most recent call last): File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2674, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 311, in __init__ self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in __init__ self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 323, in __init__ self.initialize(min_per_gpu_memory) File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 404, in initialize self.load_model() File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 758, in load_model self.model = get_model( ^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/__init__.py", line 28, in get_model return loader.load_model( ^^^^^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 600, in load_model self.load_weights_and_postprocess( File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 608, in load_weights_and_postprocess model.load_weights(weights) File "/workspace/sglang/python/sglang/srt/models/qwen3_moe.py", line 812, in load_weights for name, loaded_weight in weights: ^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 498, in _get_all_weights yield from self._get_weights_iterator(primary_weights) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 447, in _get_weights_iterator hf_folder, hf_weights_files, use_safetensors = self._prepare_weights( ^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 436, in _prepare_weights raise RuntimeError( RuntimeError: Cannot find any model weights with `/mnt/model/Qwen3-30B-A3B` `['*.safetensors', '*.bin']` `[]` ### Reproduction ```text approachingai/ktransformers v0.4.3-cu128 wsl ubuntu 2204 ``` ### Others _No response_

bsakaguchi added the

bug

label 2026-10-29 14:48:25 +00:00

bsakaguchi commented 2026-10-29 14:48:26 +00:00

Author

Owner

Copy link

@KMSorSMS commented on GitHub (Dec 15, 2025):

Seems like you didn't download the model weights (Qwen3-30B-A3B) in the right place -> /mnt/model/Qwen3-30B-A3B. Check your environment (the Docker won't contain the weights)

<!-- gh-comment-id:3655988557 --> @KMSorSMS commented on GitHub (Dec 15, 2025): Seems like you didn't download the model weights (Qwen3-30B-A3B) in the right place -> /mnt/model/Qwen3-30B-A3B. Check your environment (the Docker won't contain the weights)

bsakaguchi commented 2026-10-29 14:48:26 +00:00

Author

Owner

Copy link

@withyou971 commented on GitHub (Dec 16, 2025):

total 59681116
-rwxrwxrwx 1 wenda wenda 61095802400 Jun 21 10:48 Qwen3-30B-A3B-Q4_K_M.gguf
-rw-r--r-- 1 root root 16798 Jun 28 23:51 README.md
-rw-rw-rw- 1 wenda wenda 707 May 22 2025 added_tokens.json
-rw-r--r-- 1 root root 963 Jun 28 23:51 config.json
-rw-r--r-- 1 root root 73 Jun 28 23:51 configuration.json
-rw-r--r-- 1 root root 239 Jun 28 23:51 generation_config.json
-rw-rw-rw- 1 wenda wenda 1671853 May 2 2025 merges.txt
-rw-r--r-- 1 root root 1699758 Jun 28 23:51 model.safetensors.index.json
-rw-rw-rw- 1 wenda wenda 427 May 2 2025 quantize_config.json
-rw-rw-rw- 1 wenda wenda 613 May 22 2025 special_tokens_map.json
-rw-r--r-- 1 root root 11422654 Jun 28 23:51 tokenizer.json
-rw-r--r-- 1 root root 9732 Jun 28 23:51 tokenizer_config.json
-rw-r--r-- 1 root root 2776833 Jun 28 23:51 vocab.json
wenda@DESKTOP-A7UUURG:/mnt/model/Qwen3-30B-A3B$

<!-- gh-comment-id:3660490151 --> @withyou971 commented on GitHub (Dec 16, 2025): total 59681116 -rwxrwxrwx 1 wenda wenda 61095802400 Jun 21 10:48 Qwen3-30B-A3B-Q4_K_M.gguf -rw-r--r-- 1 root root 16798 Jun 28 23:51 README.md -rw-rw-rw- 1 wenda wenda 707 May 22 2025 added_tokens.json -rw-r--r-- 1 root root 963 Jun 28 23:51 config.json -rw-r--r-- 1 root root 73 Jun 28 23:51 configuration.json -rw-r--r-- 1 root root 239 Jun 28 23:51 generation_config.json -rw-rw-rw- 1 wenda wenda 1671853 May 2 2025 merges.txt -rw-r--r-- 1 root root 1699758 Jun 28 23:51 model.safetensors.index.json -rw-rw-rw- 1 wenda wenda 427 May 2 2025 quantize_config.json -rw-rw-rw- 1 wenda wenda 613 May 22 2025 special_tokens_map.json -rw-r--r-- 1 root root 11422654 Jun 28 23:51 tokenizer.json -rw-r--r-- 1 root root 9732 Jun 28 23:51 tokenizer_config.json -rw-r--r-- 1 root root 2776833 Jun 28 23:51 vocab.json wenda@DESKTOP-A7UUURG:/mnt/model/Qwen3-30B-A3B$

bsakaguchi commented 2026-10-29 14:48:26 +00:00

Author

Owner

Copy link

@KMSorSMS commented on GitHub (Dec 16, 2025):

['*.safetensors', '*.bin'] [] you should pass the safetensors weight, not gguf.

<!-- gh-comment-id:3660549879 --> @KMSorSMS commented on GitHub (Dec 16, 2025): `['*.safetensors', '*.bin'] []` you should pass the safetensors weight, not gguf.

bsakaguchi commented 2026-10-29 14:48:27 +00:00

Author

Owner

Copy link

@withyou971 commented on GitHub (Dec 16, 2025):

In the readme it show the example --kt-method LLAMAFILE will be gguf type

<!-- gh-comment-id:3660882111 --> @withyou971 commented on GitHub (Dec 16, 2025): In the readme it show the example --kt-method LLAMAFILE will be gguf type

bsakaguchi commented 2026-10-29 14:48:27 +00:00

Author

Owner

Copy link

@KMSorSMS commented on GitHub (Dec 17, 2025):

@withyou971 What command are you using? Are you passing the --kt-method LLAMAFILE argument? Please provide more details so we can help troubleshoot. I believe the default container startup command does not include --kt-method LLAMAFILE.

<!-- gh-comment-id:3663328227 --> @KMSorSMS commented on GitHub (Dec 17, 2025): @withyou971 What command are you using? Are you passing the --kt-method LLAMAFILE argument? Please provide more details so we can help troubleshoot. I believe the default container startup command does not include --kt-method LLAMAFILE.

bsakaguchi commented 2026-10-29 14:48:27 +00:00

Author

Owner

Copy link

@withyou971 commented on GitHub (Dec 18, 2025):

python -m sglang.launch_server
--host 0.0.0.0
--port 10002
--model /mnt/model/Qwen3-235B-A22B
--trust-remote-code
--mem-fraction-static 0.92
--chunked-prefill-size 4096
--served-model-name Qwen3-235B-A22B
--enable-mixed-chunk
--kt-method LLAMAFILE
--kt-weight-path /mnt/model/Qwen3-235B-A22B-gguf
--kt-cpuinfer 16
--kt-num-gpu-experts 32
--kt-threadpool-count 1
--kt-max-deferred-experts-per-token 2 --tp 1 --log-level debug

<!-- gh-comment-id:3670035191 --> @withyou971 commented on GitHub (Dec 18, 2025): python -m sglang.launch_server \ --host 0.0.0.0 \ --port 10002 \ --model /mnt/model/Qwen3-235B-A22B \ --trust-remote-code \ --mem-fraction-static 0.92 \ --chunked-prefill-size 4096 \ --served-model-name Qwen3-235B-A22B \ --enable-mixed-chunk \ --kt-method LLAMAFILE \ --kt-weight-path /mnt/model/Qwen3-235B-A22B-gguf \ --kt-cpuinfer 16 \ --kt-num-gpu-experts 32 \ --kt-threadpool-count 1 \ --kt-max-deferred-experts-per-token 2 --tp 1 --log-level debug

bsakaguchi commented 2026-10-29 14:48:27 +00:00

Author

Owner

Copy link

@SkqLiao commented on GitHub (Dec 18, 2025):

python -m sglang.launch_server --host 0.0.0.0 --port 10002 --model /mnt/model/Qwen3-235B-A22B --trust-remote-code --mem-fraction-static 0.92 --chunked-prefill-size 4096 --served-model-name Qwen3-235B-A22B --enable-mixed-chunk --kt-method LLAMAFILE --kt-weight-path /mnt/model/Qwen3-235B-A22B-gguf --kt-cpuinfer 16 --kt-num-gpu-experts 32 --kt-threadpool-count 1 --kt-max-deferred-experts-per-token 2 --tp 1 --log-level debug

This is a valid startup command. You need to ensure that the paths /mnt/model/Qwen3-235B-A22B and /mnt/model/Qwen3-235B-A22B-gguf are correct and contain weights.

<!-- gh-comment-id:3670078333 --> @SkqLiao commented on GitHub (Dec 18, 2025): > python -m sglang.launch_server --host 0.0.0.0 --port 10002 --model /mnt/model/Qwen3-235B-A22B --trust-remote-code --mem-fraction-static 0.92 --chunked-prefill-size 4096 --served-model-name Qwen3-235B-A22B --enable-mixed-chunk --kt-method LLAMAFILE --kt-weight-path /mnt/model/Qwen3-235B-A22B-gguf --kt-cpuinfer 16 --kt-num-gpu-experts 32 --kt-threadpool-count 1 --kt-max-deferred-experts-per-token 2 --tp 1 --log-level debug This is a valid startup command. You need to ensure that the paths /mnt/model/Qwen3-235B-A22B and /mnt/model/Qwen3-235B-A22B-gguf are correct and contain weights.

bsakaguchi commented 2026-10-29 14:48:27 +00:00

Author

Owner

Copy link

@withyou971 commented on GitHub (Dec 18, 2025):

root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# ls -l
total 18816
-rw-rw-rw- 1 wenda wenda 11343 Dec 14 11:33 LICENSE
-rw-rw-rw- 1 wenda wenda 16782 Dec 14 11:33 README.md
-rw-rw-rw- 1 wenda wenda 965 Dec 14 11:33 config.json
-rw-rw-rw- 1 wenda wenda 64 Dec 14 11:33 configuration.json
-rw-rw-rw- 1 wenda wenda 239 Dec 14 11:33 generation_config.json
-rw-rw-rw- 1 wenda wenda 1671853 Dec 14 11:33 merges.txt
-rw-rw-rw- 1 wenda wenda 3332207 Dec 14 11:33 model.safetensors.index.json
-rw-rw-rw- 1 wenda wenda 11422654 Dec 14 11:33 tokenizer.json
-rw-rw-rw- 1 wenda wenda 9732 Dec 14 11:33 tokenizer_config.json
-rw-rw-rw- 1 wenda wenda 2776833 Dec 14 11:33 vocab.json
root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# cd ../Qwen3-235B-A22B-gguf
root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B-gguf# ls -l
total 138822396
-rwxrwxrwx 1 wenda wenda 49944699456 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf
-rwxrwxrwx 1 wenda wenda 49930349824 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00002-of-00003.gguf
-rwxrwxrwx 1 wenda wenda 42279026016 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00003-of-00003.gguf

<!-- gh-comment-id:3670399837 --> @withyou971 commented on GitHub (Dec 18, 2025): root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# ls -l total 18816 -rw-rw-rw- 1 wenda wenda 11343 Dec 14 11:33 LICENSE -rw-rw-rw- 1 wenda wenda 16782 Dec 14 11:33 README.md -rw-rw-rw- 1 wenda wenda 965 Dec 14 11:33 config.json -rw-rw-rw- 1 wenda wenda 64 Dec 14 11:33 configuration.json -rw-rw-rw- 1 wenda wenda 239 Dec 14 11:33 generation_config.json -rw-rw-rw- 1 wenda wenda 1671853 Dec 14 11:33 merges.txt -rw-rw-rw- 1 wenda wenda 3332207 Dec 14 11:33 model.safetensors.index.json -rw-rw-rw- 1 wenda wenda 11422654 Dec 14 11:33 tokenizer.json -rw-rw-rw- 1 wenda wenda 9732 Dec 14 11:33 tokenizer_config.json -rw-rw-rw- 1 wenda wenda 2776833 Dec 14 11:33 vocab.json root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# cd ../Qwen3-235B-A22B-gguf root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B-gguf# ls -l total 138822396 -rwxrwxrwx 1 wenda wenda 49944699456 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf -rwxrwxrwx 1 wenda wenda 49930349824 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00002-of-00003.gguf -rwxrwxrwx 1 wenda wenda 42279026016 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00003-of-00003.gguf

bsakaguchi commented 2026-10-29 14:48:27 +00:00

Author

Owner

Copy link

@SkqLiao commented on GitHub (Dec 18, 2025):

root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# ls -l total 18816 -rw-rw-rw- 1 wenda wenda 11343 Dec 14 11:33 LICENSE -rw-rw-rw- 1 wenda wenda 16782 Dec 14 11:33 README.md -rw-rw-rw- 1 wenda wenda 965 Dec 14 11:33 config.json -rw-rw-rw- 1 wenda wenda 64 Dec 14 11:33 configuration.json -rw-rw-rw- 1 wenda wenda 239 Dec 14 11:33 generation_config.json -rw-rw-rw- 1 wenda wenda 1671853 Dec 14 11:33 merges.txt -rw-rw-rw- 1 wenda wenda 3332207 Dec 14 11:33 model.safetensors.index.json -rw-rw-rw- 1 wenda wenda 11422654 Dec 14 11:33 tokenizer.json -rw-rw-rw- 1 wenda wenda 9732 Dec 14 11:33 tokenizer_config.json -rw-rw-rw- 1 wenda wenda 2776833 Dec 14 11:33 vocab.json root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# cd ../Qwen3-235B-A22B-gguf root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B-gguf# ls -l total 138822396 -rwxrwxrwx 1 wenda wenda 49944699456 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf -rwxrwxrwx 1 wenda wenda 49930349824 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00002-of-00003.gguf -rwxrwxrwx 1 wenda wenda 42279026016 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00003-of-00003.gguf

/mnt/model/Qwen3-235B-A22B appears to have no weights. The weights are in the form of model-00001-of-00118.safetensors, which you can check at huggingface. /mnt/model/Qwen3-235B-A22B-gguf seems to be fine.

<!-- gh-comment-id:3670554600 --> @SkqLiao commented on GitHub (Dec 18, 2025): > root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# ls -l total 18816 -rw-rw-rw- 1 wenda wenda 11343 Dec 14 11:33 LICENSE -rw-rw-rw- 1 wenda wenda 16782 Dec 14 11:33 README.md -rw-rw-rw- 1 wenda wenda 965 Dec 14 11:33 config.json -rw-rw-rw- 1 wenda wenda 64 Dec 14 11:33 configuration.json -rw-rw-rw- 1 wenda wenda 239 Dec 14 11:33 generation_config.json -rw-rw-rw- 1 wenda wenda 1671853 Dec 14 11:33 merges.txt -rw-rw-rw- 1 wenda wenda 3332207 Dec 14 11:33 model.safetensors.index.json -rw-rw-rw- 1 wenda wenda 11422654 Dec 14 11:33 tokenizer.json -rw-rw-rw- 1 wenda wenda 9732 Dec 14 11:33 tokenizer_config.json -rw-rw-rw- 1 wenda wenda 2776833 Dec 14 11:33 vocab.json root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B# cd ../Qwen3-235B-A22B-gguf root@DESKTOP-A7UUURG:/mnt/model/Qwen3-235B-A22B-gguf# ls -l total 138822396 -rwxrwxrwx 1 wenda wenda 49944699456 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf -rwxrwxrwx 1 wenda wenda 49930349824 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00002-of-00003.gguf -rwxrwxrwx 1 wenda wenda 42279026016 May 2 2025 Qwen3-235B-A22B-Q4_K_M-00003-of-00003.gguf `/mnt/model/Qwen3-235B-A22B` appears to have no weights. The weights are in the form of `model-00001-of-00118.safetensors`, which you can check at [huggingface](https://huggingface.co/Qwen/Qwen3-235B-A22B/tree/main). `/mnt/model/Qwen3-235B-A22B-gguf` seems to be fine.

bsakaguchi commented 2026-10-29 14:48:27 +00:00

Author

Owner

Copy link

@withyou971 commented on GitHub (Dec 18, 2025):

[GGUFLoader] Loading GGUF files from directory: /mnt/model/Qwen3-235B-A22B-gguf
Loading: Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf
Loading: Qwen3-235B-A22B-Q4_K_M-00002-of-00003.gguf
Loading: Qwen3-235B-A22B-Q4_K_M-00003-of-00003.gguf
[GGUFLoader] Summary:
Files loaded: 3
Total tensors: 1131
Metadata keys: 49
ffn_up_exps, Dtype: Q4_K
ffn_gate_exps, Dtype: Q4_K
ffn_down_exps, Dtype: Q6_K
[LlamafileMoEWrapper] Layer 0 TP configuration:
intermediate_size: 1536
threadpool_count: 1
QK_K: 256
Total blocks: 6
Base blocks per TP: 6
Extra blocks (distributed to first TPs): 0
TP 0: size=1536, offset=0, blocks=6
CPUInfer[0x3b635a00]: Hello
WorkerPool[0x46936bc0] 1 subpools, [numa:threads][0:16]
set_mempolicy: Operation not permitted
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 16 threads
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
hwloc_set_membind_nodeset: Operation not permitted
[LlamafileMoEWrapper] Layer 1 TP configuration:
intermediate_size: 1536
threadpool_count: 1
QK_K: 256

[LlamafileMoEWrapper] Layer 93 TP configuration:
intermediate_size: 1536
threadpool_count: 1
QK_K: 256
Total blocks: 6
Base blocks per TP: 6
Extra blocks (distributed to first TPs): 0
TP 0: size=1536, offset=0, blocks=6
[2025-12-18 07:29:11] Scheduler hit an exception: Traceback (most recent call last):
File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2674, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 311, in init
self.tp_worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in init
self._model_runner = ModelRunner(
^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 323, in init
self.initialize(min_per_gpu_memory)
File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 404, in initialize
self.load_model()
File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 758, in load_model
self.model = get_model(
^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/init.py", line 28, in get_model
return loader.load_model(
^^^^^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 600, in load_model
self.load_weights_and_postprocess(
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 608, in load_weights_and_postprocess
model.load_weights(weights)
File "/workspace/sglang/python/sglang/srt/models/qwen3_moe.py", line 812, in load_weights
for name, loaded_weight in weights:
^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 498, in _get_all_weights
yield from self._get_weights_iterator(primary_weights)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 447, in _get_weights_iterator
hf_folder, hf_weights_files, use_safetensors = self._prepare_weights(
^^^^^^^^^^^^^^^^^^^^^^
File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 436, in _prepare_weights
raise RuntimeError(
RuntimeError: Cannot find any model weights with /mnt/model/Qwen3-235B-A22B ['*.safetensors', '*.bin'] []

[2025-12-18 07:29:11] Received sigquit from a child process. It usually means the child failed.
Killed

<!-- gh-comment-id:3670888351 --> @withyou971 commented on GitHub (Dec 18, 2025): [GGUFLoader] Loading GGUF files from directory: /mnt/model/Qwen3-235B-A22B-gguf Loading: Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf Loading: Qwen3-235B-A22B-Q4_K_M-00002-of-00003.gguf Loading: Qwen3-235B-A22B-Q4_K_M-00003-of-00003.gguf [GGUFLoader] Summary: Files loaded: 3 Total tensors: 1131 Metadata keys: 49 ffn_up_exps, Dtype: Q4_K ffn_gate_exps, Dtype: Q4_K ffn_down_exps, Dtype: Q6_K [LlamafileMoEWrapper] Layer 0 TP configuration: intermediate_size: 1536 threadpool_count: 1 QK_K: 256 Total blocks: 6 Base blocks per TP: 6 Extra blocks (distributed to first TPs): 0 TP 0: size=1536, offset=0, blocks=6 CPUInfer[0x3b635a00]: Hello WorkerPool[0x46936bc0] 1 subpools, [numa:threads][0:16] set_mempolicy: Operation not permitted ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 16 threads hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted hwloc_set_membind_nodeset: Operation not permitted [LlamafileMoEWrapper] Layer 1 TP configuration: intermediate_size: 1536 threadpool_count: 1 QK_K: 256 [LlamafileMoEWrapper] Layer 93 TP configuration: intermediate_size: 1536 threadpool_count: 1 QK_K: 256 Total blocks: 6 Base blocks per TP: 6 Extra blocks (distributed to first TPs): 0 TP 0: size=1536, offset=0, blocks=6 [2025-12-18 07:29:11] Scheduler hit an exception: Traceback (most recent call last): File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2674, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/managers/scheduler.py", line 311, in __init__ self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 237, in __init__ self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 323, in __init__ self.initialize(min_per_gpu_memory) File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 404, in initialize self.load_model() File "/workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 758, in load_model self.model = get_model( ^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/__init__.py", line 28, in get_model return loader.load_model( ^^^^^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 600, in load_model self.load_weights_and_postprocess( File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 608, in load_weights_and_postprocess model.load_weights(weights) File "/workspace/sglang/python/sglang/srt/models/qwen3_moe.py", line 812, in load_weights for name, loaded_weight in weights: ^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 498, in _get_all_weights yield from self._get_weights_iterator(primary_weights) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 447, in _get_weights_iterator hf_folder, hf_weights_files, use_safetensors = self._prepare_weights( ^^^^^^^^^^^^^^^^^^^^^^ File "/workspace/sglang/python/sglang/srt/model_loader/loader.py", line 436, in _prepare_weights raise RuntimeError( RuntimeError: Cannot find any model weights with `/mnt/model/Qwen3-235B-A22B` `['*.safetensors', '*.bin']` `[]` [2025-12-18 07:29:11] Received sigquit from a child process. It usually means the child failed. Killed

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#1094

No description provided.