kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1649] Error what(): no weight source with GLM 4.6-INT8 #1061

New issue
Open
opened 2026-10-29 14:48:05 +00:00 by bsakaguchi · 4 comments
bsakaguchi commented 2026-10-29 14:48:05 +00:00
Owner
Copy link

Originally created by @reasv on GitHub (Nov 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1649

Originally assigned to: @ovowei on GitHub.

Reminder

  • I have read the above rules and searched the existing issues.

System Info

CPU: 2x Intel Xeon 8592+ ES Q2SP (60c per socket)
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB)
OS: Arch Linux (kernel 6.17.8-arch1-1)

SGLang: 0.5.5.post3
kt-kernel: 0.1.0 (cloned from github)

Reproduction

python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 8000 \
  --model /data/models/GLM-4.6-FP8 \
  --trust-remote-code \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 4096 \
  --served-model-name GLM-4.6 \
  --enable-mixed-chunk \
  --kt-method AMXINT8 \
  --kt-weight-path /data/models/GLM-4.6-Channel-INT8 \
  --kt-cpuinfer 120 \
  --kt-threadpool-count 4 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

Error:

[2025-11-27 06:21:20] WARNING server_args.py:1213: Attention backend not explicitly specified. Use flashinfer backend by default.
[2025-11-27 06:21:21] server_args=ServerArgs(model_path='/data/models/GLM-4.6-FP8', tokenizer_path='/data/models/GLM-4.6-FP8', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.92, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=157567588, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='GLM-4.6', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='/data/models/GLM-4.6-Channel-INT8', kt_method='AMXINT8', kt_cpuinfer=120, kt_threadpool_count=4, kt_num_gpu_experts=32, kt_max_deferred_experts_per_token=2, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, decrypted_config_file=None, decrypted_draft_config_file=None)
[2025-11-27 06:21:21] Using default HuggingFace chat template with detected content format: openai
[2025-11-27 06:21:25] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-11-27 06:21:26] Init torch distributed ends. mem usage=0.00 GB
[2025-11-27 06:21:27] Load weight begin. avail mem=94.31 GB
CPUInfer[0x4ad548e0]: Hello
WorkerPool[0x5db02e10] 4 subpools, [numa:threads][0:30] [1:30] [2:30] [3:30]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 2, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 3, 30 threads
Loading safetensors checkpoint shards:   0% Completed | 0/93 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   1% Completed | 1/93 [00:00<00:14,  6.36it/s]
Loading safetensors checkpoint shards:   2% Completed | 2/93 [00:00<00:14,  6.26it/s]
Loading safetensors checkpoint shards:   3% Completed | 3/93 [00:00<00:15,  5.85it/s]
Loading safetensors checkpoint shards:   4% Completed | 4/93 [00:00<00:14,  6.15it/s]
Loading safetensors checkpoint shards:   5% Completed | 5/93 [00:00<00:13,  6.37it/s]
Loading safetensors checkpoint shards:   6% Completed | 6/93 [00:00<00:14,  6.06it/s]
Loading safetensors checkpoint shards:   8% Completed | 7/93 [00:01<00:14,  5.82it/s]
Loading safetensors checkpoint shards:   9% Completed | 8/93 [00:01<00:14,  6.05it/s]
Loading safetensors checkpoint shards:  10% Completed | 9/93 [00:01<00:13,  6.35it/s]
Loading safetensors checkpoint shards:  11% Completed | 10/93 [00:01<00:13,  6.34it/s]
Loading safetensors checkpoint shards:  12% Completed | 11/93 [00:01<00:13,  6.04it/s]
Loading safetensors checkpoint shards:  13% Completed | 12/93 [00:01<00:13,  5.82it/s]
Loading safetensors checkpoint shards:  14% Completed | 13/93 [00:02<00:13,  5.95it/s]
Loading safetensors checkpoint shards:  15% Completed | 14/93 [00:02<00:16,  4.88it/s]
Loading safetensors checkpoint shards:  16% Completed | 15/93 [00:02<00:14,  5.23it/s]
Loading safetensors checkpoint shards:  17% Completed | 16/93 [00:02<00:14,  5.47it/s]
Loading safetensors checkpoint shards:  18% Completed | 17/93 [00:02<00:13,  5.52it/s]
Loading safetensors checkpoint shards:  19% Completed | 18/93 [00:03<00:13,  5.72it/s]
Loading safetensors checkpoint shards:  20% Completed | 19/93 [00:03<00:12,  6.15it/s]
Loading safetensors checkpoint shards:  22% Completed | 20/93 [00:03<00:11,  6.25it/s]
Loading safetensors checkpoint shards:  23% Completed | 21/93 [00:03<00:12,  6.00it/s]
Loading safetensors checkpoint shards:  24% Completed | 22/93 [00:03<00:11,  6.17it/s]
Loading safetensors checkpoint shards:  25% Completed | 23/93 [00:03<00:10,  6.54it/s]
Loading safetensors checkpoint shards:  26% Completed | 24/93 [00:04<00:11,  6.23it/s]
Loading safetensors checkpoint shards:  27% Completed | 25/93 [00:04<00:10,  6.37it/s]
Loading safetensors checkpoint shards:  28% Completed | 26/93 [00:04<00:09,  6.74it/s]
Loading safetensors checkpoint shards:  29% Completed | 27/93 [00:04<00:09,  7.10it/s]
Loading safetensors checkpoint shards:  30% Completed | 28/93 [00:04<00:08,  7.37it/s]
Loading safetensors checkpoint shards:  31% Completed | 29/93 [00:04<00:08,  7.14it/s]
Loading safetensors checkpoint shards:  32% Completed | 30/93 [00:04<00:09,  6.53it/s]
Loading safetensors checkpoint shards:  33% Completed | 31/93 [00:05<00:10,  6.17it/s]
Loading safetensors checkpoint shards:  34% Completed | 32/93 [00:05<00:10,  5.92it/s]
Loading safetensors checkpoint shards:  35% Completed | 33/93 [00:05<00:09,  6.31it/s]
Loading safetensors checkpoint shards:  37% Completed | 34/93 [00:05<00:09,  6.47it/s]
Loading safetensors checkpoint shards:  38% Completed | 35/93 [00:05<00:09,  6.34it/s]
Loading safetensors checkpoint shards:  39% Completed | 36/93 [00:05<00:09,  6.05it/s]
Loading safetensors checkpoint shards:  40% Completed | 37/93 [00:06<00:09,  5.84it/s]
Loading safetensors checkpoint shards:  41% Completed | 38/93 [00:06<00:09,  5.95it/s]
Loading safetensors checkpoint shards:  42% Completed | 39/93 [00:06<00:08,  6.17it/s]
Loading safetensors checkpoint shards:  43% Completed | 40/93 [00:06<00:11,  4.59it/s]
Loading safetensors checkpoint shards:  44% Completed | 41/93 [00:06<00:10,  4.78it/s]
Loading safetensors checkpoint shards:  45% Completed | 42/93 [00:07<00:09,  5.12it/s]
Loading safetensors checkpoint shards:  46% Completed | 43/93 [00:07<00:09,  5.47it/s]
Loading safetensors checkpoint shards:  47% Completed | 44/93 [00:07<00:08,  5.46it/s]
Loading safetensors checkpoint shards:  48% Completed | 45/93 [00:07<00:08,  6.00it/s]
Loading safetensors checkpoint shards:  49% Completed | 46/93 [00:07<00:07,  6.00it/s]
Loading safetensors checkpoint shards:  51% Completed | 47/93 [00:07<00:07,  6.22it/s]
Loading safetensors checkpoint shards:  53% Completed | 49/93 [00:08<00:06,  7.31it/s]
Loading safetensors checkpoint shards:  54% Completed | 50/93 [00:08<00:05,  7.19it/s]
Loading safetensors checkpoint shards:  55% Completed | 51/93 [00:08<00:06,  6.66it/s]
Loading safetensors checkpoint shards:  56% Completed | 52/93 [00:08<00:06,  6.26it/s]
Loading safetensors checkpoint shards:  57% Completed | 53/93 [00:08<00:06,  6.31it/s]
Loading safetensors checkpoint shards:  58% Completed | 54/93 [00:08<00:05,  6.64it/s]
Loading safetensors checkpoint shards:  59% Completed | 55/93 [00:09<00:05,  6.48it/s]
Loading safetensors checkpoint shards:  60% Completed | 56/93 [00:09<00:05,  6.65it/s]
Loading safetensors checkpoint shards:  61% Completed | 57/93 [00:09<00:05,  6.28it/s]
Loading safetensors checkpoint shards:  62% Completed | 58/93 [00:09<00:05,  6.42it/s]
Loading safetensors checkpoint shards:  63% Completed | 59/93 [00:09<00:05,  6.58it/s]
Loading safetensors checkpoint shards:  65% Completed | 60/93 [00:09<00:04,  6.86it/s]
Loading safetensors checkpoint shards:  66% Completed | 61/93 [00:09<00:04,  6.67it/s]
Loading safetensors checkpoint shards:  67% Completed | 62/93 [00:10<00:04,  6.64it/s]
Loading safetensors checkpoint shards:  68% Completed | 63/93 [00:10<00:04,  6.66it/s]
Loading safetensors checkpoint shards:  69% Completed | 64/93 [00:10<00:04,  6.28it/s]
Loading safetensors checkpoint shards:  70% Completed | 65/93 [00:10<00:04,  5.99it/s]
Loading safetensors checkpoint shards:  71% Completed | 66/93 [00:10<00:04,  5.80it/s]
Loading safetensors checkpoint shards:  72% Completed | 67/93 [00:10<00:04,  5.87it/s]
Loading safetensors checkpoint shards:  73% Completed | 68/93 [00:11<00:04,  5.74it/s]
Loading safetensors checkpoint shards:  74% Completed | 69/93 [00:11<00:04,  5.84it/s]
Loading safetensors checkpoint shards:  75% Completed | 70/93 [00:11<00:03,  6.28it/s]
Loading safetensors checkpoint shards:  76% Completed | 71/93 [00:11<00:03,  6.07it/s]
Loading safetensors checkpoint shards:  77% Completed | 72/93 [00:11<00:03,  6.42it/s]
Loading safetensors checkpoint shards:  80% Completed | 74/93 [00:11<00:02,  7.86it/s]
Loading safetensors checkpoint shards:  81% Completed | 75/93 [00:12<00:02,  7.48it/s]
Loading safetensors checkpoint shards:  82% Completed | 76/93 [00:12<00:02,  7.02it/s]
Loading safetensors checkpoint shards:  83% Completed | 77/93 [00:12<00:02,  7.17it/s]
Loading safetensors checkpoint shards:  84% Completed | 78/93 [00:12<00:02,  6.64it/s]
Loading safetensors checkpoint shards:  85% Completed | 79/93 [00:12<00:02,  6.91it/s]
Loading safetensors checkpoint shards:  86% Completed | 80/93 [00:12<00:01,  6.64it/s]
Loading safetensors checkpoint shards:  87% Completed | 81/93 [00:13<00:01,  6.51it/s]
Loading safetensors checkpoint shards:  88% Completed | 82/93 [00:13<00:01,  6.85it/s]
Loading safetensors checkpoint shards:  90% Completed | 84/93 [00:13<00:01,  7.68it/s]
Loading safetensors checkpoint shards:  91% Completed | 85/93 [00:13<00:01,  7.03it/s]
Loading safetensors checkpoint shards:  92% Completed | 86/93 [00:13<00:00,  7.05it/s]
Loading safetensors checkpoint shards:  94% Completed | 87/93 [00:13<00:00,  6.96it/s]
Loading safetensors checkpoint shards:  95% Completed | 88/93 [00:14<00:00,  6.48it/s]
Loading safetensors checkpoint shards:  96% Completed | 89/93 [00:14<00:00,  6.13it/s]
Loading safetensors checkpoint shards:  97% Completed | 90/93 [00:14<00:00,  6.15it/s]
Loading safetensors checkpoint shards:  98% Completed | 91/93 [00:14<00:00,  6.39it/s]
Loading safetensors checkpoint shards:  99% Completed | 92/93 [00:14<00:00,  6.10it/s]
Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:14<00:00,  6.19it/s]
Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:14<00:00,  6.26it/s]

TP MOE layer 3, pool: 0x5db02e10, expert num: 160, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
terminate called after throwing an instance of 'std::runtime_error'
  what():  no weight source
Fatal Python error: Aborted

Thread 0x00007fbd2dfff6c0 (most recent call first):
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 359 in wait
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 655 in wait
  File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007fcd502e56c0 (most recent call first):
  File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 61 in _recv_msg
  File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 195 in _read_thread
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007fd38dce1b80 (most recent call first):
  File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/kt_kernel/utils/amx.py", line 294 in load_weights
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 258 in process_weights_after_loading
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 618 in load_weights_and_postprocess
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 599 in load_model
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/__init__.py", line 28 in get_model
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 767 in load_model
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 410 in initialize
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 324 in __init__
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/tp_worker.py", line 237 in __init__
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 312 in __init__
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 2712 in run_scheduler_process
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108 in run
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 135 in _main
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, pybase64._pybase64, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, msgspec._core, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda_utils (total: 101)

Others

No response

Originally created by @reasv on GitHub (Nov 27, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1649 Originally assigned to: @ovowei on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info CPU: 2x Intel Xeon 8592+ ES Q2SP (60c per socket) GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB) OS: Arch Linux (kernel 6.17.8-arch1-1) SGLang: 0.5.5.post3 kt-kernel: 0.1.0 (cloned from github) ### Reproduction ```bash python -m sglang.launch_server \ --host 0.0.0.0 \ --port 8000 \ --model /data/models/GLM-4.6-FP8 \ --trust-remote-code \ --mem-fraction-static 0.92 \ --chunked-prefill-size 4096 \ --served-model-name GLM-4.6 \ --enable-mixed-chunk \ --kt-method AMXINT8 \ --kt-weight-path /data/models/GLM-4.6-Channel-INT8 \ --kt-cpuinfer 120 \ --kt-threadpool-count 4 \ --kt-num-gpu-experts 32 \ --kt-max-deferred-experts-per-token 2 ``` Error: ```bash [2025-11-27 06:21:20] WARNING server_args.py:1213: Attention backend not explicitly specified. Use flashinfer backend by default. [2025-11-27 06:21:21] server_args=ServerArgs(model_path='/data/models/GLM-4.6-FP8', tokenizer_path='/data/models/GLM-4.6-FP8', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.92, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=157567588, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='GLM-4.6', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm='static', init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='/data/models/GLM-4.6-Channel-INT8', kt_method='AMXINT8', kt_cpuinfer=120, kt_threadpool_count=4, kt_num_gpu_experts=32, kt_max_deferred_experts_per_token=2, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, decrypted_config_file=None, decrypted_draft_config_file=None) [2025-11-27 06:21:21] Using default HuggingFace chat template with detected content format: openai [2025-11-27 06:21:25] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2025-11-27 06:21:26] Init torch distributed ends. mem usage=0.00 GB [2025-11-27 06:21:27] Load weight begin. avail mem=94.31 GB CPUInfer[0x4ad548e0]: Hello WorkerPool[0x5db02e10] 4 subpools, [numa:threads][0:30] [1:30] [2:30] [3:30] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 30 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 30 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 2, 30 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 3, 30 threads Loading safetensors checkpoint shards: 0% Completed | 0/93 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 1% Completed | 1/93 [00:00<00:14, 6.36it/s] Loading safetensors checkpoint shards: 2% Completed | 2/93 [00:00<00:14, 6.26it/s] Loading safetensors checkpoint shards: 3% Completed | 3/93 [00:00<00:15, 5.85it/s] Loading safetensors checkpoint shards: 4% Completed | 4/93 [00:00<00:14, 6.15it/s] Loading safetensors checkpoint shards: 5% Completed | 5/93 [00:00<00:13, 6.37it/s] Loading safetensors checkpoint shards: 6% Completed | 6/93 [00:00<00:14, 6.06it/s] Loading safetensors checkpoint shards: 8% Completed | 7/93 [00:01<00:14, 5.82it/s] Loading safetensors checkpoint shards: 9% Completed | 8/93 [00:01<00:14, 6.05it/s] Loading safetensors checkpoint shards: 10% Completed | 9/93 [00:01<00:13, 6.35it/s] Loading safetensors checkpoint shards: 11% Completed | 10/93 [00:01<00:13, 6.34it/s] Loading safetensors checkpoint shards: 12% Completed | 11/93 [00:01<00:13, 6.04it/s] Loading safetensors checkpoint shards: 13% Completed | 12/93 [00:01<00:13, 5.82it/s] Loading safetensors checkpoint shards: 14% Completed | 13/93 [00:02<00:13, 5.95it/s] Loading safetensors checkpoint shards: 15% Completed | 14/93 [00:02<00:16, 4.88it/s] Loading safetensors checkpoint shards: 16% Completed | 15/93 [00:02<00:14, 5.23it/s] Loading safetensors checkpoint shards: 17% Completed | 16/93 [00:02<00:14, 5.47it/s] Loading safetensors checkpoint shards: 18% Completed | 17/93 [00:02<00:13, 5.52it/s] Loading safetensors checkpoint shards: 19% Completed | 18/93 [00:03<00:13, 5.72it/s] Loading safetensors checkpoint shards: 20% Completed | 19/93 [00:03<00:12, 6.15it/s] Loading safetensors checkpoint shards: 22% Completed | 20/93 [00:03<00:11, 6.25it/s] Loading safetensors checkpoint shards: 23% Completed | 21/93 [00:03<00:12, 6.00it/s] Loading safetensors checkpoint shards: 24% Completed | 22/93 [00:03<00:11, 6.17it/s] Loading safetensors checkpoint shards: 25% Completed | 23/93 [00:03<00:10, 6.54it/s] Loading safetensors checkpoint shards: 26% Completed | 24/93 [00:04<00:11, 6.23it/s] Loading safetensors checkpoint shards: 27% Completed | 25/93 [00:04<00:10, 6.37it/s] Loading safetensors checkpoint shards: 28% Completed | 26/93 [00:04<00:09, 6.74it/s] Loading safetensors checkpoint shards: 29% Completed | 27/93 [00:04<00:09, 7.10it/s] Loading safetensors checkpoint shards: 30% Completed | 28/93 [00:04<00:08, 7.37it/s] Loading safetensors checkpoint shards: 31% Completed | 29/93 [00:04<00:08, 7.14it/s] Loading safetensors checkpoint shards: 32% Completed | 30/93 [00:04<00:09, 6.53it/s] Loading safetensors checkpoint shards: 33% Completed | 31/93 [00:05<00:10, 6.17it/s] Loading safetensors checkpoint shards: 34% Completed | 32/93 [00:05<00:10, 5.92it/s] Loading safetensors checkpoint shards: 35% Completed | 33/93 [00:05<00:09, 6.31it/s] Loading safetensors checkpoint shards: 37% Completed | 34/93 [00:05<00:09, 6.47it/s] Loading safetensors checkpoint shards: 38% Completed | 35/93 [00:05<00:09, 6.34it/s] Loading safetensors checkpoint shards: 39% Completed | 36/93 [00:05<00:09, 6.05it/s] Loading safetensors checkpoint shards: 40% Completed | 37/93 [00:06<00:09, 5.84it/s] Loading safetensors checkpoint shards: 41% Completed | 38/93 [00:06<00:09, 5.95it/s] Loading safetensors checkpoint shards: 42% Completed | 39/93 [00:06<00:08, 6.17it/s] Loading safetensors checkpoint shards: 43% Completed | 40/93 [00:06<00:11, 4.59it/s] Loading safetensors checkpoint shards: 44% Completed | 41/93 [00:06<00:10, 4.78it/s] Loading safetensors checkpoint shards: 45% Completed | 42/93 [00:07<00:09, 5.12it/s] Loading safetensors checkpoint shards: 46% Completed | 43/93 [00:07<00:09, 5.47it/s] Loading safetensors checkpoint shards: 47% Completed | 44/93 [00:07<00:08, 5.46it/s] Loading safetensors checkpoint shards: 48% Completed | 45/93 [00:07<00:08, 6.00it/s] Loading safetensors checkpoint shards: 49% Completed | 46/93 [00:07<00:07, 6.00it/s] Loading safetensors checkpoint shards: 51% Completed | 47/93 [00:07<00:07, 6.22it/s] Loading safetensors checkpoint shards: 53% Completed | 49/93 [00:08<00:06, 7.31it/s] Loading safetensors checkpoint shards: 54% Completed | 50/93 [00:08<00:05, 7.19it/s] Loading safetensors checkpoint shards: 55% Completed | 51/93 [00:08<00:06, 6.66it/s] Loading safetensors checkpoint shards: 56% Completed | 52/93 [00:08<00:06, 6.26it/s] Loading safetensors checkpoint shards: 57% Completed | 53/93 [00:08<00:06, 6.31it/s] Loading safetensors checkpoint shards: 58% Completed | 54/93 [00:08<00:05, 6.64it/s] Loading safetensors checkpoint shards: 59% Completed | 55/93 [00:09<00:05, 6.48it/s] Loading safetensors checkpoint shards: 60% Completed | 56/93 [00:09<00:05, 6.65it/s] Loading safetensors checkpoint shards: 61% Completed | 57/93 [00:09<00:05, 6.28it/s] Loading safetensors checkpoint shards: 62% Completed | 58/93 [00:09<00:05, 6.42it/s] Loading safetensors checkpoint shards: 63% Completed | 59/93 [00:09<00:05, 6.58it/s] Loading safetensors checkpoint shards: 65% Completed | 60/93 [00:09<00:04, 6.86it/s] Loading safetensors checkpoint shards: 66% Completed | 61/93 [00:09<00:04, 6.67it/s] Loading safetensors checkpoint shards: 67% Completed | 62/93 [00:10<00:04, 6.64it/s] Loading safetensors checkpoint shards: 68% Completed | 63/93 [00:10<00:04, 6.66it/s] Loading safetensors checkpoint shards: 69% Completed | 64/93 [00:10<00:04, 6.28it/s] Loading safetensors checkpoint shards: 70% Completed | 65/93 [00:10<00:04, 5.99it/s] Loading safetensors checkpoint shards: 71% Completed | 66/93 [00:10<00:04, 5.80it/s] Loading safetensors checkpoint shards: 72% Completed | 67/93 [00:10<00:04, 5.87it/s] Loading safetensors checkpoint shards: 73% Completed | 68/93 [00:11<00:04, 5.74it/s] Loading safetensors checkpoint shards: 74% Completed | 69/93 [00:11<00:04, 5.84it/s] Loading safetensors checkpoint shards: 75% Completed | 70/93 [00:11<00:03, 6.28it/s] Loading safetensors checkpoint shards: 76% Completed | 71/93 [00:11<00:03, 6.07it/s] Loading safetensors checkpoint shards: 77% Completed | 72/93 [00:11<00:03, 6.42it/s] Loading safetensors checkpoint shards: 80% Completed | 74/93 [00:11<00:02, 7.86it/s] Loading safetensors checkpoint shards: 81% Completed | 75/93 [00:12<00:02, 7.48it/s] Loading safetensors checkpoint shards: 82% Completed | 76/93 [00:12<00:02, 7.02it/s] Loading safetensors checkpoint shards: 83% Completed | 77/93 [00:12<00:02, 7.17it/s] Loading safetensors checkpoint shards: 84% Completed | 78/93 [00:12<00:02, 6.64it/s] Loading safetensors checkpoint shards: 85% Completed | 79/93 [00:12<00:02, 6.91it/s] Loading safetensors checkpoint shards: 86% Completed | 80/93 [00:12<00:01, 6.64it/s] Loading safetensors checkpoint shards: 87% Completed | 81/93 [00:13<00:01, 6.51it/s] Loading safetensors checkpoint shards: 88% Completed | 82/93 [00:13<00:01, 6.85it/s] Loading safetensors checkpoint shards: 90% Completed | 84/93 [00:13<00:01, 7.68it/s] Loading safetensors checkpoint shards: 91% Completed | 85/93 [00:13<00:01, 7.03it/s] Loading safetensors checkpoint shards: 92% Completed | 86/93 [00:13<00:00, 7.05it/s] Loading safetensors checkpoint shards: 94% Completed | 87/93 [00:13<00:00, 6.96it/s] Loading safetensors checkpoint shards: 95% Completed | 88/93 [00:14<00:00, 6.48it/s] Loading safetensors checkpoint shards: 96% Completed | 89/93 [00:14<00:00, 6.13it/s] Loading safetensors checkpoint shards: 97% Completed | 90/93 [00:14<00:00, 6.15it/s] Loading safetensors checkpoint shards: 98% Completed | 91/93 [00:14<00:00, 6.39it/s] Loading safetensors checkpoint shards: 99% Completed | 92/93 [00:14<00:00, 6.10it/s] Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:14<00:00, 6.19it/s] Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:14<00:00, 6.26it/s] TP MOE layer 3, pool: 0x5db02e10, expert num: 160, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 terminate called after throwing an instance of 'std::runtime_error' what(): no weight source Fatal Python error: Aborted Thread 0x00007fbd2dfff6c0 (most recent call first): File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 359 in wait File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 655 in wait File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007fcd502e56c0 (most recent call first): File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 61 in _recv_msg File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 195 in _read_thread File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007fd38dce1b80 (most recent call first): File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/kt_kernel/utils/amx.py", line 294 in load_weights File "/home/admin/projects/sgl/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 258 in process_weights_after_loading File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 618 in load_weights_and_postprocess File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 599 in load_model File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/__init__.py", line 28 in get_model File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 767 in load_model File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 410 in initialize File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 324 in __init__ File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/tp_worker.py", line 237 in __init__ File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 312 in __init__ File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 2712 in run_scheduler_process File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108 in run File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 135 in _main File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main File "<string>", line 1 in <module> Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, pybase64._pybase64, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, yaml._yaml, regex._regex, markupsafe._speedups, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, msgspec._core, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda_utils (total: 101) ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:48:06 +00:00
Author
Owner
Copy link

@ovowei commented on GitHub (Dec 2, 2025):

It seems that there is a problem with the weight. Is the GLM-4.6-Channel-INT8 you quantified 4numa？

<!-- gh-comment-id:3601909682 --> @ovowei commented on GitHub (Dec 2, 2025): It seems that there is a problem with the weight. Is the GLM-4.6-Channel-INT8 you quantified 4numa？
bsakaguchi commented 2026-10-29 14:48:06 +00:00
Author
Owner
Copy link

@reasv commented on GitHub (Dec 4, 2025):

I got it from here: https://huggingface.co/IntervitensInc/GLM-4.6-Channel-int8
Running the quantization script on the FP8 weights gave me an error but I don’t remember what it was right now. I’d have to try again when I go home

<!-- gh-comment-id:3613710516 --> @reasv commented on GitHub (Dec 4, 2025): I got it from here: https://huggingface.co/IntervitensInc/GLM-4.6-Channel-int8 Running the quantization script on the FP8 weights gave me an error but I don’t remember what it was right now. I’d have to try again when I go home
bsakaguchi commented 2026-10-29 14:48:06 +00:00
Author
Owner
Copy link

@reasv commented on GitHub (Dec 7, 2025):

This is what happens when I try to quantize the official FP8 weights into INT8

python scripts/convert_cpu_weights.py --input-path /data/models/GLM-4.6-FP8 --input-type fp8 --output /data/models/GLM-4.6-INT8-quant --quant-method int8
Loading model configuration...
Error during conversion: FP8 quantization_config found but 'weight_block_size' field is missing. Expected format: 'weight_block_size': [128, 128]
Traceback (most recent call last):
  File "/home/admin/projects/sgl/ktransformers/kt-kernel/scripts/convert_cpu_weights.py", line 868, in main
    model_config = load_model_config(args.input_path, args.input_type)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/admin/projects/sgl/ktransformers/kt-kernel/scripts/convert_cpu_weights.py", line 100, in load_model_config
    raise ValueError(
ValueError: FP8 quantization_config found but 'weight_block_size' field is missing. Expected format: 'weight_block_size': [128, 128]
<!-- gh-comment-id:3622792389 --> @reasv commented on GitHub (Dec 7, 2025): This is what happens when I try to quantize the official FP8 weights into INT8 ``` python scripts/convert_cpu_weights.py --input-path /data/models/GLM-4.6-FP8 --input-type fp8 --output /data/models/GLM-4.6-INT8-quant --quant-method int8 Loading model configuration... Error during conversion: FP8 quantization_config found but 'weight_block_size' field is missing. Expected format: 'weight_block_size': [128, 128] Traceback (most recent call last): File "/home/admin/projects/sgl/ktransformers/kt-kernel/scripts/convert_cpu_weights.py", line 868, in main model_config = load_model_config(args.input_path, args.input_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/admin/projects/sgl/ktransformers/kt-kernel/scripts/convert_cpu_weights.py", line 100, in load_model_config raise ValueError( ValueError: FP8 quantization_config found but 'weight_block_size' field is missing. Expected format: 'weight_block_size': [128, 128] ```
bsakaguchi commented 2026-10-29 14:48:07 +00:00
Author
Owner
Copy link

@reasv commented on GitHub (Dec 10, 2025):

Running the quantization script on the 16-bit version of the weights worked, and the no weight source error disappeared. So I guess this might be a format change from an earlier version, or something. Not sure whether to close this issue. I had a different problem afterwards so I'll open a new issue for that

<!-- gh-comment-id:3635056851 --> @reasv commented on GitHub (Dec 10, 2025): Running the quantization script on the 16-bit version of the weights worked, and the no weight source error disappeared. So I guess this might be a format change from an earlier version, or something. Not sure whether to close this issue. I had a different problem afterwards so I'll open a new issue for that
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1061
No description provided.