kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1839] GLM 4.7 tensor size mismatch #1147

New issue
Closed
opened 2026-10-29 14:49:14 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-29 14:49:14 +00:00
Owner
Copy link

Originally created by @Gracana on GitHub (Feb 8, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1839

Reminder

  • I have read the above rules and searched the existing issues.

System Info

% lscpu
Architecture:                x86_64
  CPU op-mode(s):            32-bit, 64-bit
  Address sizes:             46 bits physical, 57 bits virtual
  Byte Order:                Little Endian
CPU(s):                      32
  On-line CPU(s) list:       0-31
Vendor ID:                   GenuineIntel
  Model name:                Intel(R) Xeon(R) w5-3435X
    CPU family:              6
    Model:                   143
    Thread(s) per core:      2
    Core(s) per socket:      16
    Socket(s):               1
    Stepping:                8
    CPU(s) scaling MHz:      21%
    CPU max MHz:             4700.0000
    CPU min MHz:             800.0000
    BogoMIPS:                6192.00
    Flags:                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pe
                             bs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x
                             2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanc
                             ed tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_p
                             t avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dther
                             m ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rd
                             pid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Virtualization features:
  Virtualization:            VT-x
Caches (sum of all):
  L1d:                       768 KiB (16 instances)
  L1i:                       512 KiB (16 instances)
  L2:                        32 MiB (16 instances)
  L3:                        45 MiB (1 instance)
NUMA:
  NUMA node(s):              1
  NUMA node0 CPU(s):         0-31
Vulnerabilities:
  Gather data sampling:      Not affected
  Ghostwrite:                Not affected
  Indirect target selection: Not affected
  Itlb multihit:             Not affected
  L1tf:                      Not affected
  Mds:                       Not affected
  Meltdown:                  Not affected
  Mmio stale data:           Not affected
  Old microcode:             Vulnerable
  Reg file data sampling:    Not affected
  Retbleed:                  Not affected
  Spec rstack overflow:      Not affected
  Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
  Srbds:                     Not affected
  Tsa:                       Not affected
  Tsx async abort:           Not affected
  Vmscape:                   Mitigation; IBPB before exit to userspace

% nvidia-smi
Sun Feb  8 04:03:02 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4000               Off |   00000000:16:00.0 Off |                  Off |
|  0%   26C    P8             14W /  140W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX A4000               Off |   00000000:34:00.0 Off |                  Off |
|  0%   33C    P8             14W /  140W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX A4000               Off |   00000000:52:00.0 Off |                  Off |
|  0%   26C    P8             15W /  140W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA RTX A4000               Off |   00000000:70:00.0 Off |                  Off |
|  0%   27C    P8             14W /  140W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA RTX A4000               Off |   00000000:90:00.0 Off |                  Off |
|  0%   27C    P8             12W /  140W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA RTX A4000               Off |   00000000:AC:00.0 Off |                  Off |
|  0%   27C    P8             16W /  140W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA RTX A4000               Off |   00000000:CA:00.0 Off |                  Off |
|  0%   34C    P8             13W /  140W |       1MiB /  16376MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

% free -m
               total        used        free      shared  buff/cache   available
Mem:          773141       10653      182365          27      586834      762488
Swap:              0           0           0

% kt version

KTransformers CLI v0.1.0

  Python      3.11.14
  Platform    Linux 6.18.7-arch1-1
  CUDA        13.1

Packages:

  kt-kernel    0.5.1
  sglang       0.0.0.dev9554+g937ceab25 (PyPI)

Reproduction

Model is GLM-4.7 from here: https://huggingface.co/zai-org/GLM-4.7

I quantized it in this way:

python scripts/convert_cpu_weights.py --input-path /mnt/scratch/GLM-4.7 --input-type bf16 --output ~/llm-models/GLM-4.7-CPU-Weights --quant-method int4 --cpuinfer-threads 16 --threadpool-count 1 --no-merge-safetensor

Running with this command:

% python -m sglang.launch_server --model /mnt/scratch/GLM-4.7/ --kt-weight-path ~/llm-models/GLM-4.7-CPU-Weights --kt-num-gpu-experts 8 --kt-expert-placement-strategy uniform --kt-threadpool-count 1 --kt-cpuinfer 16

Output:

[2026-02-08 04:00:54] INFO server_args.py:1803: Attention backend not specified. Use flashinfer backend by default.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
[2026-02-08 04:00:54] server_args=ServerArgs(model_path='/mnt/scratch/GLM-4.7/', tokenizer_path='/mnt/scratch/GLM-4.7/', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='127.0.0.1', port=30000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.772, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=2048, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=509345955, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='/mnt/scratch/GLM-4.7/', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', nsa_prefill_backend=None, nsa_decode_backend=None, disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path='/home/alex/llm-models/GLM-4.7-CPU-Weights', kt_method='AMXINT4', kt_cpuinfer=16, kt_threadpool_count=1, kt_num_gpu_experts=8, kt_gpu_experts_ratio=None, kt_max_deferred_experts_per_token=None, kt_gpu_prefill_token_threshold=None, record_kt_gpu_expert_distribution=False, kt_enable_dynamic_expert_update=False, kt_expert_placement_strategy='uniform', dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=8, cuda_graph_bs=[1, 2, 4, 8], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=2048, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='in-seq-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_enable_fake_auto=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-02-08 04:00:55] Using default HuggingFace chat template with detected content format: openai
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
[2026-02-08 04:01:00] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2026-02-08 04:01:00] Init torch distributed ends. elapsed=0.15 s, mem usage=0.03 GB
[2026-02-08 04:01:00] Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-02-08 04:01:00] Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-02-08 04:01:00] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/transformers/__init__.py)
[2026-02-08 04:01:01] Load weight begin. avail mem=15.37 GB
[2026-02-08 04:01:01] Shared experts fusion optimization enabled.
[2026-02-08 04:01:01] Using kt_num_gpu_experts=8 per layer, total GPU experts: 712 (= 8 × 89 MoE layers)
[2026-02-08 04:01:01] Using uniform strategy for GPU expert placement
[2026-02-08 04:01:01] KT GPU experts: layer 3 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 4 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 5 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 6 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 7 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 8 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 9 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 10 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 11 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 12 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 13 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 14 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 15 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 16 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 17 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 18 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 19 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 20 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 21 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 22 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 23 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 24 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 25 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 26 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 27 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 28 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 29 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 30 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 31 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 32 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 33 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 34 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 35 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 36 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 37 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 38 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 39 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 40 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 41 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 42 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 43 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 44 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 45 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 46 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 47 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 48 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 49 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 50 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 51 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 52 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 53 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 54 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 55 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 56 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 57 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 58 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 59 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 60 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 61 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 62 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 63 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 64 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 65 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 66 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 67 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 68 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 69 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 70 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 71 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 72 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 73 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 74 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 75 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 76 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 77 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 78 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 79 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 80 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 81 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 82 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 83 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 84 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 85 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 86 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 87 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 88 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 89 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 90 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] KT GPU experts: layer 91 (MoE) has 8 GPU experts
[2026-02-08 04:01:01] Generated KT GPU experts masks using 'uniform' strategy: 89 MoE layers (out of 92 total layers) x 160 experts, total GPU experts in MoE layers = 712
[2026-02-08 04:01:01] [KT] Created shared staging buffer: 20.0 MiB (shape=torch.Size([2048, 5120]), dtype=torch.bfloat16)
[2026-02-08 04:01:01] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 3049, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 350, in __init__
    self.init_model_worker()
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 539, in init_model_worker
    self.init_tp_model_worker()
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 501, in init_tp_model_worker
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/tp_worker.py", line 242, in __init__
    self._init_model_runner()
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/tp_worker.py", line 325, in _init_model_runner
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_executor/model_runner.py", line 391, in __init__
    self.initialize(min_per_gpu_memory)
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_executor/model_runner.py", line 471, in initialize
    self.load_model()
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_executor/model_runner.py", line 909, in load_model
    self.model = self.loader.load_model(
                 ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_loader/loader.py", line 666, in load_model
    model = _initialize_model(
            ^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_loader/loader.py", line 281, in _initialize_model
    return model_class(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 1030, in __init__
    self.model = Glm4MoeModel(
                 ^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 919, in __init__
    self.layers, self.start_layer, self.end_layer = make_layers(
                                                    ^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/utils/common.py", line 643, in make_layers
    + get_offloader().wrap_modules(
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/utils/offloader.py", line 36, in wrap_modules
    return list(all_modules_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/utils/common.py", line 645, in <genexpr>
    layer_fn(idx=idx, prefix=add_prefix(idx, prefix))
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 921, in <lambda>
    lambda idx, prefix: Glm4MoeDecoderLayer(
                        ^^^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 740, in __init__
    self.mlp = Glm4MoeSparseMoeBlock(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 374, in __init__
    self.experts = get_moe_impl_class(quant_config)(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 292, in __init__
    self.quant_method.create_weights(
  File "/home/alex/software/ktransformers/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1980, in create_weights
    self.wrapper = KTMoEWrapper(
                   ^^^^^^^^^^^^^
  File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/kt_kernel/experts.py", line 106, in __new__
    return backend_cls(
           ^^^^^^^^^^^^
  File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 82, in __init__
    super().__init__(
  File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/kt_kernel/experts_base.py", line 203, in __init__
    self.gpu_experts_mask.copy_(gpu_experts_mask)
  File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/torch/utils/_device.py", line 103, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The size of tensor a (161) must match the size of tensor b (160) at non-singleton dimension 0

[2026-02-08 04:01:01] Received sigquit from a child process. It usually means the child failed.

Others

No response

Originally created by @Gracana on GitHub (Feb 8, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1839 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info ``` % lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) w5-3435X CPU family: 6 Model: 143 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 8 CPU(s) scaling MHz: 21% CPU max MHz: 4700.0000 CPU min MHz: 800.0000 BogoMIPS: 6192.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pe bs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x 2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanc ed tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_p t avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect user_shstk avx_vnni avx512_bf16 wbnoinvd dther m ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rd pid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr ibt amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities Virtualization features: Virtualization: VT-x Caches (sum of all): L1d: 768 KiB (16 instances) L1i: 512 KiB (16 instances) L2: 32 MiB (16 instances) L3: 45 MiB (1 instance) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerabilities: Gather data sampling: Not affected Ghostwrite: Not affected Indirect target selection: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Old microcode: Vulnerable Reg file data sampling: Not affected Retbleed: Not affected Spec rstack overflow: Not affected Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S Srbds: Not affected Tsa: Not affected Tsx async abort: Not affected Vmscape: Mitigation; IBPB before exit to userspace % nvidia-smi Sun Feb 8 04:03:02 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 590.48.01 Driver Version: 590.48.01 CUDA Version: 13.1 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A4000 Off | 00000000:16:00.0 Off | Off | | 0% 26C P8 14W / 140W | 1MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA RTX A4000 Off | 00000000:34:00.0 Off | Off | | 0% 33C P8 14W / 140W | 1MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA RTX A4000 Off | 00000000:52:00.0 Off | Off | | 0% 26C P8 15W / 140W | 1MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA RTX A4000 Off | 00000000:70:00.0 Off | Off | | 0% 27C P8 14W / 140W | 1MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA RTX A4000 Off | 00000000:90:00.0 Off | Off | | 0% 27C P8 12W / 140W | 1MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA RTX A4000 Off | 00000000:AC:00.0 Off | Off | | 0% 27C P8 16W / 140W | 1MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA RTX A4000 Off | 00000000:CA:00.0 Off | Off | | 0% 34C P8 13W / 140W | 1MiB / 16376MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ % free -m total used free shared buff/cache available Mem: 773141 10653 182365 27 586834 762488 Swap: 0 0 0 % kt version KTransformers CLI v0.1.0 Python 3.11.14 Platform Linux 6.18.7-arch1-1 CUDA 13.1 Packages: kt-kernel 0.5.1 sglang 0.0.0.dev9554+g937ceab25 (PyPI) ``` ### Reproduction Model is GLM-4.7 from here: https://huggingface.co/zai-org/GLM-4.7 I quantized it in this way: ``` python scripts/convert_cpu_weights.py --input-path /mnt/scratch/GLM-4.7 --input-type bf16 --output ~/llm-models/GLM-4.7-CPU-Weights --quant-method int4 --cpuinfer-threads 16 --threadpool-count 1 --no-merge-safetensor ``` Running with this command: ``` % python -m sglang.launch_server --model /mnt/scratch/GLM-4.7/ --kt-weight-path ~/llm-models/GLM-4.7-CPU-Weights --kt-num-gpu-experts 8 --kt-expert-placement-strategy uniform --kt-threadpool-count 1 --kt-cpuinfer 16 ``` Output: ``` [2026-02-08 04:00:54] INFO server_args.py:1803: Attention backend not specified. Use flashinfer backend by default. <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. [2026-02-08 04:00:54] server_args=ServerArgs(model_path='/mnt/scratch/GLM-4.7/', tokenizer_path='/mnt/scratch/GLM-4.7/', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=False, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='127.0.0.1', port=30000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.772, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=2048, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=509345955, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='/mnt/scratch/GLM-4.7/', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='auto', nsa_prefill_backend=None, nsa_decode_backend=None, disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path='/home/alex/llm-models/GLM-4.7-CPU-Weights', kt_method='AMXINT4', kt_cpuinfer=16, kt_threadpool_count=1, kt_num_gpu_experts=8, kt_gpu_experts_ratio=None, kt_max_deferred_experts_per_token=None, kt_gpu_prefill_token_threshold=None, record_kt_gpu_expert_distribution=False, kt_enable_dynamic_expert_update=False, kt_expert_placement_strategy='uniform', dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=8, cuda_graph_bs=[1, 2, 4, 8], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=2048, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='in-seq-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_enable_fake_auto=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None) [2026-02-08 04:00:55] Using default HuggingFace chat template with detected content format: openai <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead. <frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead. [2026-02-08 04:01:00] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2026-02-08 04:01:00] Init torch distributed ends. elapsed=0.15 s, mem usage=0.03 GB [2026-02-08 04:01:00] Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr' [2026-02-08 04:01:00] Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr' [2026-02-08 04:01:00] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/transformers/__init__.py) [2026-02-08 04:01:01] Load weight begin. avail mem=15.37 GB [2026-02-08 04:01:01] Shared experts fusion optimization enabled. [2026-02-08 04:01:01] Using kt_num_gpu_experts=8 per layer, total GPU experts: 712 (= 8 × 89 MoE layers) [2026-02-08 04:01:01] Using uniform strategy for GPU expert placement [2026-02-08 04:01:01] KT GPU experts: layer 3 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 4 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 5 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 6 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 7 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 8 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 9 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 10 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 11 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 12 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 13 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 14 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 15 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 16 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 17 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 18 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 19 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 20 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 21 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 22 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 23 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 24 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 25 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 26 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 27 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 28 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 29 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 30 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 31 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 32 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 33 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 34 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 35 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 36 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 37 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 38 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 39 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 40 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 41 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 42 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 43 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 44 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 45 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 46 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 47 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 48 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 49 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 50 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 51 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 52 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 53 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 54 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 55 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 56 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 57 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 58 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 59 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 60 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 61 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 62 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 63 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 64 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 65 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 66 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 67 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 68 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 69 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 70 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 71 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 72 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 73 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 74 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 75 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 76 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 77 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 78 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 79 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 80 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 81 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 82 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 83 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 84 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 85 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 86 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 87 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 88 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 89 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 90 (MoE) has 8 GPU experts [2026-02-08 04:01:01] KT GPU experts: layer 91 (MoE) has 8 GPU experts [2026-02-08 04:01:01] Generated KT GPU experts masks using 'uniform' strategy: 89 MoE layers (out of 92 total layers) x 160 experts, total GPU experts in MoE layers = 712 [2026-02-08 04:01:01] [KT] Created shared staging buffer: 20.0 MiB (shape=torch.Size([2048, 5120]), dtype=torch.bfloat16) [2026-02-08 04:01:01] Scheduler hit an exception: Traceback (most recent call last): File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 3049, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 350, in __init__ self.init_model_worker() File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 539, in init_model_worker self.init_tp_model_worker() File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/scheduler.py", line 501, in init_tp_model_worker self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/tp_worker.py", line 242, in __init__ self._init_model_runner() File "/home/alex/software/ktransformers/sglang/python/sglang/srt/managers/tp_worker.py", line 325, in _init_model_runner self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_executor/model_runner.py", line 391, in __init__ self.initialize(min_per_gpu_memory) File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_executor/model_runner.py", line 471, in initialize self.load_model() File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_executor/model_runner.py", line 909, in load_model self.model = self.loader.load_model( ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_loader/loader.py", line 666, in load_model model = _initialize_model( ^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/model_loader/loader.py", line 281, in _initialize_model return model_class(**kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 1030, in __init__ self.model = Glm4MoeModel( ^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 919, in __init__ self.layers, self.start_layer, self.end_layer = make_layers( ^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/utils/common.py", line 643, in make_layers + get_offloader().wrap_modules( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/utils/offloader.py", line 36, in wrap_modules return list(all_modules_generator) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/utils/common.py", line 645, in <genexpr> layer_fn(idx=idx, prefix=add_prefix(idx, prefix)) File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 921, in <lambda> lambda idx, prefix: Glm4MoeDecoderLayer( ^^^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 740, in __init__ self.mlp = Glm4MoeSparseMoeBlock( ^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/models/glm4_moe.py", line 374, in __init__ self.experts = get_moe_impl_class(quant_config)( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/alex/software/ktransformers/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 292, in __init__ self.quant_method.create_weights( File "/home/alex/software/ktransformers/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 1980, in create_weights self.wrapper = KTMoEWrapper( ^^^^^^^^^^^^^ File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/kt_kernel/experts.py", line 106, in __new__ return backend_cls( ^^^^^^^^^^^^ File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 82, in __init__ super().__init__( File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/kt_kernel/experts_base.py", line 203, in __init__ self.gpu_experts_mask.copy_(gpu_experts_mask) File "/home/alex/.pyenv/versions/3.11.14/lib/python3.11/site-packages/torch/utils/_device.py", line 103, in __torch_function__ return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ RuntimeError: The size of tensor a (161) must match the size of tensor b (160) at non-singleton dimension 0 [2026-02-08 04:01:01] Received sigquit from a child process. It usually means the child failed. ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:49:15 +00:00
Author
Owner
Copy link

@SkqLiao commented on GitHub (Feb 8, 2026):

Try adding --disable-shared-experts-fusion in your launch command.

<!-- gh-comment-id:3867144696 --> @SkqLiao commented on GitHub (Feb 8, 2026): Try adding `--disable-shared-experts-fusion` in your launch command.
bsakaguchi commented 2026-10-29 14:49:16 +00:00
Author
Owner
Copy link

@Gracana commented on GitHub (Feb 10, 2026):

Yep, that fixes the problem. Thanks!

<!-- gh-comment-id:3878136680 --> @Gracana commented on GitHub (Feb 10, 2026): Yep, that fixes the problem. Thanks!
bsakaguchi commented 2026-10-29 14:49:16 +00:00
Author
Owner
Copy link

@ErvinXie commented on GitHub (Feb 28, 2026):

已由 --disable-shared-experts-fusion 解决，关闭。

This reply was co-authored by an AI agent and a human.

<!-- gh-comment-id:3976700024 --> @ErvinXie commented on GitHub (Feb 28, 2026): 已由 `--disable-shared-experts-fusion` 解决，关闭。 --- *This reply was co-authored by an AI agent and a human.*
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1147
No description provided.