[GH-ISSUE #1741] GLM-4.7 Server starts but errors out at inferencing #1101

New issue

Open

opened 2026-10-29 14:48:33 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented 2026-10-29 14:48:33 +00:00

Owner

Copy link

Originally created by @mtcl on GitHub (Dec 23, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1741

Reminder

• I have read the above rules and searched the existing issues.

System Info

Pulled both kt-kernel and sglang on December 23 2025 and setup fresh conda environment and built it. When processing a prompt it fails.

System Specifications for Hybrid CPU/GPU Inference

System Overview

OS: Ubuntu 24.04.3 LTS
Kernel: Linux 6.14.0-37-generic

---

Motherboard Specifications

Model: ASUS Pro WS W790E-SAGE SE

• Platform: Intel W790 Chipset
• Socket: LGA4677 (supports Intel Xeon W-3000 series)
• Memory Slots: 8x DDR5 DIMM slots
• Memory Speed: Support for DDR5-4800+ (OC capable)
• PCIe Slots: 7 PCIe 5.0 x16 slots for dual GPU setup
• Form Factor: Extended ATX

---

CPU Specifications

Processor: Intel Xeon w9-3495X

• Cores: 56 physical cores / 112 threads
• Base Frequency: 800 MHz
• Max Frequency: 4.8 GHz
• AVX-512 Support: Full support
• AMX (Advanced Matrix Extensions): Full support
• NUMA Nodes: 1

---

GPU Specifications

Graphics Cards: 2x NVIDIA RTX PRO 6000 Black Edition

• VRAM per GPU: 96 GB
• Driver Version: 580.95.05
• CUDA Version: 13.0
• Power Consumption: 600W max each

---

Memory Configuration

Total RAM: 512 GB
Memory Type: DDR5 (8x DIMM slots populated)
Memory Speed: DDR5-4800+ (OC capable)
OC Supported: 6000 OC Stable
Swap: 0 GB (disabled)
Memory Architecture: 8-channel DDR5 configuration
Available for Inference: 503 GB total capacity for model loading

---

Storage Configuration

Primary Storage:

• System Drive: WD_BLACK SN850X 2000GB (1.8TB)
  • Mount: / (root filesystem)
• Data Drive: Samsung SSD 9100 PRO 4TB (3.6TB)
  • Mount: /media/mukul/data

Secondary Storage:

• Crucial Drive: CT4000X10PROSSD9 (3.6TB)
  • Mount: /media/mukul/crucial
• External T7: PSSD T7 Shield (3.6TB)
  • Mount: /media/mukul/t7

---

Reproduction

```bash
(kt-kernel) mukul@jarvis:~$ CUDA_VISIBLE_DEVICES=0 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0
python3 -m sglang.launch_server \
--host 0.0.0.0 \
--port 10002 \
--model /media/mukul/data/models/zai-org/GLM-4.7 \
--trust-remote-code \
--tool-call-parser glm \
--reasoning-parser glm45 \
--mem-fraction-static 0.92 \
--chunked-prefill-size 4096 \
--served-model-name jarvis-thinker \
--enable-mixed-chunk \
--attention-backend flashinfer \
--kt-method AMXINT4 \
--kt-weight-path /media/mukul/data/models/zai-org/GLM-4.7-AMXINT4 \
--kt-cpuinfer 56 \
--kt-threadpool-count 1 \
--kt-num-gpu-experts 4 \
--kt-max-deferred-experts-per-token 1
[2025-12-23 03:31:05] server_args=ServerArgs(model_path='/media/mukul/data/models/zai-org/GLM-4.7', tokenizer_path='/media/mukul/data/models/zai-org/GLM-4.7', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, limit_mm_data_per_request=None, revision=None, model_impl='auto', host='0.0.0.0', port=10002, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.92, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=709364858, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='jarvis-thinker', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='glm45', tool_call_parser='glm', tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=True, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='/media/mukul/data/models/zai-org/GLM-4.7-AMXINT4', kt_method='AMXINT4', kt_cpuinfer=56, kt_threadpool_count=1, kt_num_gpu_experts=4, kt_max_deferred_experts_per_token=1, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_enable_fake_auto=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2025-12-23 03:31:06] Using default HuggingFace chat template with detected content format: openai
[2025-12-23 03:31:11] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-12-23 03:31:11] Init torch distributed ends. mem usage=0.00 GB
[2025-12-23 03:31:12] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmasrConfig' from 'transformers' (/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/transformers/__init__.py)
[2025-12-23 03:31:12] Load weight begin. avail mem=93.24 GB
[2025-12-23 03:31:12] Shared experts fusion optimization enabled.
CPUInfer[0x122eae50]: Hello
WorkerPool[0x24291e90] 1 subpools, [numa:threads][0:56] 
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 56 threads
Loading safetensors checkpoint shards:   0% Completed | 0/93 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   1% Completed | 1/93 [00:00<00:27,  3.34it/s]
Loading safetensors checkpoint shards:   2% Completed | 2/93 [00:00<00:28,  3.18it/s]
Loading safetensors checkpoint shards:   3% Completed | 3/93 [00:00<00:28,  3.16it/s]
Loading safetensors checkpoint shards:   4% Completed | 4/93 [00:01<00:27,  3.18it/s]
Loading safetensors checkpoint shards:   5% Completed | 5/93 [00:01<00:27,  3.19it/s]
Loading safetensors checkpoint shards:   6% Completed | 6/93 [00:01<00:28,  3.10it/s]
Loading safetensors checkpoint shards:   8% Completed | 7/93 [00:02<00:28,  3.05it/s]
Loading safetensors checkpoint shards:   9% Completed | 8/93 [00:02<00:27,  3.05it/s]
Loading safetensors checkpoint shards:  10% Completed | 9/93 [00:02<00:27,  3.04it/s]
Loading safetensors checkpoint shards:  11% Completed | 10/93 [00:03<00:27,  3.05it/s]
Loading safetensors checkpoint shards:  12% Completed | 11/93 [00:03<00:26,  3.06it/s]
Loading safetensors checkpoint shards:  13% Completed | 12/93 [00:03<00:25,  3.12it/s]
Loading safetensors checkpoint shards:  14% Completed | 13/93 [00:04<00:25,  3.14it/s]
Loading safetensors checkpoint shards:  15% Completed | 14/93 [00:04<00:25,  3.14it/s]
Loading safetensors checkpoint shards:  16% Completed | 15/93 [00:04<00:24,  3.16it/s]
Loading safetensors checkpoint shards:  17% Completed | 16/93 [00:05<00:23,  3.26it/s]
Loading safetensors checkpoint shards:  18% Completed | 17/93 [00:05<00:18,  4.07it/s]
Loading safetensors checkpoint shards:  19% Completed | 18/93 [00:05<00:19,  3.84it/s]
Loading safetensors checkpoint shards:  20% Completed | 19/93 [00:05<00:19,  3.79it/s]
Loading safetensors checkpoint shards:  22% Completed | 20/93 [00:06<00:19,  3.77it/s]
Loading safetensors checkpoint shards:  23% Completed | 21/93 [00:06<00:20,  3.57it/s]
Loading safetensors checkpoint shards:  24% Completed | 22/93 [00:06<00:19,  3.60it/s]
Loading safetensors checkpoint shards:  25% Completed | 23/93 [00:06<00:19,  3.67it/s]
Loading safetensors checkpoint shards:  26% Completed | 24/93 [00:07<00:18,  3.76it/s]
Loading safetensors checkpoint shards:  27% Completed | 25/93 [00:07<00:19,  3.56it/s]
Loading safetensors checkpoint shards:  28% Completed | 26/93 [00:07<00:19,  3.49it/s]
Loading safetensors checkpoint shards:  29% Completed | 27/93 [00:08<00:18,  3.56it/s]
Loading safetensors checkpoint shards:  30% Completed | 28/93 [00:08<00:18,  3.60it/s]
Loading safetensors checkpoint shards:  31% Completed | 29/93 [00:08<00:16,  3.92it/s]
Loading safetensors checkpoint shards:  32% Completed | 30/93 [00:08<00:15,  4.09it/s]
Loading safetensors checkpoint shards:  33% Completed | 31/93 [00:09<00:16,  3.80it/s]
Loading safetensors checkpoint shards:  34% Completed | 32/93 [00:09<00:16,  3.63it/s]
Loading safetensors checkpoint shards:  35% Completed | 33/93 [00:09<00:16,  3.57it/s]
Loading safetensors checkpoint shards:  37% Completed | 34/93 [00:09<00:16,  3.48it/s]
Loading safetensors checkpoint shards:  38% Completed | 35/93 [00:10<00:16,  3.43it/s]
Loading safetensors checkpoint shards:  39% Completed | 36/93 [00:10<00:16,  3.37it/s]
Loading safetensors checkpoint shards:  40% Completed | 37/93 [00:10<00:16,  3.34it/s]
Loading safetensors checkpoint shards:  41% Completed | 38/93 [00:11<00:16,  3.32it/s]
Loading safetensors checkpoint shards:  42% Completed | 39/93 [00:11<00:16,  3.29it/s]
Loading safetensors checkpoint shards:  43% Completed | 40/93 [00:11<00:16,  3.28it/s]
Loading safetensors checkpoint shards:  44% Completed | 41/93 [00:12<00:15,  3.26it/s]
Loading safetensors checkpoint shards:  45% Completed | 42/93 [00:12<00:15,  3.22it/s]
Loading safetensors checkpoint shards:  46% Completed | 43/93 [00:12<00:15,  3.22it/s]
Loading safetensors checkpoint shards:  47% Completed | 44/93 [00:12<00:15,  3.23it/s]
Loading safetensors checkpoint shards:  48% Completed | 45/93 [00:13<00:14,  3.22it/s]
Loading safetensors checkpoint shards:  49% Completed | 46/93 [00:13<00:14,  3.22it/s]
Loading safetensors checkpoint shards:  51% Completed | 47/93 [00:13<00:14,  3.20it/s]
Loading safetensors checkpoint shards:  52% Completed | 48/93 [00:14<00:13,  3.33it/s]
Loading safetensors checkpoint shards:  53% Completed | 49/93 [00:14<00:13,  3.29it/s]
Loading safetensors checkpoint shards:  54% Completed | 50/93 [00:14<00:13,  3.27it/s]
Loading safetensors checkpoint shards:  55% Completed | 51/93 [00:15<00:12,  3.28it/s]
Loading safetensors checkpoint shards:  56% Completed | 52/93 [00:15<00:12,  3.28it/s]
Loading safetensors checkpoint shards:  57% Completed | 53/93 [00:15<00:12,  3.30it/s]
Loading safetensors checkpoint shards:  58% Completed | 54/93 [00:16<00:11,  3.30it/s]
Loading safetensors checkpoint shards:  59% Completed | 55/93 [00:16<00:11,  3.29it/s]
Loading safetensors checkpoint shards:  60% Completed | 56/93 [00:16<00:10,  3.40it/s]
Loading safetensors checkpoint shards:  61% Completed | 57/93 [00:16<00:10,  3.35it/s]
Loading safetensors checkpoint shards:  62% Completed | 58/93 [00:17<00:10,  3.31it/s]
Loading safetensors checkpoint shards:  63% Completed | 59/93 [00:17<00:10,  3.27it/s]
Loading safetensors checkpoint shards:  65% Completed | 60/93 [00:17<00:10,  3.28it/s]
Loading safetensors checkpoint shards:  66% Completed | 61/93 [00:18<00:09,  3.25it/s]
Loading safetensors checkpoint shards:  67% Completed | 62/93 [00:18<00:09,  3.40it/s]
Loading safetensors checkpoint shards:  68% Completed | 63/93 [00:18<00:08,  3.35it/s]
Loading safetensors checkpoint shards:  69% Completed | 64/93 [00:19<00:08,  3.33it/s]
Loading safetensors checkpoint shards:  70% Completed | 65/93 [00:19<00:08,  3.33it/s]
Loading safetensors checkpoint shards:  71% Completed | 66/93 [00:20<00:11,  2.30it/s]
Loading safetensors checkpoint shards:  72% Completed | 67/93 [00:20<00:10,  2.60it/s]
Loading safetensors checkpoint shards:  73% Completed | 68/93 [00:20<00:08,  2.79it/s]
Loading safetensors checkpoint shards:  74% Completed | 69/93 [00:20<00:08,  2.91it/s]
Loading safetensors checkpoint shards:  75% Completed | 70/93 [00:21<00:07,  3.00it/s]
Loading safetensors checkpoint shards:  76% Completed | 71/93 [00:21<00:07,  3.07it/s]
Loading safetensors checkpoint shards:  77% Completed | 72/93 [00:21<00:06,  3.15it/s]
Loading safetensors checkpoint shards:  78% Completed | 73/93 [00:22<00:06,  3.19it/s]
Loading safetensors checkpoint shards:  80% Completed | 74/93 [00:22<00:05,  3.23it/s]
Loading safetensors checkpoint shards:  81% Completed | 75/93 [00:22<00:05,  3.24it/s]
Loading safetensors checkpoint shards:  82% Completed | 76/93 [00:23<00:05,  3.26it/s]
Loading safetensors checkpoint shards:  83% Completed | 77/93 [00:23<00:04,  3.24it/s]
Loading safetensors checkpoint shards:  84% Completed | 78/93 [00:23<00:04,  3.24it/s]
Loading safetensors checkpoint shards:  85% Completed | 79/93 [00:24<00:04,  3.25it/s]
Loading safetensors checkpoint shards:  86% Completed | 80/93 [00:24<00:03,  3.29it/s]
Loading safetensors checkpoint shards:  87% Completed | 81/93 [00:24<00:03,  3.29it/s]
Loading safetensors checkpoint shards:  88% Completed | 82/93 [00:24<00:02,  3.91it/s]
Loading safetensors checkpoint shards:  89% Completed | 83/93 [00:24<00:02,  4.76it/s]
Loading safetensors checkpoint shards:  90% Completed | 84/93 [00:25<00:02,  4.06it/s]
Loading safetensors checkpoint shards:  91% Completed | 85/93 [00:25<00:01,  4.61it/s]
Loading safetensors checkpoint shards:  92% Completed | 86/93 [00:25<00:01,  5.47it/s]
Loading safetensors checkpoint shards:  94% Completed | 87/93 [00:25<00:00,  6.31it/s]
Loading safetensors checkpoint shards:  95% Completed | 88/93 [00:25<00:00,  5.33it/s]
Loading safetensors checkpoint shards:  96% Completed | 89/93 [00:25<00:00,  6.14it/s]
Loading safetensors checkpoint shards:  97% Completed | 90/93 [00:26<00:00,  6.85it/s]
Loading safetensors checkpoint shards:  98% Completed | 91/93 [00:26<00:00,  7.49it/s]
Loading safetensors checkpoint shards:  99% Completed | 92/93 [00:26<00:00,  8.07it/s]
Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:26<00:00,  3.53it/s]

TP MOE layer 3, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 4, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 5, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 6, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 7, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 8, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 9, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 10, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 11, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 12, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 13, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 14, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 15, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 16, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 17, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 18, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 19, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 20, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 21, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 22, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 23, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 24, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 25, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 26, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 27, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 28, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 29, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 30, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 31, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 32, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 33, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 34, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 35, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 36, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 37, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 38, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 39, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 40, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 41, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 42, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 43, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 44, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 45, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 46, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 47, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 48, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 49, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 50, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 51, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 52, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 53, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 54, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 55, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 56, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 57, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 58, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 59, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 60, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 61, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 62, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 63, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 64, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 65, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 66, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 67, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 68, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 69, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 70, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 71, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 72, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 73, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 74, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 75, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 76, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 77, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 78, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 79, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 80, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 81, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 82, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 83, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 84, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 85, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 86, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 87, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 88, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 89, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 90, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 91, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
[2025-12-23 03:31:56] Load weight end. type=Glm4MoeForCausalLM, dtype=torch.bfloat16, avail mem=50.02 GB, mem usage=43.22 GB.
[2025-12-23 03:31:56] Using KV cache dtype: torch.bfloat16
[2025-12-23 03:31:56] The available memory for KV cache is 42.55 GB.
[2025-12-23 03:31:56] KV Cache is allocated. #tokens: 121240, K size: 21.27 GB, V size: 21.27 GB
[2025-12-23 03:31:56] Memory pool end. avail mem=5.70 GB
[2025-12-23 03:31:56] Capture cuda graph begin. This can take up to several minutes. avail mem=5.21 GB
[2025-12-23 03:31:56] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256]
Capturing batches (bs=256 avail_mem=4.83 GB):   0%|                       | 0/36 [00:00<?, ?it/s][2025-12-23 03:31:58] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=4,N=1536,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
[2025-12-23 03:31:58] Using MoE kernel config with down_moe=False. Performance might be sub-optimal! Config file not found at /media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=4,N=1536,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition_down.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton
Capturing batches (bs=1 avail_mem=3.16 GB): 100%|████████████████| 36/36 [00:33<00:00,  1.07it/s]
[2025-12-23 03:32:30] Capture cuda graph end. Time elapsed: 34.24 s. mem usage=2.11 GB. avail mem=3.11 GB.
[2025-12-23 03:32:31] max_total_num_tokens=121240, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2048, context_len=202752, available_gpu_mem=3.11 GB
[2025-12-23 03:32:31] INFO:     Started server process [2364495]
[2025-12-23 03:32:31] INFO:     Waiting for application startup.
[2025-12-23 03:32:31] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0}
[2025-12-23 03:32:31] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0}
[2025-12-23 03:32:31] INFO:     Application startup complete.
[2025-12-23 03:32:31] INFO:     Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit)
[2025-12-23 03:32:32] INFO:     127.0.0.1:47222 - "GET /model_info HTTP/1.1" 200 OK
[2025-12-23 03:32:32] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-12-23 03:32:33] INFO:     127.0.0.1:47232 - "POST /generate HTTP/1.1" 200 OK
[2025-12-23 03:32:33] The server is fired up and ready to roll!
[2025-12-23 03:32:47] INFO:     172.18.0.3:50466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-23 03:32:47] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-12-23 03:33:03] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.03, #running-req: 0, #queue-req: 0, 
[2025-12-23 03:33:04] Scheduler hit an exception: Traceback (most recent call last):
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/scheduler.py", line 2870, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/scheduler.py", line 1110, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/scheduler.py", line 2152, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/tp_worker.py", line 421, in forward_batch_generation
    out = self.model_runner.forward(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/model_executor/model_runner.py", line 2889, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/model_executor/model_runner.py", line 2967, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/model_executor/model_runner.py", line 2834, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 1076, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 967, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 784, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 322, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 307, in forward_core
    attn_output = self.attn(*inner_state)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/radix_attention.py", line 123, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/attention/base_attn_backend.py", line 113, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 840, in forward_extend
    o, _ = merge_state(o1, s1, o2, s2)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/cascade.py", line 86, in merge_state
    get_cascade_module().merge_state(v_a, s_a, v_b, s_b, v_merged, s_merged)
  File "python/tvm_ffi/cython/function.pxi", line 923, in tvm_ffi.core.Function.__call__
  File "<unknown>", line 0, in __tvm_ffi_merge_state
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/data/csrc/cascade.cu", line 53, in merge_state(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView)::<lambda()>::<lambda()>
    TVM_FFI_ICHECK(status == cudaSuccess)
    
RuntimeError: Check failed: (status == cudaSuccess) is false: MergeState kernel launch failed: invalid argument

[2025-12-23 03:33:04] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
Killed

### Others

_No response_

Originally created by @mtcl on GitHub (Dec 23, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1741 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info Pulled both kt-kernel and sglang on December 23 2025 and setup fresh conda environment and built it. When processing a prompt it fails. # System Specifications for Hybrid CPU/GPU Inference ## System Overview **OS:** Ubuntu 24.04.3 LTS **Kernel:** Linux 6.14.0-37-generic --- ## Motherboard Specifications **Model:** ASUS Pro WS W790E-SAGE SE - **Platform:** Intel W790 Chipset - **Socket:** LGA4677 (supports Intel Xeon W-3000 series) - **Memory Slots:** 8x DDR5 DIMM slots - **Memory Speed:** Support for DDR5-4800+ (OC capable) - **PCIe Slots:** 7 PCIe 5.0 x16 slots for dual GPU setup - **Form Factor:** Extended ATX --- ## CPU Specifications **Processor:** Intel Xeon w9-3495X - **Cores:** 56 physical cores / 112 threads - **Base Frequency:** 800 MHz - **Max Frequency:** 4.8 GHz - **AVX-512 Support:** Full support - **AMX (Advanced Matrix Extensions):** Full support - **NUMA Nodes:** 1 --- ## GPU Specifications **Graphics Cards:** 2x NVIDIA RTX PRO 6000 Black Edition - **VRAM per GPU:** 96 GB - **Driver Version:** 580.95.05 - **CUDA Version:** 13.0 - **Power Consumption:** 600W max each --- ## Memory Configuration **Total RAM:** 512 GB **Memory Type:** DDR5 (8x DIMM slots populated) **Memory Speed:** DDR5-4800+ (OC capable) **OC Supported:** 6000 OC Stable **Swap:** 0 GB (disabled) **Memory Architecture:** 8-channel DDR5 configuration **Available for Inference:** 503 GB total capacity for model loading --- ## Storage Configuration **Primary Storage:** - **System Drive:** WD_BLACK SN850X 2000GB (1.8TB) - Mount: `/` (root filesystem) - **Data Drive:** Samsung SSD 9100 PRO 4TB (3.6TB) - Mount: `/media/mukul/data` **Secondary Storage:** - **Crucial Drive:** CT4000X10PROSSD9 (3.6TB) - Mount: `/media/mukul/crucial` - **External T7:** PSSD T7 Shield (3.6TB) - Mount: `/media/mukul/t7` --- ### Reproduction ```text ```bash (kt-kernel) mukul@jarvis:~$ CUDA_VISIBLE_DEVICES=0 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0 python3 -m sglang.launch_server \ --host 0.0.0.0 \ --port 10002 \ --model /media/mukul/data/models/zai-org/GLM-4.7 \ --trust-remote-code \ --tool-call-parser glm \ --reasoning-parser glm45 \ --mem-fraction-static 0.92 \ --chunked-prefill-size 4096 \ --served-model-name jarvis-thinker \ --enable-mixed-chunk \ --attention-backend flashinfer \ --kt-method AMXINT4 \ --kt-weight-path /media/mukul/data/models/zai-org/GLM-4.7-AMXINT4 \ --kt-cpuinfer 56 \ --kt-threadpool-count 1 \ --kt-num-gpu-experts 4 \ --kt-max-deferred-experts-per-token 1 [2025-12-23 03:31:05] server_args=ServerArgs(model_path='/media/mukul/data/models/zai-org/GLM-4.7', tokenizer_path='/media/mukul/data/models/zai-org/GLM-4.7', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, limit_mm_data_per_request=None, revision=None, model_impl='auto', host='0.0.0.0', port=10002, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.92, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=709364858, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='jarvis-thinker', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='glm45', tool_call_parser='glm', tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='flashinfer', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=True, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='/media/mukul/data/models/zai-org/GLM-4.7-AMXINT4', kt_method='AMXINT4', kt_cpuinfer=56, kt_threadpool_count=1, kt_num_gpu_experts=4, kt_max_deferred_experts_per_token=1, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, disaggregation_decode_enable_fake_auto=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None) [2025-12-23 03:31:06] Using default HuggingFace chat template with detected content format: openai [2025-12-23 03:31:11] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2025-12-23 03:31:11] Init torch distributed ends. mem usage=0.00 GB [2025-12-23 03:31:12] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmasrConfig' from 'transformers' (/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/transformers/__init__.py) [2025-12-23 03:31:12] Load weight begin. avail mem=93.24 GB [2025-12-23 03:31:12] Shared experts fusion optimization enabled. CPUInfer[0x122eae50]: Hello WorkerPool[0x24291e90] 1 subpools, [numa:threads][0:56] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 56 threads Loading safetensors checkpoint shards: 0% Completed | 0/93 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 1% Completed | 1/93 [00:00<00:27, 3.34it/s] Loading safetensors checkpoint shards: 2% Completed | 2/93 [00:00<00:28, 3.18it/s] Loading safetensors checkpoint shards: 3% Completed | 3/93 [00:00<00:28, 3.16it/s] Loading safetensors checkpoint shards: 4% Completed | 4/93 [00:01<00:27, 3.18it/s] Loading safetensors checkpoint shards: 5% Completed | 5/93 [00:01<00:27, 3.19it/s] Loading safetensors checkpoint shards: 6% Completed | 6/93 [00:01<00:28, 3.10it/s] Loading safetensors checkpoint shards: 8% Completed | 7/93 [00:02<00:28, 3.05it/s] Loading safetensors checkpoint shards: 9% Completed | 8/93 [00:02<00:27, 3.05it/s] Loading safetensors checkpoint shards: 10% Completed | 9/93 [00:02<00:27, 3.04it/s] Loading safetensors checkpoint shards: 11% Completed | 10/93 [00:03<00:27, 3.05it/s] Loading safetensors checkpoint shards: 12% Completed | 11/93 [00:03<00:26, 3.06it/s] Loading safetensors checkpoint shards: 13% Completed | 12/93 [00:03<00:25, 3.12it/s] Loading safetensors checkpoint shards: 14% Completed | 13/93 [00:04<00:25, 3.14it/s] Loading safetensors checkpoint shards: 15% Completed | 14/93 [00:04<00:25, 3.14it/s] Loading safetensors checkpoint shards: 16% Completed | 15/93 [00:04<00:24, 3.16it/s] Loading safetensors checkpoint shards: 17% Completed | 16/93 [00:05<00:23, 3.26it/s] Loading safetensors checkpoint shards: 18% Completed | 17/93 [00:05<00:18, 4.07it/s] Loading safetensors checkpoint shards: 19% Completed | 18/93 [00:05<00:19, 3.84it/s] Loading safetensors checkpoint shards: 20% Completed | 19/93 [00:05<00:19, 3.79it/s] Loading safetensors checkpoint shards: 22% Completed | 20/93 [00:06<00:19, 3.77it/s] Loading safetensors checkpoint shards: 23% Completed | 21/93 [00:06<00:20, 3.57it/s] Loading safetensors checkpoint shards: 24% Completed | 22/93 [00:06<00:19, 3.60it/s] Loading safetensors checkpoint shards: 25% Completed | 23/93 [00:06<00:19, 3.67it/s] Loading safetensors checkpoint shards: 26% Completed | 24/93 [00:07<00:18, 3.76it/s] Loading safetensors checkpoint shards: 27% Completed | 25/93 [00:07<00:19, 3.56it/s] Loading safetensors checkpoint shards: 28% Completed | 26/93 [00:07<00:19, 3.49it/s] Loading safetensors checkpoint shards: 29% Completed | 27/93 [00:08<00:18, 3.56it/s] Loading safetensors checkpoint shards: 30% Completed | 28/93 [00:08<00:18, 3.60it/s] Loading safetensors checkpoint shards: 31% Completed | 29/93 [00:08<00:16, 3.92it/s] Loading safetensors checkpoint shards: 32% Completed | 30/93 [00:08<00:15, 4.09it/s] Loading safetensors checkpoint shards: 33% Completed | 31/93 [00:09<00:16, 3.80it/s] Loading safetensors checkpoint shards: 34% Completed | 32/93 [00:09<00:16, 3.63it/s] Loading safetensors checkpoint shards: 35% Completed | 33/93 [00:09<00:16, 3.57it/s] Loading safetensors checkpoint shards: 37% Completed | 34/93 [00:09<00:16, 3.48it/s] Loading safetensors checkpoint shards: 38% Completed | 35/93 [00:10<00:16, 3.43it/s] Loading safetensors checkpoint shards: 39% Completed | 36/93 [00:10<00:16, 3.37it/s] Loading safetensors checkpoint shards: 40% Completed | 37/93 [00:10<00:16, 3.34it/s] Loading safetensors checkpoint shards: 41% Completed | 38/93 [00:11<00:16, 3.32it/s] Loading safetensors checkpoint shards: 42% Completed | 39/93 [00:11<00:16, 3.29it/s] Loading safetensors checkpoint shards: 43% Completed | 40/93 [00:11<00:16, 3.28it/s] Loading safetensors checkpoint shards: 44% Completed | 41/93 [00:12<00:15, 3.26it/s] Loading safetensors checkpoint shards: 45% Completed | 42/93 [00:12<00:15, 3.22it/s] Loading safetensors checkpoint shards: 46% Completed | 43/93 [00:12<00:15, 3.22it/s] Loading safetensors checkpoint shards: 47% Completed | 44/93 [00:12<00:15, 3.23it/s] Loading safetensors checkpoint shards: 48% Completed | 45/93 [00:13<00:14, 3.22it/s] Loading safetensors checkpoint shards: 49% Completed | 46/93 [00:13<00:14, 3.22it/s] Loading safetensors checkpoint shards: 51% Completed | 47/93 [00:13<00:14, 3.20it/s] Loading safetensors checkpoint shards: 52% Completed | 48/93 [00:14<00:13, 3.33it/s] Loading safetensors checkpoint shards: 53% Completed | 49/93 [00:14<00:13, 3.29it/s] Loading safetensors checkpoint shards: 54% Completed | 50/93 [00:14<00:13, 3.27it/s] Loading safetensors checkpoint shards: 55% Completed | 51/93 [00:15<00:12, 3.28it/s] Loading safetensors checkpoint shards: 56% Completed | 52/93 [00:15<00:12, 3.28it/s] Loading safetensors checkpoint shards: 57% Completed | 53/93 [00:15<00:12, 3.30it/s] Loading safetensors checkpoint shards: 58% Completed | 54/93 [00:16<00:11, 3.30it/s] Loading safetensors checkpoint shards: 59% Completed | 55/93 [00:16<00:11, 3.29it/s] Loading safetensors checkpoint shards: 60% Completed | 56/93 [00:16<00:10, 3.40it/s] Loading safetensors checkpoint shards: 61% Completed | 57/93 [00:16<00:10, 3.35it/s] Loading safetensors checkpoint shards: 62% Completed | 58/93 [00:17<00:10, 3.31it/s] Loading safetensors checkpoint shards: 63% Completed | 59/93 [00:17<00:10, 3.27it/s] Loading safetensors checkpoint shards: 65% Completed | 60/93 [00:17<00:10, 3.28it/s] Loading safetensors checkpoint shards: 66% Completed | 61/93 [00:18<00:09, 3.25it/s] Loading safetensors checkpoint shards: 67% Completed | 62/93 [00:18<00:09, 3.40it/s] Loading safetensors checkpoint shards: 68% Completed | 63/93 [00:18<00:08, 3.35it/s] Loading safetensors checkpoint shards: 69% Completed | 64/93 [00:19<00:08, 3.33it/s] Loading safetensors checkpoint shards: 70% Completed | 65/93 [00:19<00:08, 3.33it/s] Loading safetensors checkpoint shards: 71% Completed | 66/93 [00:20<00:11, 2.30it/s] Loading safetensors checkpoint shards: 72% Completed | 67/93 [00:20<00:10, 2.60it/s] Loading safetensors checkpoint shards: 73% Completed | 68/93 [00:20<00:08, 2.79it/s] Loading safetensors checkpoint shards: 74% Completed | 69/93 [00:20<00:08, 2.91it/s] Loading safetensors checkpoint shards: 75% Completed | 70/93 [00:21<00:07, 3.00it/s] Loading safetensors checkpoint shards: 76% Completed | 71/93 [00:21<00:07, 3.07it/s] Loading safetensors checkpoint shards: 77% Completed | 72/93 [00:21<00:06, 3.15it/s] Loading safetensors checkpoint shards: 78% Completed | 73/93 [00:22<00:06, 3.19it/s] Loading safetensors checkpoint shards: 80% Completed | 74/93 [00:22<00:05, 3.23it/s] Loading safetensors checkpoint shards: 81% Completed | 75/93 [00:22<00:05, 3.24it/s] Loading safetensors checkpoint shards: 82% Completed | 76/93 [00:23<00:05, 3.26it/s] Loading safetensors checkpoint shards: 83% Completed | 77/93 [00:23<00:04, 3.24it/s] Loading safetensors checkpoint shards: 84% Completed | 78/93 [00:23<00:04, 3.24it/s] Loading safetensors checkpoint shards: 85% Completed | 79/93 [00:24<00:04, 3.25it/s] Loading safetensors checkpoint shards: 86% Completed | 80/93 [00:24<00:03, 3.29it/s] Loading safetensors checkpoint shards: 87% Completed | 81/93 [00:24<00:03, 3.29it/s] Loading safetensors checkpoint shards: 88% Completed | 82/93 [00:24<00:02, 3.91it/s] Loading safetensors checkpoint shards: 89% Completed | 83/93 [00:24<00:02, 4.76it/s] Loading safetensors checkpoint shards: 90% Completed | 84/93 [00:25<00:02, 4.06it/s] Loading safetensors checkpoint shards: 91% Completed | 85/93 [00:25<00:01, 4.61it/s] Loading safetensors checkpoint shards: 92% Completed | 86/93 [00:25<00:01, 5.47it/s] Loading safetensors checkpoint shards: 94% Completed | 87/93 [00:25<00:00, 6.31it/s] Loading safetensors checkpoint shards: 95% Completed | 88/93 [00:25<00:00, 5.33it/s] Loading safetensors checkpoint shards: 96% Completed | 89/93 [00:25<00:00, 6.14it/s] Loading safetensors checkpoint shards: 97% Completed | 90/93 [00:26<00:00, 6.85it/s] Loading safetensors checkpoint shards: 98% Completed | 91/93 [00:26<00:00, 7.49it/s] Loading safetensors checkpoint shards: 99% Completed | 92/93 [00:26<00:00, 8.07it/s] Loading safetensors checkpoint shards: 100% Completed | 93/93 [00:26<00:00, 3.53it/s] TP MOE layer 3, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 4, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 5, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 6, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 7, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 8, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 9, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 10, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 11, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 12, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 13, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 14, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 15, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 16, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 17, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 18, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 19, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 20, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 21, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 22, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 23, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 24, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 25, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 26, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 27, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 28, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 29, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 30, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 31, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 32, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 33, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 34, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 35, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 36, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 37, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 38, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 39, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 40, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 41, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 42, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 43, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 44, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 45, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 46, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 47, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 48, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 49, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 50, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 51, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 52, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 53, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 54, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 55, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 56, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 57, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 58, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 59, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 60, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 61, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 62, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 63, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 64, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 65, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 66, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 67, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 68, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 69, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 70, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 71, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 72, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 73, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 74, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 75, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 76, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 77, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 78, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 79, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 80, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 81, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 82, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 83, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 84, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 85, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 86, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 87, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 88, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 89, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 90, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 91, pool: 0x24291e90, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader [2025-12-23 03:31:56] Load weight end. type=Glm4MoeForCausalLM, dtype=torch.bfloat16, avail mem=50.02 GB, mem usage=43.22 GB. [2025-12-23 03:31:56] Using KV cache dtype: torch.bfloat16 [2025-12-23 03:31:56] The available memory for KV cache is 42.55 GB. [2025-12-23 03:31:56] KV Cache is allocated. #tokens: 121240, K size: 21.27 GB, V size: 21.27 GB [2025-12-23 03:31:56] Memory pool end. avail mem=5.70 GB [2025-12-23 03:31:56] Capture cuda graph begin. This can take up to several minutes. avail mem=5.21 GB [2025-12-23 03:31:56] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256] Capturing batches (bs=256 avail_mem=4.83 GB): 0%| | 0/36 [00:00<?, ?it/s][2025-12-23 03:31:58] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=4,N=1536,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton [2025-12-23 03:31:58] Using MoE kernel config with down_moe=False. Performance might be sub-optimal! Config file not found at /media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=4,N=1536,device_name=NVIDIA_RTX_PRO_6000_Blackwell_Workstation_Edition_down.json, you can create them with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton Capturing batches (bs=1 avail_mem=3.16 GB): 100%|████████████████| 36/36 [00:33<00:00, 1.07it/s] [2025-12-23 03:32:30] Capture cuda graph end. Time elapsed: 34.24 s. mem usage=2.11 GB. avail mem=3.11 GB. [2025-12-23 03:32:31] max_total_num_tokens=121240, chunked_prefill_size=4096, max_prefill_tokens=16384, max_running_requests=2048, context_len=202752, available_gpu_mem=3.11 GB [2025-12-23 03:32:31] INFO: Started server process [2364495] [2025-12-23 03:32:31] INFO: Waiting for application startup. [2025-12-23 03:32:31] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0} [2025-12-23 03:32:31] Using default chat sampling params from model generation config: {'repetition_penalty': 1.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0} [2025-12-23 03:32:31] INFO: Application startup complete. [2025-12-23 03:32:31] INFO: Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit) [2025-12-23 03:32:32] INFO: 127.0.0.1:47222 - "GET /model_info HTTP/1.1" 200 OK [2025-12-23 03:32:32] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-12-23 03:32:33] INFO: 127.0.0.1:47232 - "POST /generate HTTP/1.1" 200 OK [2025-12-23 03:32:33] The server is fired up and ready to roll! [2025-12-23 03:32:47] INFO: 172.18.0.3:50466 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2025-12-23 03:32:47] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-12-23 03:33:03] Prefill batch, #new-seq: 1, #new-token: 4096, #cached-token: 0, token usage: 0.03, #running-req: 0, #queue-req: 0, [2025-12-23 03:33:04] Scheduler hit an exception: Traceback (most recent call last): File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/scheduler.py", line 2870, in run_scheduler_process scheduler.event_loop_overlap() File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/scheduler.py", line 1110, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/scheduler.py", line 2152, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/managers/tp_worker.py", line 421, in forward_batch_generation out = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/model_executor/model_runner.py", line 2889, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/model_executor/model_runner.py", line 2967, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/model_executor/model_runner.py", line 2834, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 1076, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 967, in forward hidden_states, residual = layer( ^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 784, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 322, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/models/glm4_moe.py", line 307, in forward_core attn_output = self.attn(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/radix_attention.py", line 123, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/attention/base_attn_backend.py", line 113, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/media/mukul/data/dev-ai/sglang/python/sglang/srt/layers/attention/flashinfer_backend.py", line 840, in forward_extend o, _ = merge_state(o1, s1, o2, s2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/cascade.py", line 86, in merge_state get_cascade_module().merge_state(v_a, s_a, v_b, s_b, v_merged, s_merged) File "python/tvm_ffi/cython/function.pxi", line 923, in tvm_ffi.core.Function.__call__ File "<unknown>", line 0, in __tvm_ffi_merge_state File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/data/csrc/cascade.cu", line 53, in merge_state(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView)::<lambda()>::<lambda()> TVM_FFI_ICHECK(status == cudaSuccess) RuntimeError: Check failed: (status == cudaSuccess) is false: MergeState kernel launch failed: invalid argument [2025-12-23 03:33:04] SIGQUIT received. signum=None, frame=None. It usually means one child failed. Killed ``` ``` ### Others _No response_

bsakaguchi added the

bug

label 2026-10-29 14:48:33 +00:00

bsakaguchi commented 2026-10-29 14:48:33 +00:00

Author

Owner

Copy link

@mtcl commented on GitHub (Dec 30, 2025):

need help please!

<!-- gh-comment-id:3698509282 --> @mtcl commented on GitHub (Dec 30, 2025): need help please!

bsakaguchi commented 2026-10-29 14:48:34 +00:00

Author

Owner

Copy link

@mtcl commented on GitHub (Dec 31, 2025):

Is there any guide for GLM-4.7 for using with ktransformers? I do not know what I am missing...

<!-- gh-comment-id:3702877343 --> @mtcl commented on GitHub (Dec 31, 2025): Is there any guide for GLM-4.7 for using with ktransformers? I do not know what I am missing...

bsakaguchi commented 2026-10-29 14:48:34 +00:00

Author

Owner

Copy link

@sirgal commented on GitHub (Jan 24, 2026):

I have the same error with --attention-backend flashinfer, but it works for me with --attention-backend triton

<!-- gh-comment-id:3795426504 --> @sirgal commented on GitHub (Jan 24, 2026): I have the same error with `--attention-backend flashinfer`, but it works for me with `--attention-backend triton`

bsakaguchi referenced this issue 2026-10-29 14:53:29 +00:00

[PR #1101] [MERGED] Update llama4.md #1558

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#1101

No description provided.