kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1694] amx.py: Segmentation Fault in AMX loading process with GLM 4.6 using hybrid inference #1086

New issue
Open
opened 2026-10-29 14:48:20 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-29 14:48:20 +00:00
Owner
Copy link

Originally created by @reasv on GitHub (Dec 10, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1694

Originally assigned to: @ovowei on GitHub.

Reminder

  • I have read the above rules and searched the existing issues.

System Info

CPU: 2x Intel Xeon 8592+ ES Q2SP (60c per socket)
GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB)

KTransformers/kt-kernel: latest main pulled today (f992de55da)
SGLang: Version: 0.5.6.post1
torch: 2.9.1

OS: Arch Linux (kernel 6.17.8-arch1-1)
Driver Version: 580.105.08
CUDA Version: 13.0

Reproduction

/data/models/GLM-4.6-INT8-quant was quantized from 16-bit official weights using the current KTransformers script

CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server \
  --host 0.0.0.0 \
  --port 8000 \
  --attention-backend triton \
  --model /data/models/GLM-4.6-FP8 \
  --trust-remote-code \
  --mem-fraction-static 0.92 \
  --chunked-prefill-size 4096 \
  --served-model-name GLM-4.6 \
  --enable-mixed-chunk \
  --kt-method AMXINT8 \
  --kt-weight-path /data/models/GLM-4.6-INT8-quant \
  --kt-cpuinfer 120 \
  --kt-threadpool-count 4 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2

Output:

[2025-12-10 02:32:25] server_args=ServerArgs(model_path='/data/models/GLM-4.6-FP8', tokenizer_path='/data/models/GLM-4.6-FP8', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.92, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=209423600, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, mm_process_config={}, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='GLM-4.6', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', enable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_moe_a2a_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='/data/models/GLM-4.6-INT8-quant', kt_method='AMXINT8', kt_cpuinfer=120, kt_threadpool_count=4, kt_num_gpu_experts=32, kt_max_deferred_experts_per_token=2, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, decrypted_config_file=None, decrypted_draft_config_file=None, mm_enable_dp_encoder=False, forward_hooks=None)
[2025-12-10 02:32:26] Using default HuggingFace chat template with detected content format: openai
[2025-12-10 02:32:31] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2025-12-10 02:32:31] Init torch distributed ends. mem usage=0.00 GB
[2025-12-10 02:32:32] Ignore import error when loading sglang.srt.models.mindspore: name 'ms' is not defined
[2025-12-10 02:32:33] Load weight begin. avail mem=94.31 GB
[2025-12-10 02:32:33] Shared experts fusion optimization enabled.
CPUInfer[0x4efe2f10]: Hello
WorkerPool[0x4efe1a60] 4 subpools, [numa:threads][0:30] [1:30] [2:30] [3:30]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 2, 30 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 3, 30 threads
Loading safetensors checkpoint shards:   0% Completed | 0/93 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   1% Completed | 1/93 [00:00<01:03,  1.45it/s]
Loading safetensors checkpoint shards:   2% Completed | 2/93 [00:00<00:34,  2.65it/s]
Loading safetensors checkpoint shards:   3% Completed | 3/93 [00:00<00:23,  3.82it/s]
Loading safetensors checkpoint shards:   4% Completed | 4/93 [00:02<01:20,  1.10it/s]
Loading safetensors checkpoint shards:   5% Completed | 5/93 [00:04<01:52,  1.27s/it]
Loading safetensors checkpoint shards:   6% Completed | 6/93 [00:06<02:09,  1.49s/it]
Loading safetensors checkpoint shards:   8% Completed | 7/93 [00:08<02:19,  1.62s/it]
Loading safetensors checkpoint shards:   9% Completed | 8/93 [00:10<02:32,  1.79s/it]
Loading safetensors checkpoint shards:  10% Completed | 9/93 [00:12<02:38,  1.89s/it]
Loading safetensors checkpoint shards:  11% Completed | 10/93 [00:14<02:36,  1.88s/it]
Loading safetensors checkpoint shards:  12% Completed | 11/93 [00:16<02:33,  1.88s/it]
Loading safetensors checkpoint shards:  13% Completed | 12/93 [00:18<02:31,  1.87s/it]
Loading safetensors checkpoint shards:  14% Completed | 13/93 [00:20<02:29,  1.87s/it]
Loading safetensors checkpoint shards:  15% Completed | 14/93 [00:22<02:26,  1.86s/it]
Loading safetensors checkpoint shards:  16% Completed | 15/93 [00:24<02:25,  1.86s/it]
Loading safetensors checkpoint shards:  17% Completed | 16/93 [00:26<02:28,  1.93s/it]
Loading safetensors checkpoint shards:  18% Completed | 17/93 [00:28<02:32,  2.00s/it]
Loading safetensors checkpoint shards:  19% Completed | 18/93 [00:30<02:32,  2.04s/it]
Loading safetensors checkpoint shards:  20% Completed | 19/93 [00:32<02:33,  2.07s/it]
Loading safetensors checkpoint shards:  22% Completed | 20/93 [00:34<02:28,  2.04s/it]
Loading safetensors checkpoint shards:  23% Completed | 21/93 [00:36<02:24,  2.00s/it]
Loading safetensors checkpoint shards:  24% Completed | 22/93 [00:38<02:20,  1.98s/it]
Loading safetensors checkpoint shards:  25% Completed | 23/93 [00:40<02:16,  1.95s/it]
Loading safetensors checkpoint shards:  26% Completed | 24/93 [00:42<02:13,  1.94s/it]
Loading safetensors checkpoint shards:  27% Completed | 25/93 [00:44<02:10,  1.92s/it]
Loading safetensors checkpoint shards:  28% Completed | 26/93 [00:46<02:10,  1.94s/it]
Loading safetensors checkpoint shards:  29% Completed | 27/93 [00:47<02:06,  1.92s/it]
Loading safetensors checkpoint shards:  30% Completed | 28/93 [00:49<02:03,  1.91s/it]
Loading safetensors checkpoint shards:  31% Completed | 29/93 [00:51<02:07,  1.99s/it]
Loading safetensors checkpoint shards:  32% Completed | 30/93 [00:54<02:09,  2.05s/it]
Loading safetensors checkpoint shards:  33% Completed | 31/93 [00:56<02:10,  2.11s/it]
Loading safetensors checkpoint shards:  34% Completed | 32/93 [00:58<02:10,  2.13s/it]
Loading safetensors checkpoint shards:  35% Completed | 33/93 [01:00<02:10,  2.17s/it]
Loading safetensors checkpoint shards:  37% Completed | 34/93 [01:03<02:09,  2.20s/it]
Loading safetensors checkpoint shards:  38% Completed | 35/93 [01:05<02:08,  2.22s/it]
Loading safetensors checkpoint shards:  39% Completed | 36/93 [01:07<02:07,  2.24s/it]
Loading safetensors checkpoint shards:  40% Completed | 37/93 [01:09<02:05,  2.24s/it]
Loading safetensors checkpoint shards:  41% Completed | 38/93 [01:12<02:03,  2.25s/it]
Loading safetensors checkpoint shards:  42% Completed | 39/93 [01:14<02:01,  2.25s/it]
Loading safetensors checkpoint shards:  43% Completed | 40/93 [01:16<01:59,  2.25s/it]
Loading safetensors checkpoint shards:  44% Completed | 41/93 [01:18<01:57,  2.26s/it]
Loading safetensors checkpoint shards:  45% Completed | 42/93 [01:21<01:55,  2.26s/it]
Loading safetensors checkpoint shards:  46% Completed | 43/93 [01:23<01:55,  2.31s/it]
Loading safetensors checkpoint shards:  47% Completed | 44/93 [01:26<01:56,  2.37s/it]
Loading safetensors checkpoint shards:  48% Completed | 45/93 [01:28<01:52,  2.34s/it]
Loading safetensors checkpoint shards:  49% Completed | 46/93 [01:30<01:49,  2.32s/it]
Loading safetensors checkpoint shards:  51% Completed | 47/93 [01:32<01:44,  2.28s/it]
Loading safetensors checkpoint shards:  52% Completed | 48/93 [01:34<01:37,  2.17s/it]
Loading safetensors checkpoint shards:  53% Completed | 49/93 [01:36<01:30,  2.06s/it]
Loading safetensors checkpoint shards:  54% Completed | 50/93 [01:38<01:30,  2.10s/it]
Loading safetensors checkpoint shards:  55% Completed | 51/93 [01:40<01:26,  2.05s/it]
Loading safetensors checkpoint shards:  56% Completed | 52/93 [01:42<01:21,  1.99s/it]
Loading safetensors checkpoint shards:  57% Completed | 53/93 [01:45<01:25,  2.14s/it]
Loading safetensors checkpoint shards:  58% Completed | 54/93 [01:47<01:28,  2.26s/it]
Loading safetensors checkpoint shards:  59% Completed | 55/93 [01:50<01:28,  2.32s/it]
Loading safetensors checkpoint shards:  60% Completed | 56/93 [01:52<01:27,  2.37s/it]
Loading safetensors checkpoint shards:  61% Completed | 57/93 [01:54<01:24,  2.34s/it]
Loading safetensors checkpoint shards:  62% Completed | 58/93 [01:57<01:20,  2.31s/it]
Loading safetensors checkpoint shards:  63% Completed | 59/93 [01:59<01:19,  2.34s/it]
Loading safetensors checkpoint shards:  65% Completed | 60/93 [02:01<01:18,  2.38s/it]
Loading safetensors checkpoint shards:  66% Completed | 61/93 [02:04<01:14,  2.34s/it]
Loading safetensors checkpoint shards:  67% Completed | 62/93 [02:06<01:11,  2.30s/it]
Loading safetensors checkpoint shards:  68% Completed | 63/93 [02:08<01:09,  2.32s/it]
Loading safetensors checkpoint shards:  69% Completed | 64/93 [02:11<01:09,  2.38s/it]
Loading safetensors checkpoint shards:  70% Completed | 65/93 [02:13<01:06,  2.39s/it]
Loading safetensors checkpoint shards:  71% Completed | 66/93 [02:16<01:05,  2.43s/it]
Loading safetensors checkpoint shards:  72% Completed | 67/93 [02:18<01:04,  2.48s/it]
Loading safetensors checkpoint shards:  73% Completed | 68/93 [02:21<01:02,  2.50s/it]
Loading safetensors checkpoint shards:  74% Completed | 69/93 [02:23<01:00,  2.52s/it]
Loading safetensors checkpoint shards:  75% Completed | 70/93 [02:26<00:58,  2.53s/it]
Loading safetensors checkpoint shards:  76% Completed | 71/93 [02:28<00:54,  2.49s/it]
Loading safetensors checkpoint shards:  77% Completed | 72/93 [02:31<00:51,  2.43s/it]
Loading safetensors checkpoint shards:  78% Completed | 73/93 [02:33<00:48,  2.40s/it]
Loading safetensors checkpoint shards:  80% Completed | 74/93 [02:35<00:45,  2.38s/it]
Loading safetensors checkpoint shards:  81% Completed | 75/93 [02:38<00:43,  2.44s/it]
Loading safetensors checkpoint shards:  82% Completed | 76/93 [02:40<00:42,  2.48s/it]
Loading safetensors checkpoint shards:  83% Completed | 77/93 [02:43<00:38,  2.44s/it]
Loading safetensors checkpoint shards:  84% Completed | 78/93 [02:45<00:36,  2.43s/it]
Loading safetensors checkpoint shards:  85% Completed | 79/93 [02:48<00:34,  2.47s/it]
Loading safetensors checkpoint shards:  86% Completed | 80/93 [02:50<00:32,  2.50s/it]
Loading safetensors checkpoint shards:  87% Completed | 81/93 [02:53<00:30,  2.52s/it]
Loading safetensors checkpoint shards:  88% Completed | 82/93 [02:55<00:27,  2.52s/it]
Loading safetensors checkpoint shards:  89% Completed | 83/93 [02:58<00:25,  2.54s/it]
Loading safetensors checkpoint shards:  90% Completed | 84/93 [03:01<00:22,  2.54s/it]
Loading safetensors checkpoint shards:  91% Completed | 85/93 [03:03<00:20,  2.55s/it]
Loading safetensors checkpoint shards:  92% Completed | 86/93 [03:06<00:17,  2.55s/it]
Loading safetensors checkpoint shards:  94% Completed | 87/93 [03:08<00:15,  2.55s/it]
Loading safetensors checkpoint shards:  95% Completed | 88/93 [03:11<00:12,  2.55s/it]
Loading safetensors checkpoint shards:  96% Completed | 89/93 [03:13<00:10,  2.55s/it]
Loading safetensors checkpoint shards:  97% Completed | 90/93 [03:16<00:07,  2.57s/it]
Loading safetensors checkpoint shards:  98% Completed | 91/93 [03:19<00:05,  2.57s/it]
Loading safetensors checkpoint shards:  99% Completed | 92/93 [03:22<00:02,  2.81s/it]
Loading safetensors checkpoint shards: 100% Completed | 93/93 [03:24<00:00,  2.70s/it]
Loading safetensors checkpoint shards: 100% Completed | 93/93 [03:24<00:00,  2.20s/it]

TP MOE layer 3, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 4, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 5, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 6, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 7, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 8, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 9, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 10, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 11, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 12, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 13, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 14, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 15, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 16, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 17, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 18, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 19, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 20, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 21, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 22, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 23, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 24, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 25, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 26, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 27, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 28, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 29, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 30, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 31, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 32, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 33, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 34, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 35, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 36, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 37, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 38, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 39, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 40, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 41, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 42, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 43, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 44, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 45, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 46, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 47, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 48, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 49, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 50, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 51, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 52, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 53, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 54, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 55, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 56, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 57, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 58, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 59, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 60, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 61, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 62, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 63, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 64, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 65, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 66, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 67, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 68, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 69, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 70, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 71, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 72, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 73, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 74, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 75, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 76, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 77, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 78, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 79, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 80, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 81, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 82, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
TP Load from loader
TP MOE layer 83, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 84, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 85, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 86, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 87, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
TP Load from loader
TP MOE layer 88, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 89, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
TP MOE layer 90, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
TP MOE layer 91, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
TP Load from loader
Fatal Python error: Segmentation fault

Thread 0x00007ed427fff6c0 (most recent call first):
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 359 in wait
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 655 in wait
  File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007f39a0232b80 (most recent call first):
  File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/kt_kernel/utils/amx.py", line 294 in load_weights
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 258 in process_weights_after_loading
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 619 in load_weights_and_postprocess
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 600 in load_model
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/__init__.py", line 28 in get_model
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 797 in load_model
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 432 in initialize
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 357 in __init__
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/tp_worker.py", line 245 in __init__
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 316 in __init__
  File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 2659 in run_scheduler_process
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108 in run
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 135 in _main
  File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, pybase64._pybase64, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, regex._regex, yaml._yaml, markupsafe._speedups, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, msgspec._core, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda_utils (total: 101)
!!!!!!! Segfault encountered !!!!!!!
  File "/usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine
  File "<unknown>", line 0, in 0xffffffffffffffff

Others

No response

Originally created by @reasv on GitHub (Dec 10, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1694 Originally assigned to: @ovowei on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info CPU: 2x Intel Xeon 8592+ ES Q2SP (60c per socket) GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (96GB) KTransformers/kt-kernel: latest main pulled today (f992de55dac66599fa429b3a3af8ce11d6b6bb32) SGLang: Version: 0.5.6.post1 torch: 2.9.1 OS: Arch Linux (kernel 6.17.8-arch1-1) Driver Version: 580.105.08 CUDA Version: 13.0 ### Reproduction `/data/models/GLM-4.6-INT8-quant` was quantized from 16-bit official weights using the current KTransformers script ```text CUDA_VISIBLE_DEVICES=1 python -m sglang.launch_server \ --host 0.0.0.0 \ --port 8000 \ --attention-backend triton \ --model /data/models/GLM-4.6-FP8 \ --trust-remote-code \ --mem-fraction-static 0.92 \ --chunked-prefill-size 4096 \ --served-model-name GLM-4.6 \ --enable-mixed-chunk \ --kt-method AMXINT8 \ --kt-weight-path /data/models/GLM-4.6-INT8-quant \ --kt-cpuinfer 120 \ --kt-threadpool-count 4 \ --kt-num-gpu-experts 32 \ --kt-max-deferred-experts-per-token 2 ``` Output: ``` [2025-12-10 02:32:25] server_args=ServerArgs(model_path='/data/models/GLM-4.6-FP8', tokenizer_path='/data/models/GLM-4.6-FP8', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, mem_fraction_static=0.92, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=4096, max_prefill_tokens=16384, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, stream_interval=1, stream_output=False, random_seed=209423600, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, mm_process_config={}, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='GLM-4.6', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', enable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_moe_runner_backend=None, speculative_moe_a2a_backend=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='/data/models/GLM-4.6-INT8-quant', kt_method='AMXINT8', kt_cpuinfer=120, kt_threadpool_count=4, kt_num_gpu_experts=32, kt_max_deferred_experts_per_token=2, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=256, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=False, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, decrypted_config_file=None, decrypted_draft_config_file=None, mm_enable_dp_encoder=False, forward_hooks=None) [2025-12-10 02:32:26] Using default HuggingFace chat template with detected content format: openai [2025-12-10 02:32:31] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2025-12-10 02:32:31] Init torch distributed ends. mem usage=0.00 GB [2025-12-10 02:32:32] Ignore import error when loading sglang.srt.models.mindspore: name 'ms' is not defined [2025-12-10 02:32:33] Load weight begin. avail mem=94.31 GB [2025-12-10 02:32:33] Shared experts fusion optimization enabled. CPUInfer[0x4efe2f10]: Hello WorkerPool[0x4efe1a60] 4 subpools, [numa:threads][0:30] [1:30] [2:30] [3:30] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 30 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 30 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 2, 30 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 3, 30 threads Loading safetensors checkpoint shards: 0% Completed | 0/93 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 1% Completed | 1/93 [00:00<01:03, 1.45it/s] Loading safetensors checkpoint shards: 2% Completed | 2/93 [00:00<00:34, 2.65it/s] Loading safetensors checkpoint shards: 3% Completed | 3/93 [00:00<00:23, 3.82it/s] Loading safetensors checkpoint shards: 4% Completed | 4/93 [00:02<01:20, 1.10it/s] Loading safetensors checkpoint shards: 5% Completed | 5/93 [00:04<01:52, 1.27s/it] Loading safetensors checkpoint shards: 6% Completed | 6/93 [00:06<02:09, 1.49s/it] Loading safetensors checkpoint shards: 8% Completed | 7/93 [00:08<02:19, 1.62s/it] Loading safetensors checkpoint shards: 9% Completed | 8/93 [00:10<02:32, 1.79s/it] Loading safetensors checkpoint shards: 10% Completed | 9/93 [00:12<02:38, 1.89s/it] Loading safetensors checkpoint shards: 11% Completed | 10/93 [00:14<02:36, 1.88s/it] Loading safetensors checkpoint shards: 12% Completed | 11/93 [00:16<02:33, 1.88s/it] Loading safetensors checkpoint shards: 13% Completed | 12/93 [00:18<02:31, 1.87s/it] Loading safetensors checkpoint shards: 14% Completed | 13/93 [00:20<02:29, 1.87s/it] Loading safetensors checkpoint shards: 15% Completed | 14/93 [00:22<02:26, 1.86s/it] Loading safetensors checkpoint shards: 16% Completed | 15/93 [00:24<02:25, 1.86s/it] Loading safetensors checkpoint shards: 17% Completed | 16/93 [00:26<02:28, 1.93s/it] Loading safetensors checkpoint shards: 18% Completed | 17/93 [00:28<02:32, 2.00s/it] Loading safetensors checkpoint shards: 19% Completed | 18/93 [00:30<02:32, 2.04s/it] Loading safetensors checkpoint shards: 20% Completed | 19/93 [00:32<02:33, 2.07s/it] Loading safetensors checkpoint shards: 22% Completed | 20/93 [00:34<02:28, 2.04s/it] Loading safetensors checkpoint shards: 23% Completed | 21/93 [00:36<02:24, 2.00s/it] Loading safetensors checkpoint shards: 24% Completed | 22/93 [00:38<02:20, 1.98s/it] Loading safetensors checkpoint shards: 25% Completed | 23/93 [00:40<02:16, 1.95s/it] Loading safetensors checkpoint shards: 26% Completed | 24/93 [00:42<02:13, 1.94s/it] Loading safetensors checkpoint shards: 27% Completed | 25/93 [00:44<02:10, 1.92s/it] Loading safetensors checkpoint shards: 28% Completed | 26/93 [00:46<02:10, 1.94s/it] Loading safetensors checkpoint shards: 29% Completed | 27/93 [00:47<02:06, 1.92s/it] Loading safetensors checkpoint shards: 30% Completed | 28/93 [00:49<02:03, 1.91s/it] Loading safetensors checkpoint shards: 31% Completed | 29/93 [00:51<02:07, 1.99s/it] Loading safetensors checkpoint shards: 32% Completed | 30/93 [00:54<02:09, 2.05s/it] Loading safetensors checkpoint shards: 33% Completed | 31/93 [00:56<02:10, 2.11s/it] Loading safetensors checkpoint shards: 34% Completed | 32/93 [00:58<02:10, 2.13s/it] Loading safetensors checkpoint shards: 35% Completed | 33/93 [01:00<02:10, 2.17s/it] Loading safetensors checkpoint shards: 37% Completed | 34/93 [01:03<02:09, 2.20s/it] Loading safetensors checkpoint shards: 38% Completed | 35/93 [01:05<02:08, 2.22s/it] Loading safetensors checkpoint shards: 39% Completed | 36/93 [01:07<02:07, 2.24s/it] Loading safetensors checkpoint shards: 40% Completed | 37/93 [01:09<02:05, 2.24s/it] Loading safetensors checkpoint shards: 41% Completed | 38/93 [01:12<02:03, 2.25s/it] Loading safetensors checkpoint shards: 42% Completed | 39/93 [01:14<02:01, 2.25s/it] Loading safetensors checkpoint shards: 43% Completed | 40/93 [01:16<01:59, 2.25s/it] Loading safetensors checkpoint shards: 44% Completed | 41/93 [01:18<01:57, 2.26s/it] Loading safetensors checkpoint shards: 45% Completed | 42/93 [01:21<01:55, 2.26s/it] Loading safetensors checkpoint shards: 46% Completed | 43/93 [01:23<01:55, 2.31s/it] Loading safetensors checkpoint shards: 47% Completed | 44/93 [01:26<01:56, 2.37s/it] Loading safetensors checkpoint shards: 48% Completed | 45/93 [01:28<01:52, 2.34s/it] Loading safetensors checkpoint shards: 49% Completed | 46/93 [01:30<01:49, 2.32s/it] Loading safetensors checkpoint shards: 51% Completed | 47/93 [01:32<01:44, 2.28s/it] Loading safetensors checkpoint shards: 52% Completed | 48/93 [01:34<01:37, 2.17s/it] Loading safetensors checkpoint shards: 53% Completed | 49/93 [01:36<01:30, 2.06s/it] Loading safetensors checkpoint shards: 54% Completed | 50/93 [01:38<01:30, 2.10s/it] Loading safetensors checkpoint shards: 55% Completed | 51/93 [01:40<01:26, 2.05s/it] Loading safetensors checkpoint shards: 56% Completed | 52/93 [01:42<01:21, 1.99s/it] Loading safetensors checkpoint shards: 57% Completed | 53/93 [01:45<01:25, 2.14s/it] Loading safetensors checkpoint shards: 58% Completed | 54/93 [01:47<01:28, 2.26s/it] Loading safetensors checkpoint shards: 59% Completed | 55/93 [01:50<01:28, 2.32s/it] Loading safetensors checkpoint shards: 60% Completed | 56/93 [01:52<01:27, 2.37s/it] Loading safetensors checkpoint shards: 61% Completed | 57/93 [01:54<01:24, 2.34s/it] Loading safetensors checkpoint shards: 62% Completed | 58/93 [01:57<01:20, 2.31s/it] Loading safetensors checkpoint shards: 63% Completed | 59/93 [01:59<01:19, 2.34s/it] Loading safetensors checkpoint shards: 65% Completed | 60/93 [02:01<01:18, 2.38s/it] Loading safetensors checkpoint shards: 66% Completed | 61/93 [02:04<01:14, 2.34s/it] Loading safetensors checkpoint shards: 67% Completed | 62/93 [02:06<01:11, 2.30s/it] Loading safetensors checkpoint shards: 68% Completed | 63/93 [02:08<01:09, 2.32s/it] Loading safetensors checkpoint shards: 69% Completed | 64/93 [02:11<01:09, 2.38s/it] Loading safetensors checkpoint shards: 70% Completed | 65/93 [02:13<01:06, 2.39s/it] Loading safetensors checkpoint shards: 71% Completed | 66/93 [02:16<01:05, 2.43s/it] Loading safetensors checkpoint shards: 72% Completed | 67/93 [02:18<01:04, 2.48s/it] Loading safetensors checkpoint shards: 73% Completed | 68/93 [02:21<01:02, 2.50s/it] Loading safetensors checkpoint shards: 74% Completed | 69/93 [02:23<01:00, 2.52s/it] Loading safetensors checkpoint shards: 75% Completed | 70/93 [02:26<00:58, 2.53s/it] Loading safetensors checkpoint shards: 76% Completed | 71/93 [02:28<00:54, 2.49s/it] Loading safetensors checkpoint shards: 77% Completed | 72/93 [02:31<00:51, 2.43s/it] Loading safetensors checkpoint shards: 78% Completed | 73/93 [02:33<00:48, 2.40s/it] Loading safetensors checkpoint shards: 80% Completed | 74/93 [02:35<00:45, 2.38s/it] Loading safetensors checkpoint shards: 81% Completed | 75/93 [02:38<00:43, 2.44s/it] Loading safetensors checkpoint shards: 82% Completed | 76/93 [02:40<00:42, 2.48s/it] Loading safetensors checkpoint shards: 83% Completed | 77/93 [02:43<00:38, 2.44s/it] Loading safetensors checkpoint shards: 84% Completed | 78/93 [02:45<00:36, 2.43s/it] Loading safetensors checkpoint shards: 85% Completed | 79/93 [02:48<00:34, 2.47s/it] Loading safetensors checkpoint shards: 86% Completed | 80/93 [02:50<00:32, 2.50s/it] Loading safetensors checkpoint shards: 87% Completed | 81/93 [02:53<00:30, 2.52s/it] Loading safetensors checkpoint shards: 88% Completed | 82/93 [02:55<00:27, 2.52s/it] Loading safetensors checkpoint shards: 89% Completed | 83/93 [02:58<00:25, 2.54s/it] Loading safetensors checkpoint shards: 90% Completed | 84/93 [03:01<00:22, 2.54s/it] Loading safetensors checkpoint shards: 91% Completed | 85/93 [03:03<00:20, 2.55s/it] Loading safetensors checkpoint shards: 92% Completed | 86/93 [03:06<00:17, 2.55s/it] Loading safetensors checkpoint shards: 94% Completed | 87/93 [03:08<00:15, 2.55s/it] Loading safetensors checkpoint shards: 95% Completed | 88/93 [03:11<00:12, 2.55s/it] Loading safetensors checkpoint shards: 96% Completed | 89/93 [03:13<00:10, 2.55s/it] Loading safetensors checkpoint shards: 97% Completed | 90/93 [03:16<00:07, 2.57s/it] Loading safetensors checkpoint shards: 98% Completed | 91/93 [03:19<00:05, 2.57s/it] Loading safetensors checkpoint shards: 99% Completed | 92/93 [03:22<00:02, 2.81s/it] Loading safetensors checkpoint shards: 100% Completed | 93/93 [03:24<00:00, 2.70s/it] Loading safetensors checkpoint shards: 100% Completed | 93/93 [03:24<00:00, 2.20s/it] TP MOE layer 3, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 4, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 5, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 6, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 7, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 8, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 9, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 10, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 11, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 12, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 13, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 14, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 15, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 16, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 17, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 18, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 19, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 20, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 21, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 22, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 23, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 24, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 25, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 26, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 27, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 28, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 29, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 30, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 31, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 32, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 33, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 34, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 35, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 36, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 37, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 38, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 39, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 40, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 41, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 42, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 43, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 44, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 45, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 46, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 47, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 48, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 49, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 50, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 51, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 52, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 53, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 54, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 55, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 56, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 57, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 58, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 59, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 60, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 61, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 62, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 63, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 64, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 65, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 66, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 67, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 68, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 69, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 70, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 71, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 72, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 73, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 74, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 75, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 76, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 77, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 78, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 79, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 80, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 81, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 82, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 83, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 84, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 85, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 86, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 87, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 TP Load from loader TP MOE layer 88, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 89, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader TP MOE layer 90, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader TP MOE layer 91, pool: 0x4efe1a60, expert num: 161, num_experts_per_tok: 9 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 TP Load from loader Fatal Python error: Segmentation fault Thread 0x00007ed427fff6c0 (most recent call first): File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 359 in wait File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 655 in wait File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x00007f39a0232b80 (most recent call first): File "/home/admin/projects/sgl/.venv/lib/python3.12/site-packages/kt_kernel/utils/amx.py", line 294 in load_weights File "/home/admin/projects/sgl/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 258 in process_weights_after_loading File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 619 in load_weights_and_postprocess File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/loader.py", line 600 in load_model File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_loader/__init__.py", line 28 in get_model File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 797 in load_model File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 432 in initialize File "/home/admin/projects/sgl/sglang/python/sglang/srt/model_executor/model_runner.py", line 357 in __init__ File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/tp_worker.py", line 245 in __init__ File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 316 in __init__ File "/home/admin/projects/sgl/sglang/python/sglang/srt/managers/scheduler.py", line 2659 in run_scheduler_process File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 108 in run File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 135 in _main File "/home/admin/.local/share/uv/python/cpython-3.12.10-linux-x86_64-gnu/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main File "<string>", line 1 in <module> Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, pybase64._pybase64, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, regex._regex, yaml._yaml, markupsafe._speedups, PIL._imagingft, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, msgspec._core, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda_utils (total: 101) !!!!!!! Segfault encountered !!!!!!! File "/usr/src/debug/gcc/gcc/libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine File "<unknown>", line 0, in 0xffffffffffffffff ``` ### Others _No response_
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1086
No description provided.