kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1794] DeepSeek V3.2 FP8/INT8 都会crash #1129

New issue
Closed
opened 2026-10-29 14:48:56 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-29 14:48:56 +00:00
Owner
Copy link

Originally created by @SchenbergZY on GitHub (Jan 14, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1794

Reminder

  • I have read the above rules and searched the existing issues.

System Info

如题目所言，我运行如下命令：

python -m sglang.launch_server   --host 0.0.0.0   --port 8000   --model models/DeepSeek-V3.2   --trust-remote-code   --mem-fraction-static 0.6   --chunked-prefill-size 512   --served-model-name DeepSeek-V3.2   --enable-mixed-chunk   --kt-weight-path models/DeepSeek-V3.2-INT8   --kt-cpuinfer 64   --kt-threadpool-count 8   --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --tensor-parallel-size 1 --kt-method AMXINT8 --attention-backend triton --cuda-graph-max-bs 1

的时候，系统会卡死。
看记录一般是在40层。

我在convert weight的时候，也会在34层的地方卡死。使用了resume-layer才转换完成。

我的系统配置如下（在jupyterlab中）

Image

想问下卡死是不是因为我这个配置不够。

Reproduction

Put your message here.

Others

No response

Originally created by @SchenbergZY on GitHub (Jan 14, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1794 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info 如题目所言，我运行如下命令： ``` python -m sglang.launch_server --host 0.0.0.0 --port 8000 --model models/DeepSeek-V3.2 --trust-remote-code --mem-fraction-static 0.6 --chunked-prefill-size 512 --served-model-name DeepSeek-V3.2 --enable-mixed-chunk --kt-weight-path models/DeepSeek-V3.2-INT8 --kt-cpuinfer 64 --kt-threadpool-count 8 --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --tensor-parallel-size 1 --kt-method AMXINT8 --attention-backend triton --cuda-graph-max-bs 1 ``` 的时候，系统会卡死。 看记录一般是在40层。 我在convert weight的时候，也会在34层的地方卡死。使用了resume-layer才转换完成。 我的系统配置如下（在jupyterlab中） <img width="1502" height="505" alt="Image" src="https://github.com/user-attachments/assets/a56425e0-0673-41f2-b3bf-ed2860e99b58" /> 想问下卡死是不是因为我这个配置不够。 ### Reproduction ```text Put your message here. ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:48:57 +00:00
Author
Owner
Copy link

@CodeZ-Hao commented on GitHub (Jan 15, 2026):

--kt-threadpool-count 这个对应你的numa数，几numa就设置几，我估计是这个问题，如果确实是8 numa，BIOS里应该有设置可以修改，调到2或者4，对应你实际socket数量

<!-- gh-comment-id:3752531303 --> @CodeZ-Hao commented on GitHub (Jan 15, 2026): --kt-threadpool-count 这个对应你的numa数，几numa就设置几，我估计是这个问题，如果确实是8 numa，BIOS里应该有设置可以修改，调到2或者4，对应你实际socket数量
bsakaguchi commented 2026-10-29 14:48:57 +00:00
Author
Owner
Copy link

@ErvinXie commented on GitHub (Jan 15, 2026):

Your environment is far good enough to run int8/fp8, please directly run v3.2, and put your error message.

<!-- gh-comment-id:3754646829 --> @ErvinXie commented on GitHub (Jan 15, 2026): Your environment is far good enough to run int8/fp8, please directly run v3.2, and put your error message.
bsakaguchi commented 2026-10-29 14:48:57 +00:00
Author
Owner
Copy link

@SchenbergZY commented on GitHub (Jan 15, 2026):

--kt-threadpool-count 这个对应你的numa数，几numa就设置几，我估计是这个问题，如果确实是8 numa，BIOS里应该有设置可以修改，调到2或者4，对应你实际socket数量

我的lscpu如下：

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          52 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   384
  On-line CPU(s) list:    0-383
Vendor ID:                AuthenticAMD
  Model name:             AMD EPYC 9654 96-Core Processor
    CPU family:           25
    Model:                17
    Thread(s) per core:   2
    Core(s) per socket:   96
    Socket(s):            2
    Stepping:             1
    Frequency boost:      enabled
    CPU(s) scaling MHz:   63%
    CPU max MHz:          2400.0000
    CPU min MHz:          1500.0000
    BogoMIPS:             4800.08
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr
                           sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_
                          v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx
                          16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy s
                          vm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoex
                          t perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba 
                          perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpc
                          id cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni
                           avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cq
                          m_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc
                           amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeas
                          sists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vb
                          mi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpo
                          pcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap
Virtualization features:  
  Virtualization:         AMD-V
Caches (sum of all):      
  L1d:                    6 MiB (192 instances)
  L1i:                    6 MiB (192 instances)
  L2:                     192 MiB (192 instances)
  L3:                     768 MiB (24 instances)
NUMA:                     
  NUMA node(s):           24
  NUMA node0 CPU(s):      0-7,192-199
  NUMA node1 CPU(s):      8-15,200-207
  NUMA node2 CPU(s):      16-23,208-215
  NUMA node3 CPU(s):      24-31,216-223
  NUMA node4 CPU(s):      32-39,224-231
  NUMA node5 CPU(s):      40-47,232-239
  NUMA node6 CPU(s):      48-55,240-247
  NUMA node7 CPU(s):      56-63,248-255
  NUMA node8 CPU(s):      64-71,256-263
  NUMA node9 CPU(s):      72-79,264-271
  NUMA node10 CPU(s):     80-87,272-279
  NUMA node11 CPU(s):     88-95,280-287
  NUMA node12 CPU(s):     96-103,288-295
  NUMA node13 CPU(s):     104-111,296-303
  NUMA node14 CPU(s):     112-119,304-311
  NUMA node15 CPU(s):     120-127,312-319
  NUMA node16 CPU(s):     128-135,320-327
  NUMA node17 CPU(s):     136-143,328-335
  NUMA node18 CPU(s):     144-151,336-343
  NUMA node19 CPU(s):     152-159,344-351
  NUMA node20 CPU(s):     160-167,352-359
  NUMA node21 CPU(s):     168-175,360-367
  NUMA node22 CPU(s):     176-183,368-375
  NUMA node23 CPU(s):     184-191,376-383
Vulnerabilities:          
  Gather data sampling:   Not affected
  Itlb multihit:          Not affected
  L1tf:                   Not affected
  Mds:                    Not affected
  Meltdown:               Not affected
  Mmio stale data:        Not affected
  Reg file data sampling: Not affected
  Retbleed:               Not affected
  Spec rstack overflow:   Vulnerable
  Spec store bypass:      Vulnerable
  Spectre v1:             Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
  Spectre v2:             Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Not affected; BHI: Not affecte
                          d
  Srbds:                  Not affected
  Tsx async abort:        Not affected
  Vmscape:                Vulnerable

想请问下--kt-threadpool-count 该怎么设置呢，是24么，可是24不能被DS-V3.2的2048的intermediate_size整除。所以设16？

下面是我运行INT8_NUMA8的命令：

python -m sglang.launch_server   --host 0.0.0.0   --port 8000   --model models/DeepSeek-V3.2   --trust-remote-code   --mem-fraction-static 0.6   --chunked-prefill-size 512   --served-model-name DeepSeek-V3.2   --enable-mixed-chunk   --kt-weight-path models/DeepSeek-V3.2-INT8   --kt-cpuinfer 64   --kt-threadpool-count 8   --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --tensor-parallel-size 1 --kt-method AMXINT8 --attention-backend triton --cuda-graph-max-bs 1

结果在loading第40层的时候卡死：

[2026-01-15 16:40:57] WARNING server_args.py:1032: DP attention is enabled for DeepSeek NSA.
[2026-01-15 16:40:57] WARNING server_args.py:1055: Setting page size to 64 for DeepSeek NSA.
[2026-01-15 16:40:57] WARNING server_args.py:1063: Setting KV cache dtype to bfloat16 for DeepSeek NSA.
NSA_DUAL_STREAM=True NSA_FUSE_TOPK=True NSA_FLASHMLA_BACKEND_DECODE_COMPUTE_FP8=True NSA_QUANT_K_CACHE_FAST=True NSA_DEQUANT_K_CACHE_FAST=True NSA_ENABLE_MTP_PRECOMPUTE_METADATA=True 
[2026-01-15 16:40:57] server_args=ServerArgs(model_path='models/DeepSeek-V3.2', tokenizer_path='models/DeepSeek-V3.2', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, limit_mm_data_per_request=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='bfloat16', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.9, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=512, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=64, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=284622794, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='DeepSeek-V3.2', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='deepseek-v3', tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='models/DeepSeek-V3.2-INT8', kt_method='AMXINT8', kt_cpuinfer=64, kt_threadpool_count=8, kt_num_gpu_experts=0, kt_max_deferred_experts_per_token=2, kt_gpu_prefill_token_threshold=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=1, cuda_graph_bs=[1], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-01-15 16:40:57] No chat template found, defaulting to 'string' content format
[2026-01-15 16:41:03] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2026-01-15 16:41:03] Init torch distributed ends. mem usage=0.00 GB
[2026-01-15 16:41:04] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmasrConfig' from 'transformers' (/home/jovyan/yue/venv_py12/lib/python3.12/site-packages/transformers/__init__.py)
[2026-01-15 16:41:04] Load weight begin. avail mem=46.85 GB
[2026-01-15 16:41:04] Detected fp8 checkpoint.

Loading safetensors checkpoint shards:   0% Completed | 0/163 [00:00<?, ?it/s]

Loading safetensors checkpoint shards:   2% Completed | 3/163 [00:00<00:06, 24.96it/s]

Loading safetensors checkpoint shards:   4% Completed | 6/163 [00:00<00:18,  8.37it/s]

Loading safetensors checkpoint shards:   5% Completed | 8/163 [00:00<00:16,  9.14it/s]

Loading safetensors checkpoint shards:   7% Completed | 11/163 [00:00<00:12, 12.10it/s]

Loading safetensors checkpoint shards:   9% Completed | 15/163 [00:01<00:08, 17.37it/s]

Loading safetensors checkpoint shards:  12% Completed | 19/163 [00:01<00:06, 21.11it/s]

Loading safetensors checkpoint shards:  13% Completed | 22/163 [00:01<00:11, 11.90it/s]

Loading safetensors checkpoint shards:  15% Completed | 24/163 [00:01<00:12, 10.70it/s]

Loading safetensors checkpoint shards:  16% Completed | 26/163 [00:02<00:13, 10.21it/s]

Loading safetensors checkpoint shards:  17% Completed | 28/163 [00:02<00:13,  9.83it/s]

Loading safetensors checkpoint shards:  18% Completed | 30/163 [00:02<00:13, 10.00it/s]

Loading safetensors checkpoint shards:  20% Completed | 32/163 [00:02<00:13,  9.47it/s]

Loading safetensors checkpoint shards:  21% Completed | 34/163 [00:03<00:13,  9.49it/s]

Loading safetensors checkpoint shards:  22% Completed | 36/163 [00:03<00:17,  7.45it/s]

Loading safetensors checkpoint shards:  23% Completed | 37/163 [00:03<00:16,  7.60it/s]

Loading safetensors checkpoint shards:  23% Completed | 38/163 [00:03<00:16,  7.60it/s]

Loading safetensors checkpoint shards:  24% Completed | 39/163 [00:03<00:16,  7.70it/s]

Loading safetensors checkpoint shards:  25% Completed | 41/163 [00:04<00:14,  8.21it/s]

Loading safetensors checkpoint shards:  26% Completed | 42/163 [00:04<00:14,  8.45it/s]

Loading safetensors checkpoint shards:  26% Completed | 43/163 [00:04<00:14,  8.29it/s]

Loading safetensors checkpoint shards:  27% Completed | 44/163 [00:04<00:13,  8.55it/s]

Loading safetensors checkpoint shards:  28% Completed | 45/163 [00:04<00:13,  8.68it/s]

Loading safetensors checkpoint shards:  28% Completed | 46/163 [00:04<00:13,  8.36it/s]

Loading safetensors checkpoint shards:  29% Completed | 47/163 [00:04<00:13,  8.32it/s]

Loading safetensors checkpoint shards:  30% Completed | 49/163 [00:04<00:13,  8.74it/s]

Loading safetensors checkpoint shards:  31% Completed | 50/163 [00:05<00:12,  8.95it/s]

Loading safetensors checkpoint shards:  31% Completed | 51/163 [00:05<00:12,  8.68it/s]

Loading safetensors checkpoint shards:  32% Completed | 52/163 [00:05<00:12,  8.88it/s]

Loading safetensors checkpoint shards:  33% Completed | 54/163 [00:05<00:11,  9.90it/s]

Loading safetensors checkpoint shards:  34% Completed | 55/163 [00:05<00:11,  9.19it/s]

Loading safetensors checkpoint shards:  35% Completed | 57/163 [00:05<00:11,  9.22it/s]

Loading safetensors checkpoint shards:  36% Completed | 58/163 [00:05<00:11,  9.14it/s]

Loading safetensors checkpoint shards:  36% Completed | 59/163 [00:06<00:11,  9.27it/s]

Loading safetensors checkpoint shards:  37% Completed | 60/163 [00:06<00:11,  8.82it/s]

Loading safetensors checkpoint shards:  38% Completed | 62/163 [00:06<00:11,  8.87it/s]

Loading safetensors checkpoint shards:  39% Completed | 63/163 [00:06<00:11,  8.72it/s]

Loading safetensors checkpoint shards:  39% Completed | 64/163 [00:06<00:11,  8.86it/s]

Loading safetensors checkpoint shards:  40% Completed | 65/163 [00:06<00:11,  8.55it/s]

Loading safetensors checkpoint shards:  40% Completed | 66/163 [00:06<00:11,  8.66it/s]

Loading safetensors checkpoint shards:  41% Completed | 67/163 [00:07<00:11,  8.35it/s]

Loading safetensors checkpoint shards:  42% Completed | 69/163 [00:07<00:10,  8.66it/s]

Loading safetensors checkpoint shards:  43% Completed | 70/163 [00:07<00:10,  8.61it/s]

Loading safetensors checkpoint shards:  44% Completed | 71/163 [00:07<00:10,  8.84it/s]

Loading safetensors checkpoint shards:  44% Completed | 72/163 [00:07<00:10,  8.57it/s]

Loading safetensors checkpoint shards:  45% Completed | 73/163 [00:07<00:10,  8.73it/s]

Loading safetensors checkpoint shards:  45% Completed | 74/163 [00:07<00:10,  8.73it/s]

Loading safetensors checkpoint shards:  47% Completed | 76/163 [00:08<00:15,  5.73it/s]

Loading safetensors checkpoint shards:  48% Completed | 78/163 [00:08<00:11,  7.14it/s]

Loading safetensors checkpoint shards:  48% Completed | 79/163 [00:08<00:11,  7.51it/s]

Loading safetensors checkpoint shards:  49% Completed | 80/163 [00:08<00:10,  7.69it/s]

Loading safetensors checkpoint shards:  50% Completed | 81/163 [00:08<00:10,  7.59it/s]

Loading safetensors checkpoint shards:  50% Completed | 82/163 [00:08<00:10,  8.05it/s]

Loading safetensors checkpoint shards:  51% Completed | 83/163 [00:09<00:09,  8.02it/s]

Loading safetensors checkpoint shards:  52% Completed | 84/163 [00:09<00:09,  8.01it/s]

Loading safetensors checkpoint shards:  52% Completed | 85/163 [00:09<00:09,  8.24it/s]

Loading safetensors checkpoint shards:  53% Completed | 86/163 [00:09<00:09,  7.89it/s]

Loading safetensors checkpoint shards:  53% Completed | 87/163 [00:09<00:09,  8.40it/s]

Loading safetensors checkpoint shards:  54% Completed | 88/163 [00:09<00:08,  8.62it/s]

Loading safetensors checkpoint shards:  55% Completed | 89/163 [00:09<00:08,  8.40it/s]

Loading safetensors checkpoint shards:  56% Completed | 91/163 [00:09<00:08,  8.94it/s]

Loading safetensors checkpoint shards:  56% Completed | 92/163 [00:10<00:07,  8.90it/s]

Loading safetensors checkpoint shards:  57% Completed | 93/163 [00:10<00:07,  9.14it/s]

Loading safetensors checkpoint shards:  58% Completed | 94/163 [00:10<00:07,  8.86it/s]

Loading safetensors checkpoint shards:  59% Completed | 96/163 [00:10<00:06, 10.12it/s]

Loading safetensors checkpoint shards:  60% Completed | 97/163 [00:10<00:06,  9.85it/s]

Loading safetensors checkpoint shards:  60% Completed | 98/163 [00:10<00:07,  9.16it/s]

Loading safetensors checkpoint shards:  61% Completed | 99/163 [00:10<00:06,  9.29it/s]

Loading safetensors checkpoint shards:  61% Completed | 100/163 [00:10<00:07,  8.87it/s]

Loading safetensors checkpoint shards:  62% Completed | 101/163 [00:11<00:06,  8.93it/s]

Loading safetensors checkpoint shards:  63% Completed | 102/163 [00:11<00:06,  8.78it/s]

Loading safetensors checkpoint shards:  63% Completed | 103/163 [00:11<00:06,  9.00it/s]

Loading safetensors checkpoint shards:  64% Completed | 104/163 [00:11<00:07,  8.41it/s]

Loading safetensors checkpoint shards:  64% Completed | 105/163 [00:11<00:07,  8.23it/s]

Loading safetensors checkpoint shards:  65% Completed | 106/163 [00:11<00:06,  8.30it/s]

Loading safetensors checkpoint shards:  66% Completed | 107/163 [00:11<00:06,  8.69it/s]

Loading safetensors checkpoint shards:  66% Completed | 108/163 [00:11<00:06,  8.43it/s]

Loading safetensors checkpoint shards:  67% Completed | 109/163 [00:12<00:06,  8.76it/s]

Loading safetensors checkpoint shards:  67% Completed | 110/163 [00:12<00:06,  8.83it/s]

Loading safetensors checkpoint shards:  69% Completed | 112/163 [00:12<00:05,  8.73it/s]

Loading safetensors checkpoint shards:  69% Completed | 113/163 [00:12<00:05,  8.60it/s]

Loading safetensors checkpoint shards:  70% Completed | 114/163 [00:12<00:05,  8.50it/s]

Loading safetensors checkpoint shards:  71% Completed | 115/163 [00:12<00:05,  8.71it/s]

Loading safetensors checkpoint shards:  71% Completed | 116/163 [00:12<00:05,  8.42it/s]

Loading safetensors checkpoint shards:  72% Completed | 117/163 [00:12<00:05,  8.58it/s]

Loading safetensors checkpoint shards:  73% Completed | 119/163 [00:13<00:04,  9.73it/s]

Loading safetensors checkpoint shards:  74% Completed | 120/163 [00:13<00:04,  9.07it/s]

Loading safetensors checkpoint shards:  75% Completed | 122/163 [00:13<00:04,  9.21it/s]

Loading safetensors checkpoint shards:  75% Completed | 123/163 [00:13<00:04,  9.27it/s]

Loading safetensors checkpoint shards:  76% Completed | 124/163 [00:13<00:04,  9.19it/s]

Loading safetensors checkpoint shards:  77% Completed | 125/163 [00:13<00:04,  8.74it/s]

Loading safetensors checkpoint shards:  78% Completed | 127/163 [00:14<00:06,  5.85it/s]

Loading safetensors checkpoint shards:  79% Completed | 129/163 [00:14<00:04,  7.09it/s]

Loading safetensors checkpoint shards:  80% Completed | 130/163 [00:14<00:04,  7.12it/s]

Loading safetensors checkpoint shards:  80% Completed | 131/163 [00:14<00:04,  7.46it/s]

Loading safetensors checkpoint shards:  81% Completed | 132/163 [00:14<00:04,  7.65it/s]

Loading safetensors checkpoint shards:  82% Completed | 133/163 [00:15<00:04,  7.31it/s]

Loading safetensors checkpoint shards:  83% Completed | 135/163 [00:15<00:03,  8.25it/s]

Loading safetensors checkpoint shards:  83% Completed | 136/163 [00:15<00:03,  8.44it/s]

Loading safetensors checkpoint shards:  85% Completed | 138/163 [00:15<00:02,  8.81it/s]

Loading safetensors checkpoint shards:  86% Completed | 140/163 [00:15<00:02,  9.31it/s]

Loading safetensors checkpoint shards:  87% Completed | 141/163 [00:15<00:02,  9.39it/s]

Loading safetensors checkpoint shards:  88% Completed | 143/163 [00:16<00:02,  9.55it/s]

Loading safetensors checkpoint shards:  88% Completed | 144/163 [00:16<00:02,  9.18it/s]

Loading safetensors checkpoint shards:  90% Completed | 146/163 [00:16<00:01,  9.42it/s]

Loading safetensors checkpoint shards:  90% Completed | 147/163 [00:16<00:01,  9.32it/s]

Loading safetensors checkpoint shards:  91% Completed | 149/163 [00:16<00:01,  9.33it/s]

Loading safetensors checkpoint shards:  92% Completed | 150/163 [00:16<00:01,  9.45it/s]

Loading safetensors checkpoint shards:  93% Completed | 151/163 [00:16<00:01,  9.49it/s]

Loading safetensors checkpoint shards:  93% Completed | 152/163 [00:17<00:01,  9.02it/s]

Loading safetensors checkpoint shards:  94% Completed | 154/163 [00:17<00:00,  9.47it/s]

Loading safetensors checkpoint shards:  95% Completed | 155/163 [00:17<00:00,  9.37it/s]

Loading safetensors checkpoint shards:  96% Completed | 157/163 [00:17<00:00,  9.35it/s]

Loading safetensors checkpoint shards:  97% Completed | 158/163 [00:17<00:00,  9.48it/s]

Loading safetensors checkpoint shards:  98% Completed | 159/163 [00:17<00:00,  9.53it/s]

Loading safetensors checkpoint shards:  99% Completed | 162/163 [00:17<00:00, 13.45it/s]

Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:17<00:00,  9.11it/s]

CPUInfer[0x60998f7d97b0]: Hello
WorkerPool[0x60998f7d97d0] 8 subpools, [numa:threads][0:8] [1:8] [2:8] [3:8] [4:8] [5:8] [6:8] [7:8] 
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 2, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 3, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 4, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 5, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 6, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 7, 8 threads
TP MOE layer 3, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_5"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_2"
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 4 at numa 4
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_1"
"models/DeepSeek-V3.2-INT8/_layer_3/_numa_4"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_7"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_3"
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_0"
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_6"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_0"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_6""models/DeepSeek-V3.2-INT8/_layer_3/_numa_2"
Loading from 
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_3"
"models/DeepSeek-V3.2-INT8/_layer_3/_numa_1"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_3/_numa_5"
TP MOE layer 4, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_0"
Creating AMX_MOE_TP 4 at numa 4
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_4"
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_7"
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 3 at numa 3
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_3""models/DeepSeek-V3.2-INT8/_layer_4/_numa_2"

Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_6"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_0"
Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_1"
"models/DeepSeek-V3.2-INT8/_layer_4/_numa_4"
Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_3""models/DeepSeek-V3.2-INT8/_layer_4/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_4/_numa_5"

Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_6"
TP MOE layer 5, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_4"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_5"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_2"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_7"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_0"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_1"
Creating AMX_MOE_TP 6 at numa 6
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_6"
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_3"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_4"
Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_2"
"models/DeepSeek-V3.2-INT8/_layer_5/_numa_7"Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_1"
"models/DeepSeek-V3.2-INT8/_layer_5/_numa_3"
Loading from 
"models/DeepSeek-V3.2-INT8/_layer_5/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_5/_numa_5"
TP MOE layer 6, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_4"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_6"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_5"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_3"
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_7"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_0"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_2"Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_1"

Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_6/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_6/_numa_5""models/DeepSeek-V3.2-INT8/_layer_6/_numa_6"

TP MOE layer 7, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_4"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_7"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_2"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_3"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_0"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_1"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_6"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_5"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_0"
Loading from Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_4""models/DeepSeek-V3.2-INT8/_layer_7/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_7/_numa_1""models/DeepSeek-V3.2-INT8/_layer_7/_numa_6""models/DeepSeek-V3.2-INT8/_layer_7/_numa_2"
Loading from 


"models/DeepSeek-V3.2-INT8/_layer_7/_numa_3"
"models/DeepSeek-V3.2-INT8/_layer_7/_numa_7"
TP MOE layer 8, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_4"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_5"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_7"
Creating AMX_MOE_TP 2 at numa 2
Loading from Creating AMX_MOE_TP 1 at numa 1
"models/DeepSeek-V3.2-INT8/_layer_8/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_0"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_6"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_8/_numa_4""models/DeepSeek-V3.2-INT8/_layer_8/_numa_7"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_8/_numa_2"Loading from 

"models/DeepSeek-V3.2-INT8/_layer_8/_numa_3"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_8/_numa_1"
TP MOE layer 9, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_2"
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_7"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_3"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_4"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_6"
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_1"
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_0"
Loading from Loading from Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_2"
"models/DeepSeek-V3.2-INT8/_layer_9/_numa_6"Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_1""models/DeepSeek-V3.2-INT8/_layer_9/_numa_7""models/DeepSeek-V3.2-INT8/_layer_9/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_9/_numa_4"

"models/DeepSeek-V3.2-INT8/_layer_9/_numa_3"


TP MOE layer 10, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_4"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_3"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_7"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_5"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_2"
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 6 at numa 6
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_1""models/DeepSeek-V3.2-INT8/_layer_10/_numa_6"

Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_2"Loading from Loading from Loading from 
Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_10/_numa_1"
"models/DeepSeek-V3.2-INT8/_layer_10/_numa_5""models/DeepSeek-V3.2-INT8/_layer_10/_numa_3""models/DeepSeek-V3.2-INT8/_layer_10/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_10/_numa_4"


TP MOE layer 11, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_2"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_1"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_6"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_4"
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_0"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_7"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_3""models/DeepSeek-V3.2-INT8/_layer_11/_numa_1"

Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_4"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_11/_numa_7""models/DeepSeek-V3.2-INT8/_layer_11/_numa_2"

Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_6"
TP MOE layer 12, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_4"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_6"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_2"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_7"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_3"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_0"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_1"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_5"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_2"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_12/_numa_5""models/DeepSeek-V3.2-INT8/_layer_12/_numa_6""models/DeepSeek-V3.2-INT8/_layer_12/_numa_1""models/DeepSeek-V3.2-INT8/_layer_12/_numa_7"
Loading from 

Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_4"

"models/DeepSeek-V3.2-INT8/_layer_12/_numa_3"
TP MOE layer 13, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_1"
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_7"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_4"
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_0"
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_6"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_2"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_0"Loading from Loading from 
Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_1"Loading from Loading from 
"models/DeepSeek-V3.2-INT8/_layer_13/_numa_4""models/DeepSeek-V3.2-INT8/_layer_13/_numa_5"Loading from 

"models/DeepSeek-V3.2-INT8/_layer_13/_numa_7""models/DeepSeek-V3.2-INT8/_layer_13/_numa_2"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_13/_numa_3"

"models/DeepSeek-V3.2-INT8/_layer_13/_numa_6"
TP MOE layer 14, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_4"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_2"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_6"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_7"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_0"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_5"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_3"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_5""models/DeepSeek-V3.2-INT8/_layer_14/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_14/_numa_2"

Loading from 
"models/DeepSeek-V3.2-INT8/_layer_14/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_14/_numa_7"

TP MOE layer 15, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_6"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_1"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_5"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_3"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_7"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_0"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_2"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_4"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_1"
"models/DeepSeek-V3.2-INT8/_layer_15/_numa_2""models/DeepSeek-V3.2-INT8/_layer_15/_numa_4"
Loading from 
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_3""models/DeepSeek-V3.2-INT8/_layer_15/_numa_6"

"models/DeepSeek-V3.2-INT8/_layer_15/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_5"
TP MOE layer 16, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 1 at numa 1
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_16/_numa_1"
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_0"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_2"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 6 at numa 6
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_3"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_16/_numa_4"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_0"
Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_16/_numa_5""models/DeepSeek-V3.2-INT8/_layer_16/_numa_3""models/DeepSeek-V3.2-INT8/_layer_16/_numa_7"


Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_4""models/DeepSeek-V3.2-INT8/_layer_16/_numa_2"
Loading from 
"models/DeepSeek-V3.2-INT8/_layer_16/_numa_1"
TP MOE layer 17, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_5"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_4"
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_1"
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_6"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_7"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_0"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_3"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_2"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_2""models/DeepSeek-V3.2-INT8/_layer_17/_numa_0"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_1"Loading from 

"models/DeepSeek-V3.2-INT8/_layer_17/_numa_3""models/DeepSeek-V3.2-INT8/_layer_17/_numa_4"


Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_17/_numa_5"
TP MOE layer 18, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_5"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_0"
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 3 at numa 3
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_18/_numa_3"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_2"
Creating AMX_MOE_TP 4 at numa 4
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_4"
Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_1"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_6"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_1"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_18/_numa_3"Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_0"

Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_2""models/DeepSeek-V3.2-INT8/_layer_18/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_18/_numa_6"
Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_18/_numa_7"

TP MOE layer 19, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_7"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_6"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_5"
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_0"
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_2"
Creating AMX_MOE_TP 4 at numa 4
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_4"
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_3"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_1"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_1"
Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_5""models/DeepSeek-V3.2-INT8/_layer_19/_numa_0"
"models/DeepSeek-V3.2-INT8/_layer_19/_numa_6""models/DeepSeek-V3.2-INT8/_layer_19/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_4"Loading from 



"models/DeepSeek-V3.2-INT8/_layer_19/_numa_3""models/DeepSeek-V3.2-INT8/_layer_19/_numa_7"

TP MOE layer 20, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_7"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_1"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_6"
Creating AMX_MOE_TP 4 at numa 4
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_4"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_0"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_5"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_0"Loading from Loading from 
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_20/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_6"

Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_20/_numa_3"
"models/DeepSeek-V3.2-INT8/_layer_20/_numa_1"
TP MOE layer 21, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_7"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_2"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_4"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_5"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_0"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_6"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_3"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_1"
Loading from 
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_2""models/DeepSeek-V3.2-INT8/_layer_21/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_4""models/DeepSeek-V3.2-INT8/_layer_21/_numa_6"


Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_5""models/DeepSeek-V3.2-INT8/_layer_21/_numa_7"

TP MOE layer 22, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_7"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_1"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_4"
Creating AMX_MOE_TP 6 at numa 6
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_6"
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_5"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_3"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_2"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_0"Loading from 
Loading from Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_4"Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_7"

"models/DeepSeek-V3.2-INT8/_layer_22/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_22/_numa_2""models/DeepSeek-V3.2-INT8/_layer_22/_numa_1""models/DeepSeek-V3.2-INT8/_layer_22/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_22/_numa_3"


TP MOE layer 23, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_4"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_2"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_1"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_7"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_3"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_6"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_0"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_5"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_0"
Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_23/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_6""models/DeepSeek-V3.2-INT8/_layer_23/_numa_3"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_23/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_1"


"models/DeepSeek-V3.2-INT8/_layer_23/_numa_5"
TP MOE layer 24, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_4"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_7"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_6"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_5"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_0"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_2"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_3"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_1"

"models/DeepSeek-V3.2-INT8/_layer_24/_numa_3"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_2""models/DeepSeek-V3.2-INT8/_layer_24/_numa_4"

Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_7"

"models/DeepSeek-V3.2-INT8/_layer_24/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_24/_numa_5"
TP MOE layer 25, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_6"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_2"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_4"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_3"
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_0"
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_5"
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_1"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_25/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_25/_numa_5"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_25/_numa_7""models/DeepSeek-V3.2-INT8/_layer_25/_numa_1"

Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_2"
TP MOE layer 26, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_7"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_6"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_3"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_5"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_1"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_2"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_4"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_1"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_4""models/DeepSeek-V3.2-INT8/_layer_26/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_26/_numa_3"

"models/DeepSeek-V3.2-INT8/_layer_26/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_26/_numa_2"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_26/_numa_6"
TP MOE layer 27, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_2"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_0"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_5"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_1"
Creating AMX_MOE_TP 6 at numa 6
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_6"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_27/_numa_7"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_0"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_2"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_27/_numa_1"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_4""models/DeepSeek-V3.2-INT8/_layer_27/_numa_3"
"models/DeepSeek-V3.2-INT8/_layer_27/_numa_7""models/DeepSeek-V3.2-INT8/_layer_27/_numa_6"


"models/DeepSeek-V3.2-INT8/_layer_27/_numa_5"
TP MOE layer 28, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_6"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_4"
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_1"
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_7"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_5"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_2"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_3"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_0"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_1""models/DeepSeek-V3.2-INT8/_layer_28/_numa_2"

Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_6"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_5""models/DeepSeek-V3.2-INT8/_layer_28/_numa_4""models/DeepSeek-V3.2-INT8/_layer_28/_numa_7"

"models/DeepSeek-V3.2-INT8/_layer_28/_numa_3"

TP MOE layer 29, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_0"
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 5 at numa 5
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_29/_numa_1"
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_6"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_3"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_4"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_7"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_0""models/DeepSeek-V3.2-INT8/_layer_29/_numa_1"Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_7"

Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_6""models/DeepSeek-V3.2-INT8/_layer_29/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_29/_numa_5"

Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_3"

TP MOE layer 30, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_4"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_1"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_7"
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 0 at numa 0
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_30/_numa_0"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_6"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_3"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_2"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_0"
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_4"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_30/_numa_1"
Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_2"
"models/DeepSeek-V3.2-INT8/_layer_30/_numa_6"
"models/DeepSeek-V3.2-INT8/_layer_30/_numa_7""models/DeepSeek-V3.2-INT8/_layer_30/_numa_3"

TP MOE layer 31, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_2"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_6"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_4"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_0"
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 7 at numa 7
Loading from Creating AMX_MOE_TP 3 at numa 3
"models/DeepSeek-V3.2-INT8/_layer_31/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_3"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_0"Loading from 
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_3"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_31/_numa_4""models/DeepSeek-V3.2-INT8/_layer_31/_numa_7"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_31/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_31/_numa_1"

"models/DeepSeek-V3.2-INT8/_layer_31/_numa_2"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_31/_numa_6"
TP MOE layer 32, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_6"
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_5"
Creating AMX_MOE_TP 4 at numa 4
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_4"
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_2"
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_1"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_0"
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_3"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_0"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_6"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_5"Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_2"

"models/DeepSeek-V3.2-INT8/_layer_32/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_4"

"models/DeepSeek-V3.2-INT8/_layer_32/_numa_1"
TP MOE layer 33, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_6"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_4"
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_5"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_3"
Creating AMX_MOE_TP 2 at numa 2
Creating AMX_MOE_TP 0 at numa 0
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_2""models/DeepSeek-V3.2-INT8/_layer_33/_numa_0"

Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_1"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_1"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_2"
Loading from 
"models/DeepSeek-V3.2-INT8/_layer_33/_numa_6"
Loading from 
Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_3""models/DeepSeek-V3.2-INT8/_layer_33/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_33/_numa_4"

Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_7"
TP MOE layer 34, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_7"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_5"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_0"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_6"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_2"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_1"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_4"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_0""models/DeepSeek-V3.2-INT8/_layer_34/_numa_7"

Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_6"
Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_3"
"models/DeepSeek-V3.2-INT8/_layer_34/_numa_4""models/DeepSeek-V3.2-INT8/_layer_34/_numa_1"
"models/DeepSeek-V3.2-INT8/_layer_34/_numa_5"

TP MOE layer 35, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_3"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_4"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_5"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_6"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_7"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_1"
Creating AMX_MOE_TP 0 at numa 0
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_0"
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_2"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_0"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_2"
Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_3"Loading from 
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_5"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_35/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_35/_numa_4""models/DeepSeek-V3.2-INT8/_layer_35/_numa_1"
"models/DeepSeek-V3.2-INT8/_layer_35/_numa_6"

TP MOE layer 36, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_4"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_2"
Creating AMX_MOE_TP 3 at numa 3
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_3"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_6"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_1"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_5"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_0"
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_2"Loading from 

"models/DeepSeek-V3.2-INT8/_layer_36/_numa_5"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_36/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_36/_numa_6"Loading from Loading from 
"models/DeepSeek-V3.2-INT8/_layer_36/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_36/_numa_3"
TP MOE layer 37, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_4"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_5"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_1"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_2"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_6"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_7"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_3"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_0"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_0""models/DeepSeek-V3.2-INT8/_layer_37/_numa_2""models/DeepSeek-V3.2-INT8/_layer_37/_numa_1"
Loading from Loading from Loading from 
"models/DeepSeek-V3.2-INT8/_layer_37/_numa_6"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_37/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_37/_numa_4""models/DeepSeek-V3.2-INT8/_layer_37/_numa_7""models/DeepSeek-V3.2-INT8/_layer_37/_numa_3"



TP MOE layer 38, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_3"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_6"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_5"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_2"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_0"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_1"
Creating AMX_MOE_TP 7 at numa 7
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_7"
Creating AMX_MOE_TP 4 at numa 4
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_4"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_0""models/DeepSeek-V3.2-INT8/_layer_38/_numa_4"
Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_5"


Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_7""models/DeepSeek-V3.2-INT8/_layer_38/_numa_2""models/DeepSeek-V3.2-INT8/_layer_38/_numa_6""models/DeepSeek-V3.2-INT8/_layer_38/_numa_3"



TP MOE layer 39, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_0"
Creating AMX_MOE_TP 4 at numa 4
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 3 at numa 3
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_4"
"models/DeepSeek-V3.2-INT8/_layer_39/_numa_7"
Creating AMX_MOE_TP 5 at numa 5
Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_3"
Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_5"
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_1"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_2"
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_6"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_1"Loading from Loading from 
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_39/_numa_5"
Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_6"Loading from 
"models/DeepSeek-V3.2-INT8/_layer_39/_numa_3""models/DeepSeek-V3.2-INT8/_layer_39/_numa_2""models/DeepSeek-V3.2-INT8/_layer_39/_numa_0"


"models/DeepSeek-V3.2-INT8/_layer_39/_numa_4"
TP MOE layer 40, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8
Creating AMX_MOE_TP 1 at numa 1
Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_1"
Creating AMX_MOE_TP 2 at numa 2
Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_2"
Creating AMX_MOE_TP 5 at numa 5
Creating AMX_MOE_TP 4 at numa 4
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_5"
"models/DeepSeek-V3.2-INT8/_layer_40/_numa_4"
Creating AMX_MOE_TP 0 at numa 0
Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_0"
Creating AMX_MOE_TP 3 at numa 3
Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_3"
Creating AMX_MOE_TP 7 at numa 7
Creating AMX_MOE_TP 6 at numa 6
Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_7"
Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_6"
TP Load from file models/DeepSeek-V3.2-INT8
Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_1"
Loading from 
Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_5"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_6""models/DeepSeek-V3.2-INT8/_layer_40/_numa_7"
"models/DeepSeek-V3.2-INT8/_layer_40/_numa_3""models/DeepSeek-V3.2-INT8/_layer_40/_numa_4"
Loading from 

"models/DeepSeek-V3.2-INT8/_layer_40/_numa_2"

如果切换成FP8原生weight:

python -m sglang.launch_server   --host 0.0.0.0   --port 8000   --model models/DeepSeek-V3.2   --trust-remote-code   --mem-fraction-static 0.9   --chunked-prefill-size 512   --served-model-name DeepSeek-V3.2   --enable-mixed-chunk   --kt-weight-path models/DeepSeek-V3.2   --kt-cpuinfer 64  --kt-threadpool-count 8   --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --kt-method FP8 --attention-backend triton --cuda-graph-max-bs 1 --reasoning-parser deepseek-v3

那么会出现eof错误：

[2026-01-15 19:55:31] WARNING server_args.py:1032: DP attention is enabled for DeepSeek NSA.
[2026-01-15 19:55:31] WARNING server_args.py:1055: Setting page size to 64 for DeepSeek NSA.
[2026-01-15 19:55:31] WARNING server_args.py:1063: Setting KV cache dtype to bfloat16 for DeepSeek NSA.
NSA_DUAL_STREAM=True NSA_FUSE_TOPK=True NSA_FLASHMLA_BACKEND_DECODE_COMPUTE_FP8=True NSA_QUANT_K_CACHE_FAST=True NSA_DEQUANT_K_CACHE_FAST=True NSA_ENABLE_MTP_PRECOMPUTE_METADATA=True 
[2026-01-15 19:55:32] server_args=ServerArgs(model_path='models/DeepSeek-V3.2', tokenizer_path='models/DeepSeek-V3.2', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, limit_mm_data_per_request=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='bfloat16', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.9, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=512, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=64, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=714365051, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='DeepSeek-V3.2', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='deepseek-v3', tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='models/DeepSeek-V3.2', kt_method='FP8', kt_cpuinfer=64, kt_threadpool_count=8, kt_num_gpu_experts=0, kt_max_deferred_experts_per_token=2, kt_gpu_prefill_token_threshold=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=1, cuda_graph_bs=[1], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-01-15 19:55:32] No chat template found, defaulting to 'string' content format
[2026-01-15 19:55:38] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2026-01-15 19:55:38] Init torch distributed ends. mem usage=0.00 GB
[2026-01-15 19:55:38] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmasrConfig' from 'transformers' (/home/jovyan/yue/venv_py12/lib/python3.12/site-packages/transformers/__init__.py)
[2026-01-15 19:55:39] Load weight begin. avail mem=46.85 GB
[2026-01-15 19:55:39] Detected fp8 checkpoint.
[FP8SafeTensorLoader] Detected format: deepseek

Loading safetensors checkpoint shards:   0% Completed | 0/163 [00:00<?, ?it/s]

Loading safetensors checkpoint shards:   2% Completed | 3/163 [00:00<00:06, 26.40it/s]

Loading safetensors checkpoint shards:   4% Completed | 6/163 [00:00<00:19,  8.25it/s]

Loading safetensors checkpoint shards:   5% Completed | 8/163 [00:00<00:17,  8.73it/s]

Loading safetensors checkpoint shards:   7% Completed | 11/163 [00:00<00:12, 11.76it/s]

Loading safetensors checkpoint shards:   9% Completed | 15/163 [00:01<00:08, 17.06it/s]

Loading safetensors checkpoint shards:  12% Completed | 19/163 [00:01<00:06, 21.15it/s]

Loading safetensors checkpoint shards:  13% Completed | 22/163 [00:01<00:10, 13.18it/s]

Loading safetensors checkpoint shards:  15% Completed | 25/163 [00:01<00:11, 11.87it/s]

Loading safetensors checkpoint shards:  17% Completed | 27/163 [00:02<00:12, 11.18it/s]

Loading safetensors checkpoint shards:  18% Completed | 29/163 [00:02<00:12, 10.74it/s]

Loading safetensors checkpoint shards:  19% Completed | 31/163 [00:02<00:11, 11.28it/s]

Loading safetensors checkpoint shards:  20% Completed | 33/163 [00:02<00:16,  7.94it/s]

Loading safetensors checkpoint shards:  22% Completed | 36/163 [00:03<00:13,  9.60it/s]

Loading safetensors checkpoint shards:  23% Completed | 38/163 [00:03<00:13,  9.49it/s]

Loading safetensors checkpoint shards:  25% Completed | 40/163 [00:03<00:12,  9.64it/s]

Loading safetensors checkpoint shards:  26% Completed | 42/163 [00:03<00:12,  9.70it/s]

Loading safetensors checkpoint shards:  27% Completed | 44/163 [00:04<00:12,  9.63it/s]

Loading safetensors checkpoint shards:  28% Completed | 46/163 [00:04<00:12,  9.39it/s]

Loading safetensors checkpoint shards:  29% Completed | 47/163 [00:04<00:12,  9.31it/s]

Loading safetensors checkpoint shards:  30% Completed | 49/163 [00:04<00:11,  9.53it/s]

Loading safetensors checkpoint shards:  31% Completed | 51/163 [00:04<00:11,  9.66it/s]

Loading safetensors checkpoint shards:  33% Completed | 53/163 [00:04<00:10, 10.58it/s]

Loading safetensors checkpoint shards:  34% Completed | 55/163 [00:05<00:10, 10.07it/s]

Loading safetensors checkpoint shards:  35% Completed | 57/163 [00:05<00:10, 10.08it/s]

Loading safetensors checkpoint shards:  36% Completed | 59/163 [00:05<00:10, 10.09it/s]

Loading safetensors checkpoint shards:  37% Completed | 61/163 [00:05<00:10,  9.98it/s]

Loading safetensors checkpoint shards:  39% Completed | 63/163 [00:05<00:10,  9.75it/s]

Loading safetensors checkpoint shards:  40% Completed | 65/163 [00:06<00:10,  9.61it/s]

Loading safetensors checkpoint shards:  40% Completed | 66/163 [00:06<00:10,  9.61it/s]

Loading safetensors checkpoint shards:  41% Completed | 67/163 [00:06<00:10,  9.30it/s]

Loading safetensors checkpoint shards:  42% Completed | 69/163 [00:06<00:09,  9.55it/s]

Loading safetensors checkpoint shards:  43% Completed | 70/163 [00:06<00:09,  9.45it/s]

Loading safetensors checkpoint shards:  44% Completed | 71/163 [00:07<00:15,  5.90it/s]

Loading safetensors checkpoint shards:  45% Completed | 73/163 [00:07<00:11,  7.66it/s]

Loading safetensors checkpoint shards:  45% Completed | 74/163 [00:07<00:11,  8.01it/s]

Loading safetensors checkpoint shards:  47% Completed | 76/163 [00:07<00:09,  8.99it/s]

Loading safetensors checkpoint shards:  48% Completed | 78/163 [00:07<00:09,  9.41it/s]

Loading safetensors checkpoint shards:  49% Completed | 80/163 [00:07<00:08,  9.58it/s]

Loading safetensors checkpoint shards:  50% Completed | 82/163 [00:08<00:08,  9.46it/s]

Loading safetensors checkpoint shards:  51% Completed | 83/163 [00:08<00:08,  9.31it/s]

Loading safetensors checkpoint shards:  52% Completed | 84/163 [00:08<00:08,  9.18it/s]

Loading safetensors checkpoint shards:  52% Completed | 85/163 [00:08<00:08,  9.35it/s]

Loading safetensors checkpoint shards:  53% Completed | 86/163 [00:08<00:08,  9.02it/s]

Loading safetensors checkpoint shards:  54% Completed | 88/163 [00:08<00:08,  9.31it/s]

Loading safetensors checkpoint shards:  55% Completed | 89/163 [00:08<00:08,  8.95it/s]

Loading safetensors checkpoint shards:  56% Completed | 91/163 [00:09<00:07,  9.22it/s]

Loading safetensors checkpoint shards:  56% Completed | 92/163 [00:09<00:07,  9.12it/s]

Loading safetensors checkpoint shards:  57% Completed | 93/163 [00:09<00:07,  9.26it/s]

Loading safetensors checkpoint shards:  58% Completed | 94/163 [00:09<00:07,  9.09it/s]

Loading safetensors checkpoint shards:  59% Completed | 96/163 [00:09<00:06, 10.29it/s]

Loading safetensors checkpoint shards:  60% Completed | 98/163 [00:09<00:06,  9.55it/s]

Loading safetensors checkpoint shards:  61% Completed | 100/163 [00:10<00:06,  9.57it/s]

Loading safetensors checkpoint shards:  63% Completed | 102/163 [00:10<00:06,  9.64it/s]

Loading safetensors checkpoint shards:  64% Completed | 104/163 [00:10<00:06,  9.46it/s]

Loading safetensors checkpoint shards:  64% Completed | 105/163 [00:10<00:06,  9.30it/s]

Loading safetensors checkpoint shards:  65% Completed | 106/163 [00:10<00:06,  9.21it/s]

Loading safetensors checkpoint shards:  66% Completed | 108/163 [00:10<00:06,  9.16it/s]

Loading safetensors checkpoint shards:  67% Completed | 109/163 [00:11<00:05,  9.28it/s]

Loading safetensors checkpoint shards:  67% Completed | 110/163 [00:11<00:05,  9.28it/s]

Loading safetensors checkpoint shards:  69% Completed | 112/163 [00:11<00:05,  9.25it/s]

Loading safetensors checkpoint shards:  69% Completed | 113/163 [00:11<00:05,  9.25it/s]

Loading safetensors checkpoint shards:  70% Completed | 114/163 [00:11<00:05,  9.21it/s]

Loading safetensors checkpoint shards:  71% Completed | 116/163 [00:11<00:05,  9.30it/s]

Loading safetensors checkpoint shards:  72% Completed | 118/163 [00:11<00:04, 10.41it/s]

Loading safetensors checkpoint shards:  74% Completed | 120/163 [00:12<00:04,  9.87it/s]

Loading safetensors checkpoint shards:  74% Completed | 121/163 [00:12<00:06,  6.01it/s]

Loading safetensors checkpoint shards:  76% Completed | 124/163 [00:12<00:04,  8.11it/s]

Loading safetensors checkpoint shards:  77% Completed | 125/163 [00:12<00:04,  8.13it/s]

Loading safetensors checkpoint shards:  78% Completed | 127/163 [00:13<00:04,  8.67it/s]

Loading safetensors checkpoint shards:  79% Completed | 128/163 [00:13<00:03,  8.80it/s]

Loading safetensors checkpoint shards:  80% Completed | 130/163 [00:13<00:03,  9.00it/s]

Loading safetensors checkpoint shards:  80% Completed | 131/163 [00:13<00:03,  9.16it/s]

Loading safetensors checkpoint shards:  81% Completed | 132/163 [00:13<00:03,  9.30it/s]

Loading safetensors checkpoint shards:  82% Completed | 133/163 [00:13<00:03,  9.09it/s]

Loading safetensors checkpoint shards:  83% Completed | 135/163 [00:13<00:02,  9.40it/s]

Loading safetensors checkpoint shards:  83% Completed | 136/163 [00:14<00:02,  9.30it/s]

Loading safetensors checkpoint shards:  85% Completed | 138/163 [00:14<00:02,  9.29it/s]

Loading safetensors checkpoint shards:  86% Completed | 140/163 [00:14<00:02,  9.58it/s]

Loading safetensors checkpoint shards:  87% Completed | 141/163 [00:14<00:02,  9.47it/s]

Loading safetensors checkpoint shards:  87% Completed | 142/163 [00:14<00:02,  9.57it/s]

Loading safetensors checkpoint shards:  88% Completed | 143/163 [00:14<00:02,  9.48it/s]

Loading safetensors checkpoint shards:  88% Completed | 144/163 [00:14<00:02,  9.16it/s]

Loading safetensors checkpoint shards:  90% Completed | 146/163 [00:15<00:01,  9.37it/s]

Loading safetensors checkpoint shards:  90% Completed | 147/163 [00:15<00:01,  9.28it/s]

Loading safetensors checkpoint shards:  91% Completed | 149/163 [00:15<00:01,  9.27it/s]

Loading safetensors checkpoint shards:  92% Completed | 150/163 [00:15<00:01,  9.27it/s]

Loading safetensors checkpoint shards:  93% Completed | 151/163 [00:15<00:01,  9.27it/s]

Loading safetensors checkpoint shards:  93% Completed | 152/163 [00:15<00:01,  8.84it/s]

Loading safetensors checkpoint shards:  94% Completed | 154/163 [00:16<00:00,  9.25it/s]

Loading safetensors checkpoint shards:  95% Completed | 155/163 [00:16<00:00,  9.12it/s]

Loading safetensors checkpoint shards:  96% Completed | 157/163 [00:16<00:00,  9.12it/s]

Loading safetensors checkpoint shards:  97% Completed | 158/163 [00:16<00:00,  9.24it/s]

Loading safetensors checkpoint shards:  98% Completed | 159/163 [00:16<00:00,  9.30it/s]

Loading safetensors checkpoint shards:  99% Completed | 162/163 [00:16<00:00, 13.14it/s]

Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:16<00:00,  9.75it/s]

CPUInfer[0x5dd5a207d430]: Hello
WorkerPool[0x5dd5a207d450] 8 subpools, [numa:threads][0:8] [1:8] [2:8] [3:8] [4:8] [5:8] [6:8] [7:8] 
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 2, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 3, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 4, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 5, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 6, 8 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 7, 8 threads
TP MOE layer 3, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 4, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 5, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 6, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 3 at numa 3
TP MOE layer 7, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 8, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 5 at numa 5
TP MOE layer 9, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 2 at numa 2
TP MOE layer 10, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 1 at numa 1
TP MOE layer 11, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 12, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 6 at numa 6
TP MOE layer 13, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 14, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 15, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 6 at numa 6
TP MOE layer 16, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 17, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 3 at numa 3
TP MOE layer 18, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 19, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 20, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 21, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 22, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 23, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 24, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 6 at numa 6
TP MOE layer 25, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 5 at numa 5
TP MOE layer 26, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 1 at numa 1
TP MOE layer 27, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 0 at numa 0
TP MOE layer 28, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 29, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 4 at numa 4
TP MOE layer 30, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 6 at numa 6
TP MOE layer 31, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 1 at numa 1
TP MOE layer 32, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 7 at numa 7
TP MOE layer 33, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 1 at numa 1
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 0 at numa 0
TP MOE layer 34, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8
Created AMX_FP8_MOE_TP 3 at numa 3
Created AMX_FP8_MOE_TP 0 at numa 0
Created AMX_FP8_MOE_TP 7 at numa 7
Created AMX_FP8_MOE_TP 2 at numa 2
Created AMX_FP8_MOE_TP 5 at numa 5
Created AMX_FP8_MOE_TP 4 at numa 4
Created AMX_FP8_MOE_TP 6 at numa 6
Created AMX_FP8_MOE_TP 1 at numa 1
TP MOE layer 35, pool: 0x5dd5a207d450, expert num: 256, num_exper[2026-01-15 19:57:43] Rank 0 scheduler is dead. Please check if there are relevant logs.
[2026-01-15 19:57:43] Exit code: -9
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/jovyan/yue/sglang/python/sglang/launch_server.py", line 32, in <module>
    run_server(server_args)
  File "/home/jovyan/yue/sglang/python/sglang/launch_server.py", line 25, in run_server
    launch_server(server_args)
  File "/home/jovyan/yue/sglang/python/sglang/srt/entrypoints/http_server.py", line 1692, in launch_server
    _launch_subprocesses(
  File "/home/jovyan/yue/sglang/python/sglang/srt/entrypoints/engine.py", line 966, in _launch_subprocesses
    data = scheduler_pipe_readers[i].recv()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError
<!-- gh-comment-id:3756663931 --> @SchenbergZY commented on GitHub (Jan 15, 2026): > --kt-threadpool-count 这个对应你的numa数，几numa就设置几，我估计是这个问题，如果确实是8 numa，BIOS里应该有设置可以修改，调到2或者4，对应你实际socket数量 我的lscpu如下： ``` Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 384 On-line CPU(s) list: 0-383 Vendor ID: AuthenticAMD Model name: AMD EPYC 9654 96-Core Processor CPU family: 25 Model: 17 Thread(s) per core: 2 Core(s) per socket: 96 Socket(s): 2 Stepping: 1 Frequency boost: enabled CPU(s) scaling MHz: 63% CPU max MHz: 2400.0000 CPU min MHz: 1500.0000 BogoMIPS: 4800.08 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good amd_lbr_ v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx 16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy s vm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoex t perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpc id cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cq m_mbm_local user_shstk avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin cppc amd_ibpb_ret arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeas sists pausefilter pfthreshold avic v_vmsave_vmload vgif x2avic v_spec_ctrl vnmi avx512vb mi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpo pcntdq la57 rdpid overflow_recov succor smca fsrm flush_l1d debug_swap Virtualization features: Virtualization: AMD-V Caches (sum of all): L1d: 6 MiB (192 instances) L1i: 6 MiB (192 instances) L2: 192 MiB (192 instances) L3: 768 MiB (24 instances) NUMA: NUMA node(s): 24 NUMA node0 CPU(s): 0-7,192-199 NUMA node1 CPU(s): 8-15,200-207 NUMA node2 CPU(s): 16-23,208-215 NUMA node3 CPU(s): 24-31,216-223 NUMA node4 CPU(s): 32-39,224-231 NUMA node5 CPU(s): 40-47,232-239 NUMA node6 CPU(s): 48-55,240-247 NUMA node7 CPU(s): 56-63,248-255 NUMA node8 CPU(s): 64-71,256-263 NUMA node9 CPU(s): 72-79,264-271 NUMA node10 CPU(s): 80-87,272-279 NUMA node11 CPU(s): 88-95,280-287 NUMA node12 CPU(s): 96-103,288-295 NUMA node13 CPU(s): 104-111,296-303 NUMA node14 CPU(s): 112-119,304-311 NUMA node15 CPU(s): 120-127,312-319 NUMA node16 CPU(s): 128-135,320-327 NUMA node17 CPU(s): 136-143,328-335 NUMA node18 CPU(s): 144-151,336-343 NUMA node19 CPU(s): 152-159,344-351 NUMA node20 CPU(s): 160-167,352-359 NUMA node21 CPU(s): 168-175,360-367 NUMA node22 CPU(s): 176-183,368-375 NUMA node23 CPU(s): 184-191,376-383 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Reg file data sampling: Not affected Retbleed: Not affected Spec rstack overflow: Vulnerable Spec store bypass: Vulnerable Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers Spectre v2: Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Not affected; BHI: Not affecte d Srbds: Not affected Tsx async abort: Not affected Vmscape: Vulnerable ``` 想请问下--kt-threadpool-count 该怎么设置呢，是24么，可是24不能被DS-V3.2的2048的intermediate_size整除。所以设16？ 下面是我运行INT8_NUMA8的命令： ``` python -m sglang.launch_server --host 0.0.0.0 --port 8000 --model models/DeepSeek-V3.2 --trust-remote-code --mem-fraction-static 0.6 --chunked-prefill-size 512 --served-model-name DeepSeek-V3.2 --enable-mixed-chunk --kt-weight-path models/DeepSeek-V3.2-INT8 --kt-cpuinfer 64 --kt-threadpool-count 8 --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --tensor-parallel-size 1 --kt-method AMXINT8 --attention-backend triton --cuda-graph-max-bs 1 ``` 结果在loading第40层的时候卡死： ``` [2026-01-15 16:40:57] WARNING server_args.py:1032: DP attention is enabled for DeepSeek NSA. [2026-01-15 16:40:57] WARNING server_args.py:1055: Setting page size to 64 for DeepSeek NSA. [2026-01-15 16:40:57] WARNING server_args.py:1063: Setting KV cache dtype to bfloat16 for DeepSeek NSA. NSA_DUAL_STREAM=True NSA_FUSE_TOPK=True NSA_FLASHMLA_BACKEND_DECODE_COMPUTE_FP8=True NSA_QUANT_K_CACHE_FAST=True NSA_DEQUANT_K_CACHE_FAST=True NSA_ENABLE_MTP_PRECOMPUTE_METADATA=True [2026-01-15 16:40:57] server_args=ServerArgs(model_path='models/DeepSeek-V3.2', tokenizer_path='models/DeepSeek-V3.2', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, limit_mm_data_per_request=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='bfloat16', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.9, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=512, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=64, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=284622794, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='DeepSeek-V3.2', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='deepseek-v3', tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='models/DeepSeek-V3.2-INT8', kt_method='AMXINT8', kt_cpuinfer=64, kt_threadpool_count=8, kt_num_gpu_experts=0, kt_max_deferred_experts_per_token=2, kt_gpu_prefill_token_threshold=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=1, cuda_graph_bs=[1], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None) [2026-01-15 16:40:57] No chat template found, defaulting to 'string' content format [2026-01-15 16:41:03] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2026-01-15 16:41:03] Init torch distributed ends. mem usage=0.00 GB [2026-01-15 16:41:04] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmasrConfig' from 'transformers' (/home/jovyan/yue/venv_py12/lib/python3.12/site-packages/transformers/__init__.py) [2026-01-15 16:41:04] Load weight begin. avail mem=46.85 GB [2026-01-15 16:41:04] Detected fp8 checkpoint. Loading safetensors checkpoint shards: 0% Completed | 0/163 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 2% Completed | 3/163 [00:00<00:06, 24.96it/s] Loading safetensors checkpoint shards: 4% Completed | 6/163 [00:00<00:18, 8.37it/s] Loading safetensors checkpoint shards: 5% Completed | 8/163 [00:00<00:16, 9.14it/s] Loading safetensors checkpoint shards: 7% Completed | 11/163 [00:00<00:12, 12.10it/s] Loading safetensors checkpoint shards: 9% Completed | 15/163 [00:01<00:08, 17.37it/s] Loading safetensors checkpoint shards: 12% Completed | 19/163 [00:01<00:06, 21.11it/s] Loading safetensors checkpoint shards: 13% Completed | 22/163 [00:01<00:11, 11.90it/s] Loading safetensors checkpoint shards: 15% Completed | 24/163 [00:01<00:12, 10.70it/s] Loading safetensors checkpoint shards: 16% Completed | 26/163 [00:02<00:13, 10.21it/s] Loading safetensors checkpoint shards: 17% Completed | 28/163 [00:02<00:13, 9.83it/s] Loading safetensors checkpoint shards: 18% Completed | 30/163 [00:02<00:13, 10.00it/s] Loading safetensors checkpoint shards: 20% Completed | 32/163 [00:02<00:13, 9.47it/s] Loading safetensors checkpoint shards: 21% Completed | 34/163 [00:03<00:13, 9.49it/s] Loading safetensors checkpoint shards: 22% Completed | 36/163 [00:03<00:17, 7.45it/s] Loading safetensors checkpoint shards: 23% Completed | 37/163 [00:03<00:16, 7.60it/s] Loading safetensors checkpoint shards: 23% Completed | 38/163 [00:03<00:16, 7.60it/s] Loading safetensors checkpoint shards: 24% Completed | 39/163 [00:03<00:16, 7.70it/s] Loading safetensors checkpoint shards: 25% Completed | 41/163 [00:04<00:14, 8.21it/s] Loading safetensors checkpoint shards: 26% Completed | 42/163 [00:04<00:14, 8.45it/s] Loading safetensors checkpoint shards: 26% Completed | 43/163 [00:04<00:14, 8.29it/s] Loading safetensors checkpoint shards: 27% Completed | 44/163 [00:04<00:13, 8.55it/s] Loading safetensors checkpoint shards: 28% Completed | 45/163 [00:04<00:13, 8.68it/s] Loading safetensors checkpoint shards: 28% Completed | 46/163 [00:04<00:13, 8.36it/s] Loading safetensors checkpoint shards: 29% Completed | 47/163 [00:04<00:13, 8.32it/s] Loading safetensors checkpoint shards: 30% Completed | 49/163 [00:04<00:13, 8.74it/s] Loading safetensors checkpoint shards: 31% Completed | 50/163 [00:05<00:12, 8.95it/s] Loading safetensors checkpoint shards: 31% Completed | 51/163 [00:05<00:12, 8.68it/s] Loading safetensors checkpoint shards: 32% Completed | 52/163 [00:05<00:12, 8.88it/s] Loading safetensors checkpoint shards: 33% Completed | 54/163 [00:05<00:11, 9.90it/s] Loading safetensors checkpoint shards: 34% Completed | 55/163 [00:05<00:11, 9.19it/s] Loading safetensors checkpoint shards: 35% Completed | 57/163 [00:05<00:11, 9.22it/s] Loading safetensors checkpoint shards: 36% Completed | 58/163 [00:05<00:11, 9.14it/s] Loading safetensors checkpoint shards: 36% Completed | 59/163 [00:06<00:11, 9.27it/s] Loading safetensors checkpoint shards: 37% Completed | 60/163 [00:06<00:11, 8.82it/s] Loading safetensors checkpoint shards: 38% Completed | 62/163 [00:06<00:11, 8.87it/s] Loading safetensors checkpoint shards: 39% Completed | 63/163 [00:06<00:11, 8.72it/s] Loading safetensors checkpoint shards: 39% Completed | 64/163 [00:06<00:11, 8.86it/s] Loading safetensors checkpoint shards: 40% Completed | 65/163 [00:06<00:11, 8.55it/s] Loading safetensors checkpoint shards: 40% Completed | 66/163 [00:06<00:11, 8.66it/s] Loading safetensors checkpoint shards: 41% Completed | 67/163 [00:07<00:11, 8.35it/s] Loading safetensors checkpoint shards: 42% Completed | 69/163 [00:07<00:10, 8.66it/s] Loading safetensors checkpoint shards: 43% Completed | 70/163 [00:07<00:10, 8.61it/s] Loading safetensors checkpoint shards: 44% Completed | 71/163 [00:07<00:10, 8.84it/s] Loading safetensors checkpoint shards: 44% Completed | 72/163 [00:07<00:10, 8.57it/s] Loading safetensors checkpoint shards: 45% Completed | 73/163 [00:07<00:10, 8.73it/s] Loading safetensors checkpoint shards: 45% Completed | 74/163 [00:07<00:10, 8.73it/s] Loading safetensors checkpoint shards: 47% Completed | 76/163 [00:08<00:15, 5.73it/s] Loading safetensors checkpoint shards: 48% Completed | 78/163 [00:08<00:11, 7.14it/s] Loading safetensors checkpoint shards: 48% Completed | 79/163 [00:08<00:11, 7.51it/s] Loading safetensors checkpoint shards: 49% Completed | 80/163 [00:08<00:10, 7.69it/s] Loading safetensors checkpoint shards: 50% Completed | 81/163 [00:08<00:10, 7.59it/s] Loading safetensors checkpoint shards: 50% Completed | 82/163 [00:08<00:10, 8.05it/s] Loading safetensors checkpoint shards: 51% Completed | 83/163 [00:09<00:09, 8.02it/s] Loading safetensors checkpoint shards: 52% Completed | 84/163 [00:09<00:09, 8.01it/s] Loading safetensors checkpoint shards: 52% Completed | 85/163 [00:09<00:09, 8.24it/s] Loading safetensors checkpoint shards: 53% Completed | 86/163 [00:09<00:09, 7.89it/s] Loading safetensors checkpoint shards: 53% Completed | 87/163 [00:09<00:09, 8.40it/s] Loading safetensors checkpoint shards: 54% Completed | 88/163 [00:09<00:08, 8.62it/s] Loading safetensors checkpoint shards: 55% Completed | 89/163 [00:09<00:08, 8.40it/s] Loading safetensors checkpoint shards: 56% Completed | 91/163 [00:09<00:08, 8.94it/s] Loading safetensors checkpoint shards: 56% Completed | 92/163 [00:10<00:07, 8.90it/s] Loading safetensors checkpoint shards: 57% Completed | 93/163 [00:10<00:07, 9.14it/s] Loading safetensors checkpoint shards: 58% Completed | 94/163 [00:10<00:07, 8.86it/s] Loading safetensors checkpoint shards: 59% Completed | 96/163 [00:10<00:06, 10.12it/s] Loading safetensors checkpoint shards: 60% Completed | 97/163 [00:10<00:06, 9.85it/s] Loading safetensors checkpoint shards: 60% Completed | 98/163 [00:10<00:07, 9.16it/s] Loading safetensors checkpoint shards: 61% Completed | 99/163 [00:10<00:06, 9.29it/s] Loading safetensors checkpoint shards: 61% Completed | 100/163 [00:10<00:07, 8.87it/s] Loading safetensors checkpoint shards: 62% Completed | 101/163 [00:11<00:06, 8.93it/s] Loading safetensors checkpoint shards: 63% Completed | 102/163 [00:11<00:06, 8.78it/s] Loading safetensors checkpoint shards: 63% Completed | 103/163 [00:11<00:06, 9.00it/s] Loading safetensors checkpoint shards: 64% Completed | 104/163 [00:11<00:07, 8.41it/s] Loading safetensors checkpoint shards: 64% Completed | 105/163 [00:11<00:07, 8.23it/s] Loading safetensors checkpoint shards: 65% Completed | 106/163 [00:11<00:06, 8.30it/s] Loading safetensors checkpoint shards: 66% Completed | 107/163 [00:11<00:06, 8.69it/s] Loading safetensors checkpoint shards: 66% Completed | 108/163 [00:11<00:06, 8.43it/s] Loading safetensors checkpoint shards: 67% Completed | 109/163 [00:12<00:06, 8.76it/s] Loading safetensors checkpoint shards: 67% Completed | 110/163 [00:12<00:06, 8.83it/s] Loading safetensors checkpoint shards: 69% Completed | 112/163 [00:12<00:05, 8.73it/s] Loading safetensors checkpoint shards: 69% Completed | 113/163 [00:12<00:05, 8.60it/s] Loading safetensors checkpoint shards: 70% Completed | 114/163 [00:12<00:05, 8.50it/s] Loading safetensors checkpoint shards: 71% Completed | 115/163 [00:12<00:05, 8.71it/s] Loading safetensors checkpoint shards: 71% Completed | 116/163 [00:12<00:05, 8.42it/s] Loading safetensors checkpoint shards: 72% Completed | 117/163 [00:12<00:05, 8.58it/s] Loading safetensors checkpoint shards: 73% Completed | 119/163 [00:13<00:04, 9.73it/s] Loading safetensors checkpoint shards: 74% Completed | 120/163 [00:13<00:04, 9.07it/s] Loading safetensors checkpoint shards: 75% Completed | 122/163 [00:13<00:04, 9.21it/s] Loading safetensors checkpoint shards: 75% Completed | 123/163 [00:13<00:04, 9.27it/s] Loading safetensors checkpoint shards: 76% Completed | 124/163 [00:13<00:04, 9.19it/s] Loading safetensors checkpoint shards: 77% Completed | 125/163 [00:13<00:04, 8.74it/s] Loading safetensors checkpoint shards: 78% Completed | 127/163 [00:14<00:06, 5.85it/s] Loading safetensors checkpoint shards: 79% Completed | 129/163 [00:14<00:04, 7.09it/s] Loading safetensors checkpoint shards: 80% Completed | 130/163 [00:14<00:04, 7.12it/s] Loading safetensors checkpoint shards: 80% Completed | 131/163 [00:14<00:04, 7.46it/s] Loading safetensors checkpoint shards: 81% Completed | 132/163 [00:14<00:04, 7.65it/s] Loading safetensors checkpoint shards: 82% Completed | 133/163 [00:15<00:04, 7.31it/s] Loading safetensors checkpoint shards: 83% Completed | 135/163 [00:15<00:03, 8.25it/s] Loading safetensors checkpoint shards: 83% Completed | 136/163 [00:15<00:03, 8.44it/s] Loading safetensors checkpoint shards: 85% Completed | 138/163 [00:15<00:02, 8.81it/s] Loading safetensors checkpoint shards: 86% Completed | 140/163 [00:15<00:02, 9.31it/s] Loading safetensors checkpoint shards: 87% Completed | 141/163 [00:15<00:02, 9.39it/s] Loading safetensors checkpoint shards: 88% Completed | 143/163 [00:16<00:02, 9.55it/s] Loading safetensors checkpoint shards: 88% Completed | 144/163 [00:16<00:02, 9.18it/s] Loading safetensors checkpoint shards: 90% Completed | 146/163 [00:16<00:01, 9.42it/s] Loading safetensors checkpoint shards: 90% Completed | 147/163 [00:16<00:01, 9.32it/s] Loading safetensors checkpoint shards: 91% Completed | 149/163 [00:16<00:01, 9.33it/s] Loading safetensors checkpoint shards: 92% Completed | 150/163 [00:16<00:01, 9.45it/s] Loading safetensors checkpoint shards: 93% Completed | 151/163 [00:16<00:01, 9.49it/s] Loading safetensors checkpoint shards: 93% Completed | 152/163 [00:17<00:01, 9.02it/s] Loading safetensors checkpoint shards: 94% Completed | 154/163 [00:17<00:00, 9.47it/s] Loading safetensors checkpoint shards: 95% Completed | 155/163 [00:17<00:00, 9.37it/s] Loading safetensors checkpoint shards: 96% Completed | 157/163 [00:17<00:00, 9.35it/s] Loading safetensors checkpoint shards: 97% Completed | 158/163 [00:17<00:00, 9.48it/s] Loading safetensors checkpoint shards: 98% Completed | 159/163 [00:17<00:00, 9.53it/s] Loading safetensors checkpoint shards: 99% Completed | 162/163 [00:17<00:00, 13.45it/s] Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:17<00:00, 9.11it/s] CPUInfer[0x60998f7d97b0]: Hello WorkerPool[0x60998f7d97d0] 8 subpools, [numa:threads][0:8] [1:8] [2:8] [3:8] [4:8] [5:8] [6:8] [7:8] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 2, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 3, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 4, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 5, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 6, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 7, 8 threads TP MOE layer 3, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_5" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_2" Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 4 at numa 4 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_3/_numa_4" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_7" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_3" Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_0" Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_6" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_0" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_6""models/DeepSeek-V3.2-INT8/_layer_3/_numa_2" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_3" "models/DeepSeek-V3.2-INT8/_layer_3/_numa_1" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_3/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_3/_numa_5" TP MOE layer 4, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_0" Creating AMX_MOE_TP 4 at numa 4 Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_7" Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 3 at numa 3 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_3""models/DeepSeek-V3.2-INT8/_layer_4/_numa_2" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_6" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_0" Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_4/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_3""models/DeepSeek-V3.2-INT8/_layer_4/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_4/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_4/_numa_6" TP MOE layer 5, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_4" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_5" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_2" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_7" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_0" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_1" Creating AMX_MOE_TP 6 at numa 6 Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_6" Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_3" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_2" "models/DeepSeek-V3.2-INT8/_layer_5/_numa_7"Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_5/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_5/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_5/_numa_5" TP MOE layer 6, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_4" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_6" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_5" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_3" Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_7" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_0" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_2"Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_1" Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_6/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_6/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_6/_numa_5""models/DeepSeek-V3.2-INT8/_layer_6/_numa_6" TP MOE layer 7, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_4" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_7" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_2" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_3" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_0" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_1" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_6" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_5" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_0" Loading from Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_4""models/DeepSeek-V3.2-INT8/_layer_7/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_7/_numa_1""models/DeepSeek-V3.2-INT8/_layer_7/_numa_6""models/DeepSeek-V3.2-INT8/_layer_7/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_7/_numa_3" "models/DeepSeek-V3.2-INT8/_layer_7/_numa_7" TP MOE layer 8, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_4" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_5" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_7" Creating AMX_MOE_TP 2 at numa 2 Loading from Creating AMX_MOE_TP 1 at numa 1 "models/DeepSeek-V3.2-INT8/_layer_8/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_0" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_6" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_8/_numa_4""models/DeepSeek-V3.2-INT8/_layer_8/_numa_7"Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_3"Loading from "models/DeepSeek-V3.2-INT8/_layer_8/_numa_1" TP MOE layer 9, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_2" Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_7" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_3" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_4" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_6" Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_1" Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_0" Loading from Loading from Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_2" "models/DeepSeek-V3.2-INT8/_layer_9/_numa_6"Loading from "models/DeepSeek-V3.2-INT8/_layer_9/_numa_1""models/DeepSeek-V3.2-INT8/_layer_9/_numa_7""models/DeepSeek-V3.2-INT8/_layer_9/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_9/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_9/_numa_3" TP MOE layer 10, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_4" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_3" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_7" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_5" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_2" Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 6 at numa 6 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_1""models/DeepSeek-V3.2-INT8/_layer_10/_numa_6" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_2"Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_10/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_10/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_10/_numa_5""models/DeepSeek-V3.2-INT8/_layer_10/_numa_3""models/DeepSeek-V3.2-INT8/_layer_10/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_10/_numa_4" TP MOE layer 11, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_2" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_1" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_6" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_4" Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_0" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_7" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_3""models/DeepSeek-V3.2-INT8/_layer_11/_numa_1" Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_4"Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_7""models/DeepSeek-V3.2-INT8/_layer_11/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_11/_numa_6" TP MOE layer 12, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_4" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_6" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_2" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_7" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_3" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_0" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_1" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_5" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_5""models/DeepSeek-V3.2-INT8/_layer_12/_numa_6""models/DeepSeek-V3.2-INT8/_layer_12/_numa_1""models/DeepSeek-V3.2-INT8/_layer_12/_numa_7" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_12/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_12/_numa_3" TP MOE layer 13, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_1" Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_7" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_4" Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_0" Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_6" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_2" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_0"Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_1"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_4""models/DeepSeek-V3.2-INT8/_layer_13/_numa_5"Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_7""models/DeepSeek-V3.2-INT8/_layer_13/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_13/_numa_3" "models/DeepSeek-V3.2-INT8/_layer_13/_numa_6" TP MOE layer 14, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_4" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_2" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_6" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_7" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_0" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_5" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_3"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_5""models/DeepSeek-V3.2-INT8/_layer_14/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_14/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_14/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_14/_numa_7" TP MOE layer 15, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_6" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_1" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_5" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_3" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_7" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_0" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_2" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_4" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_15/_numa_2""models/DeepSeek-V3.2-INT8/_layer_15/_numa_4" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_3""models/DeepSeek-V3.2-INT8/_layer_15/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_15/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_15/_numa_5" TP MOE layer 16, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 1 at numa 1 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_16/_numa_1" Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_0" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_2" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 6 at numa 6 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_3" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_16/_numa_4" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_0" Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_16/_numa_5""models/DeepSeek-V3.2-INT8/_layer_16/_numa_3""models/DeepSeek-V3.2-INT8/_layer_16/_numa_7" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_4""models/DeepSeek-V3.2-INT8/_layer_16/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_16/_numa_1" TP MOE layer 17, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_5" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_4" Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_1" Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_6" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_7" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_0" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_3" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_2" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_2""models/DeepSeek-V3.2-INT8/_layer_17/_numa_0"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_3""models/DeepSeek-V3.2-INT8/_layer_17/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_17/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_17/_numa_5" TP MOE layer 18, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_5" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_0" Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 3 at numa 3 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_18/_numa_3" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_2" Creating AMX_MOE_TP 4 at numa 4 Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_1" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_6" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_3"Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_0" Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_2""models/DeepSeek-V3.2-INT8/_layer_18/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_18/_numa_6" Loading from "models/DeepSeek-V3.2-INT8/_layer_18/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_18/_numa_7" TP MOE layer 19, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_7" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_6" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_5" Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_0" Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_2" Creating AMX_MOE_TP 4 at numa 4 Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_3" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_1" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_1" Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_5""models/DeepSeek-V3.2-INT8/_layer_19/_numa_0" "models/DeepSeek-V3.2-INT8/_layer_19/_numa_6""models/DeepSeek-V3.2-INT8/_layer_19/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_4"Loading from "models/DeepSeek-V3.2-INT8/_layer_19/_numa_3""models/DeepSeek-V3.2-INT8/_layer_19/_numa_7" TP MOE layer 20, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_7" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_1" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_6" Creating AMX_MOE_TP 4 at numa 4 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_4" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_0" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_5" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_0"Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_20/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_6" Loading from "models/DeepSeek-V3.2-INT8/_layer_20/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_20/_numa_3" "models/DeepSeek-V3.2-INT8/_layer_20/_numa_1" TP MOE layer 21, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_7" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_2" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_4" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_5" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_0" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_6" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_3" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_1" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_2""models/DeepSeek-V3.2-INT8/_layer_21/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_4""models/DeepSeek-V3.2-INT8/_layer_21/_numa_6" Loading from "models/DeepSeek-V3.2-INT8/_layer_21/_numa_5""models/DeepSeek-V3.2-INT8/_layer_21/_numa_7" TP MOE layer 22, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_7" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_1" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_4" Creating AMX_MOE_TP 6 at numa 6 Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_6" Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_5" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_3" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_2" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_0"Loading from Loading from Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_4"Loading from "models/DeepSeek-V3.2-INT8/_layer_22/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_22/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_22/_numa_2""models/DeepSeek-V3.2-INT8/_layer_22/_numa_1""models/DeepSeek-V3.2-INT8/_layer_22/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_22/_numa_3" TP MOE layer 23, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_4" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_2" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_1" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_7" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_3" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_6" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_0" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_5" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_0" Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_23/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_6""models/DeepSeek-V3.2-INT8/_layer_23/_numa_3"Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_23/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_23/_numa_5" TP MOE layer 24, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_4" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_7" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_6" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_5" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_0" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_2" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_3" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_24/_numa_3"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_2""models/DeepSeek-V3.2-INT8/_layer_24/_numa_4" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_24/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_24/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_24/_numa_5" TP MOE layer 25, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_6" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_2" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_4" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_3" Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_0" Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_5" Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_1" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_25/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_25/_numa_5"Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_7""models/DeepSeek-V3.2-INT8/_layer_25/_numa_1" Loading from "models/DeepSeek-V3.2-INT8/_layer_25/_numa_2" TP MOE layer 26, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_7" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_6" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_3" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_5" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_1" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_2" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_4" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_1" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_4""models/DeepSeek-V3.2-INT8/_layer_26/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_26/_numa_3" "models/DeepSeek-V3.2-INT8/_layer_26/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_26/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_26/_numa_6" TP MOE layer 27, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_2" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_0" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_5" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_1" Creating AMX_MOE_TP 6 at numa 6 Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_6" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_27/_numa_7" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_0" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_1" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_27/_numa_4""models/DeepSeek-V3.2-INT8/_layer_27/_numa_3" "models/DeepSeek-V3.2-INT8/_layer_27/_numa_7""models/DeepSeek-V3.2-INT8/_layer_27/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_27/_numa_5" TP MOE layer 28, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_6" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_4" Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_1" Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_7" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_5" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_2" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_3" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_0" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_1""models/DeepSeek-V3.2-INT8/_layer_28/_numa_2" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_6" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_28/_numa_5""models/DeepSeek-V3.2-INT8/_layer_28/_numa_4""models/DeepSeek-V3.2-INT8/_layer_28/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_28/_numa_3" TP MOE layer 29, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_0" Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 5 at numa 5 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_29/_numa_1" Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_6" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_3" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_4" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_7" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_0""models/DeepSeek-V3.2-INT8/_layer_29/_numa_1"Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_6""models/DeepSeek-V3.2-INT8/_layer_29/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_29/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_29/_numa_3" TP MOE layer 30, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_4" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_1" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_7" Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 0 at numa 0 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_30/_numa_0" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_6" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_3" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_2" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_0" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_4"Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_1" Loading from "models/DeepSeek-V3.2-INT8/_layer_30/_numa_2" "models/DeepSeek-V3.2-INT8/_layer_30/_numa_6" "models/DeepSeek-V3.2-INT8/_layer_30/_numa_7""models/DeepSeek-V3.2-INT8/_layer_30/_numa_3" TP MOE layer 31, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_2" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_6" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_4" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_0" Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 7 at numa 7 Loading from Creating AMX_MOE_TP 3 at numa 3 "models/DeepSeek-V3.2-INT8/_layer_31/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_3" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_0"Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_3"Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_4""models/DeepSeek-V3.2-INT8/_layer_31/_numa_7"Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_31/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_31/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_31/_numa_6" TP MOE layer 32, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_6" Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_5" Creating AMX_MOE_TP 4 at numa 4 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_2" Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_1" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_0" Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_3" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_0" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_6"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_5"Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_2" "models/DeepSeek-V3.2-INT8/_layer_32/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_32/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_32/_numa_1" TP MOE layer 33, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_6" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_4" Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_5" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_3" Creating AMX_MOE_TP 2 at numa 2 Creating AMX_MOE_TP 0 at numa 0 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_2""models/DeepSeek-V3.2-INT8/_layer_33/_numa_0" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_1" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_1"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_6" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_3""models/DeepSeek-V3.2-INT8/_layer_33/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_33/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_33/_numa_7" TP MOE layer 34, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_7" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_5" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_0" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_6" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_2" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_1" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_4" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_0""models/DeepSeek-V3.2-INT8/_layer_34/_numa_7" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_6" Loading from "models/DeepSeek-V3.2-INT8/_layer_34/_numa_3" "models/DeepSeek-V3.2-INT8/_layer_34/_numa_4""models/DeepSeek-V3.2-INT8/_layer_34/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_34/_numa_5" TP MOE layer 35, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_3" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_4" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_5" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_6" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_7" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_1" Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_0" Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_2" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_0" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_2" Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_3"Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_5"Loading from "models/DeepSeek-V3.2-INT8/_layer_35/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_35/_numa_4""models/DeepSeek-V3.2-INT8/_layer_35/_numa_1" "models/DeepSeek-V3.2-INT8/_layer_35/_numa_6" TP MOE layer 36, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_4" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_2" Creating AMX_MOE_TP 3 at numa 3 Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_3" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_6" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_1" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_5" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_0" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_2"Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_5"Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_36/_numa_6"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_36/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_36/_numa_3" TP MOE layer 37, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_4" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_5" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_1" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_2" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_6" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_7" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_3" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_0" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_0""models/DeepSeek-V3.2-INT8/_layer_37/_numa_2""models/DeepSeek-V3.2-INT8/_layer_37/_numa_1" Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_6"Loading from "models/DeepSeek-V3.2-INT8/_layer_37/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_37/_numa_4""models/DeepSeek-V3.2-INT8/_layer_37/_numa_7""models/DeepSeek-V3.2-INT8/_layer_37/_numa_3" TP MOE layer 38, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_3" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_6" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_5" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_2" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_0" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_1" Creating AMX_MOE_TP 7 at numa 7 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_7" Creating AMX_MOE_TP 4 at numa 4 Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_4" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_0""models/DeepSeek-V3.2-INT8/_layer_38/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_1"Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_5" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_38/_numa_7""models/DeepSeek-V3.2-INT8/_layer_38/_numa_2""models/DeepSeek-V3.2-INT8/_layer_38/_numa_6""models/DeepSeek-V3.2-INT8/_layer_38/_numa_3" TP MOE layer 39, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_0" Creating AMX_MOE_TP 4 at numa 4 Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 3 at numa 3 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_4" "models/DeepSeek-V3.2-INT8/_layer_39/_numa_7" Creating AMX_MOE_TP 5 at numa 5 Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_3" Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_5" Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_1" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_2" Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_6" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_1"Loading from Loading from Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_39/_numa_5" Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_6"Loading from "models/DeepSeek-V3.2-INT8/_layer_39/_numa_3""models/DeepSeek-V3.2-INT8/_layer_39/_numa_2""models/DeepSeek-V3.2-INT8/_layer_39/_numa_0" "models/DeepSeek-V3.2-INT8/_layer_39/_numa_4" TP MOE layer 40, pool: 0x60998f7d97d0, expert num: 256, num_experts_per_tok: 8 Creating AMX_MOE_TP 1 at numa 1 Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_1" Creating AMX_MOE_TP 2 at numa 2 Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_2" Creating AMX_MOE_TP 5 at numa 5 Creating AMX_MOE_TP 4 at numa 4 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_5" "models/DeepSeek-V3.2-INT8/_layer_40/_numa_4" Creating AMX_MOE_TP 0 at numa 0 Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_0" Creating AMX_MOE_TP 3 at numa 3 Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_3" Creating AMX_MOE_TP 7 at numa 7 Creating AMX_MOE_TP 6 at numa 6 Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_7" Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_6" TP Load from file models/DeepSeek-V3.2-INT8 Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_0"Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_1" Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_5"Loading from Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_6""models/DeepSeek-V3.2-INT8/_layer_40/_numa_7" "models/DeepSeek-V3.2-INT8/_layer_40/_numa_3""models/DeepSeek-V3.2-INT8/_layer_40/_numa_4" Loading from "models/DeepSeek-V3.2-INT8/_layer_40/_numa_2" ``` 如果切换成FP8原生weight: ``` python -m sglang.launch_server --host 0.0.0.0 --port 8000 --model models/DeepSeek-V3.2 --trust-remote-code --mem-fraction-static 0.9 --chunked-prefill-size 512 --served-model-name DeepSeek-V3.2 --enable-mixed-chunk --kt-weight-path models/DeepSeek-V3.2 --kt-cpuinfer 64 --kt-threadpool-count 8 --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --kt-method FP8 --attention-backend triton --cuda-graph-max-bs 1 --reasoning-parser deepseek-v3 ``` 那么会出现eof错误： ``` [2026-01-15 19:55:31] WARNING server_args.py:1032: DP attention is enabled for DeepSeek NSA. [2026-01-15 19:55:31] WARNING server_args.py:1055: Setting page size to 64 for DeepSeek NSA. [2026-01-15 19:55:31] WARNING server_args.py:1063: Setting KV cache dtype to bfloat16 for DeepSeek NSA. NSA_DUAL_STREAM=True NSA_FUSE_TOPK=True NSA_FLASHMLA_BACKEND_DECODE_COMPUTE_FP8=True NSA_QUANT_K_CACHE_FAST=True NSA_DEQUANT_K_CACHE_FAST=True NSA_ENABLE_MTP_PRECOMPUTE_METADATA=True [2026-01-15 19:55:32] server_args=ServerArgs(model_path='models/DeepSeek-V3.2', tokenizer_path='models/DeepSeek-V3.2', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, limit_mm_data_per_request=None, revision=None, model_impl='auto', host='0.0.0.0', port=8000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='bfloat16', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.9, max_running_requests=None, max_queued_requests=None, max_total_tokens=None, chunked_prefill_size=512, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=64, hybrid_kvcache_ratio=None, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=714365051, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, served_model_name='DeepSeek-V3.2', weight_version='default', chat_template=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='deepseek-v3', tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', load_watch_interval=0.1, prefill_round_robin_balance=False, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', nsa_prefill_backend='flashmla_sparse', nsa_decode_backend='fa3', disable_flashinfer_autotune=False, speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype='float32', mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_lmcache=False, kt_weight_path='models/DeepSeek-V3.2', kt_method='FP8', kt_cpuinfer=64, kt_threadpool_count=8, kt_num_gpu_experts=0, kt_max_deferred_experts_per_token=2, kt_gpu_prefill_token_threshold=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=1, cuda_graph_bs=[1], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 640, 768, 896, 1024, 1152, 1280, 1408, 1536, 1664, 1792, 1920, 2048, 2176, 2304, 2432, 2560, 2688, 2816, 2944, 3072, 3200, 3328, 3456, 3584, 3712, 3840, 3968, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, enable_fused_qk_norm_rope=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None) [2026-01-15 19:55:32] No chat template found, defaulting to 'string' content format [2026-01-15 19:55:38] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2026-01-15 19:55:38] Init torch distributed ends. mem usage=0.00 GB [2026-01-15 19:55:38] Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmasrConfig' from 'transformers' (/home/jovyan/yue/venv_py12/lib/python3.12/site-packages/transformers/__init__.py) [2026-01-15 19:55:39] Load weight begin. avail mem=46.85 GB [2026-01-15 19:55:39] Detected fp8 checkpoint. [FP8SafeTensorLoader] Detected format: deepseek Loading safetensors checkpoint shards: 0% Completed | 0/163 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 2% Completed | 3/163 [00:00<00:06, 26.40it/s] Loading safetensors checkpoint shards: 4% Completed | 6/163 [00:00<00:19, 8.25it/s] Loading safetensors checkpoint shards: 5% Completed | 8/163 [00:00<00:17, 8.73it/s] Loading safetensors checkpoint shards: 7% Completed | 11/163 [00:00<00:12, 11.76it/s] Loading safetensors checkpoint shards: 9% Completed | 15/163 [00:01<00:08, 17.06it/s] Loading safetensors checkpoint shards: 12% Completed | 19/163 [00:01<00:06, 21.15it/s] Loading safetensors checkpoint shards: 13% Completed | 22/163 [00:01<00:10, 13.18it/s] Loading safetensors checkpoint shards: 15% Completed | 25/163 [00:01<00:11, 11.87it/s] Loading safetensors checkpoint shards: 17% Completed | 27/163 [00:02<00:12, 11.18it/s] Loading safetensors checkpoint shards: 18% Completed | 29/163 [00:02<00:12, 10.74it/s] Loading safetensors checkpoint shards: 19% Completed | 31/163 [00:02<00:11, 11.28it/s] Loading safetensors checkpoint shards: 20% Completed | 33/163 [00:02<00:16, 7.94it/s] Loading safetensors checkpoint shards: 22% Completed | 36/163 [00:03<00:13, 9.60it/s] Loading safetensors checkpoint shards: 23% Completed | 38/163 [00:03<00:13, 9.49it/s] Loading safetensors checkpoint shards: 25% Completed | 40/163 [00:03<00:12, 9.64it/s] Loading safetensors checkpoint shards: 26% Completed | 42/163 [00:03<00:12, 9.70it/s] Loading safetensors checkpoint shards: 27% Completed | 44/163 [00:04<00:12, 9.63it/s] Loading safetensors checkpoint shards: 28% Completed | 46/163 [00:04<00:12, 9.39it/s] Loading safetensors checkpoint shards: 29% Completed | 47/163 [00:04<00:12, 9.31it/s] Loading safetensors checkpoint shards: 30% Completed | 49/163 [00:04<00:11, 9.53it/s] Loading safetensors checkpoint shards: 31% Completed | 51/163 [00:04<00:11, 9.66it/s] Loading safetensors checkpoint shards: 33% Completed | 53/163 [00:04<00:10, 10.58it/s] Loading safetensors checkpoint shards: 34% Completed | 55/163 [00:05<00:10, 10.07it/s] Loading safetensors checkpoint shards: 35% Completed | 57/163 [00:05<00:10, 10.08it/s] Loading safetensors checkpoint shards: 36% Completed | 59/163 [00:05<00:10, 10.09it/s] Loading safetensors checkpoint shards: 37% Completed | 61/163 [00:05<00:10, 9.98it/s] Loading safetensors checkpoint shards: 39% Completed | 63/163 [00:05<00:10, 9.75it/s] Loading safetensors checkpoint shards: 40% Completed | 65/163 [00:06<00:10, 9.61it/s] Loading safetensors checkpoint shards: 40% Completed | 66/163 [00:06<00:10, 9.61it/s] Loading safetensors checkpoint shards: 41% Completed | 67/163 [00:06<00:10, 9.30it/s] Loading safetensors checkpoint shards: 42% Completed | 69/163 [00:06<00:09, 9.55it/s] Loading safetensors checkpoint shards: 43% Completed | 70/163 [00:06<00:09, 9.45it/s] Loading safetensors checkpoint shards: 44% Completed | 71/163 [00:07<00:15, 5.90it/s] Loading safetensors checkpoint shards: 45% Completed | 73/163 [00:07<00:11, 7.66it/s] Loading safetensors checkpoint shards: 45% Completed | 74/163 [00:07<00:11, 8.01it/s] Loading safetensors checkpoint shards: 47% Completed | 76/163 [00:07<00:09, 8.99it/s] Loading safetensors checkpoint shards: 48% Completed | 78/163 [00:07<00:09, 9.41it/s] Loading safetensors checkpoint shards: 49% Completed | 80/163 [00:07<00:08, 9.58it/s] Loading safetensors checkpoint shards: 50% Completed | 82/163 [00:08<00:08, 9.46it/s] Loading safetensors checkpoint shards: 51% Completed | 83/163 [00:08<00:08, 9.31it/s] Loading safetensors checkpoint shards: 52% Completed | 84/163 [00:08<00:08, 9.18it/s] Loading safetensors checkpoint shards: 52% Completed | 85/163 [00:08<00:08, 9.35it/s] Loading safetensors checkpoint shards: 53% Completed | 86/163 [00:08<00:08, 9.02it/s] Loading safetensors checkpoint shards: 54% Completed | 88/163 [00:08<00:08, 9.31it/s] Loading safetensors checkpoint shards: 55% Completed | 89/163 [00:08<00:08, 8.95it/s] Loading safetensors checkpoint shards: 56% Completed | 91/163 [00:09<00:07, 9.22it/s] Loading safetensors checkpoint shards: 56% Completed | 92/163 [00:09<00:07, 9.12it/s] Loading safetensors checkpoint shards: 57% Completed | 93/163 [00:09<00:07, 9.26it/s] Loading safetensors checkpoint shards: 58% Completed | 94/163 [00:09<00:07, 9.09it/s] Loading safetensors checkpoint shards: 59% Completed | 96/163 [00:09<00:06, 10.29it/s] Loading safetensors checkpoint shards: 60% Completed | 98/163 [00:09<00:06, 9.55it/s] Loading safetensors checkpoint shards: 61% Completed | 100/163 [00:10<00:06, 9.57it/s] Loading safetensors checkpoint shards: 63% Completed | 102/163 [00:10<00:06, 9.64it/s] Loading safetensors checkpoint shards: 64% Completed | 104/163 [00:10<00:06, 9.46it/s] Loading safetensors checkpoint shards: 64% Completed | 105/163 [00:10<00:06, 9.30it/s] Loading safetensors checkpoint shards: 65% Completed | 106/163 [00:10<00:06, 9.21it/s] Loading safetensors checkpoint shards: 66% Completed | 108/163 [00:10<00:06, 9.16it/s] Loading safetensors checkpoint shards: 67% Completed | 109/163 [00:11<00:05, 9.28it/s] Loading safetensors checkpoint shards: 67% Completed | 110/163 [00:11<00:05, 9.28it/s] Loading safetensors checkpoint shards: 69% Completed | 112/163 [00:11<00:05, 9.25it/s] Loading safetensors checkpoint shards: 69% Completed | 113/163 [00:11<00:05, 9.25it/s] Loading safetensors checkpoint shards: 70% Completed | 114/163 [00:11<00:05, 9.21it/s] Loading safetensors checkpoint shards: 71% Completed | 116/163 [00:11<00:05, 9.30it/s] Loading safetensors checkpoint shards: 72% Completed | 118/163 [00:11<00:04, 10.41it/s] Loading safetensors checkpoint shards: 74% Completed | 120/163 [00:12<00:04, 9.87it/s] Loading safetensors checkpoint shards: 74% Completed | 121/163 [00:12<00:06, 6.01it/s] Loading safetensors checkpoint shards: 76% Completed | 124/163 [00:12<00:04, 8.11it/s] Loading safetensors checkpoint shards: 77% Completed | 125/163 [00:12<00:04, 8.13it/s] Loading safetensors checkpoint shards: 78% Completed | 127/163 [00:13<00:04, 8.67it/s] Loading safetensors checkpoint shards: 79% Completed | 128/163 [00:13<00:03, 8.80it/s] Loading safetensors checkpoint shards: 80% Completed | 130/163 [00:13<00:03, 9.00it/s] Loading safetensors checkpoint shards: 80% Completed | 131/163 [00:13<00:03, 9.16it/s] Loading safetensors checkpoint shards: 81% Completed | 132/163 [00:13<00:03, 9.30it/s] Loading safetensors checkpoint shards: 82% Completed | 133/163 [00:13<00:03, 9.09it/s] Loading safetensors checkpoint shards: 83% Completed | 135/163 [00:13<00:02, 9.40it/s] Loading safetensors checkpoint shards: 83% Completed | 136/163 [00:14<00:02, 9.30it/s] Loading safetensors checkpoint shards: 85% Completed | 138/163 [00:14<00:02, 9.29it/s] Loading safetensors checkpoint shards: 86% Completed | 140/163 [00:14<00:02, 9.58it/s] Loading safetensors checkpoint shards: 87% Completed | 141/163 [00:14<00:02, 9.47it/s] Loading safetensors checkpoint shards: 87% Completed | 142/163 [00:14<00:02, 9.57it/s] Loading safetensors checkpoint shards: 88% Completed | 143/163 [00:14<00:02, 9.48it/s] Loading safetensors checkpoint shards: 88% Completed | 144/163 [00:14<00:02, 9.16it/s] Loading safetensors checkpoint shards: 90% Completed | 146/163 [00:15<00:01, 9.37it/s] Loading safetensors checkpoint shards: 90% Completed | 147/163 [00:15<00:01, 9.28it/s] Loading safetensors checkpoint shards: 91% Completed | 149/163 [00:15<00:01, 9.27it/s] Loading safetensors checkpoint shards: 92% Completed | 150/163 [00:15<00:01, 9.27it/s] Loading safetensors checkpoint shards: 93% Completed | 151/163 [00:15<00:01, 9.27it/s] Loading safetensors checkpoint shards: 93% Completed | 152/163 [00:15<00:01, 8.84it/s] Loading safetensors checkpoint shards: 94% Completed | 154/163 [00:16<00:00, 9.25it/s] Loading safetensors checkpoint shards: 95% Completed | 155/163 [00:16<00:00, 9.12it/s] Loading safetensors checkpoint shards: 96% Completed | 157/163 [00:16<00:00, 9.12it/s] Loading safetensors checkpoint shards: 97% Completed | 158/163 [00:16<00:00, 9.24it/s] Loading safetensors checkpoint shards: 98% Completed | 159/163 [00:16<00:00, 9.30it/s] Loading safetensors checkpoint shards: 99% Completed | 162/163 [00:16<00:00, 13.14it/s] Loading safetensors checkpoint shards: 100% Completed | 163/163 [00:16<00:00, 9.75it/s] CPUInfer[0x5dd5a207d430]: Hello WorkerPool[0x5dd5a207d450] 8 subpools, [numa:threads][0:8] [1:8] [2:8] [3:8] [4:8] [5:8] [6:8] [7:8] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 2, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 3, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 4, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 5, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 6, 8 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 7, 8 threads TP MOE layer 3, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 4, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 5, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 6, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 3 at numa 3 TP MOE layer 7, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 8, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 5 at numa 5 TP MOE layer 9, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 2 at numa 2 TP MOE layer 10, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 1 at numa 1 TP MOE layer 11, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 12, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 6 at numa 6 TP MOE layer 13, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 14, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 15, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 6 at numa 6 TP MOE layer 16, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 17, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 3 at numa 3 TP MOE layer 18, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 19, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 20, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 21, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 22, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 23, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 24, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 6 at numa 6 TP MOE layer 25, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 5 at numa 5 TP MOE layer 26, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 1 at numa 1 TP MOE layer 27, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 0 at numa 0 TP MOE layer 28, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 29, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 4 at numa 4 TP MOE layer 30, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 6 at numa 6 TP MOE layer 31, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 1 at numa 1 TP MOE layer 32, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 7 at numa 7 TP MOE layer 33, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 1 at numa 1 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 0 at numa 0 TP MOE layer 34, pool: 0x5dd5a207d450, expert num: 256, num_experts_per_tok: 8 Created AMX_FP8_MOE_TP 3 at numa 3 Created AMX_FP8_MOE_TP 0 at numa 0 Created AMX_FP8_MOE_TP 7 at numa 7 Created AMX_FP8_MOE_TP 2 at numa 2 Created AMX_FP8_MOE_TP 5 at numa 5 Created AMX_FP8_MOE_TP 4 at numa 4 Created AMX_FP8_MOE_TP 6 at numa 6 Created AMX_FP8_MOE_TP 1 at numa 1 TP MOE layer 35, pool: 0x5dd5a207d450, expert num: 256, num_exper[2026-01-15 19:57:43] Rank 0 scheduler is dead. Please check if there are relevant logs. [2026-01-15 19:57:43] Exit code: -9 Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/jovyan/yue/sglang/python/sglang/launch_server.py", line 32, in <module> run_server(server_args) File "/home/jovyan/yue/sglang/python/sglang/launch_server.py", line 25, in run_server launch_server(server_args) File "/home/jovyan/yue/sglang/python/sglang/srt/entrypoints/http_server.py", line 1692, in launch_server _launch_subprocesses( File "/home/jovyan/yue/sglang/python/sglang/srt/entrypoints/engine.py", line 966, in _launch_subprocesses data = scheduler_pipe_readers[i].recv() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() ^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/multiprocessing/connection.py", line 430, in _recv_bytes buf = self._recv(4) ^^^^^^^^^^^^^ File "/opt/conda/lib/python3.12/multiprocessing/connection.py", line 399, in _recv raise EOFError EOFError ```
bsakaguchi commented 2026-10-29 14:48:57 +00:00
Author
Owner
Copy link

@CodeZ-Hao commented on GitHub (Jan 16, 2026):

根据你上面的信息，大概率是numa的问题了，你的CPU是双路AMD，numa数达到24，说明ccd是相互独立没有合并的，需要设置NPS=1，这样应该会变成2numa，不过你的配置估计4numa也能跑起来。
但是我没有用过AMD的平台，所以具体设置方法不太清楚，你可以参考下这篇博文 https://blog.csdn.net/guganly/article/details/149828237

<!-- gh-comment-id:3757757998 --> @CodeZ-Hao commented on GitHub (Jan 16, 2026): 根据你上面的信息，大概率是numa的问题了，你的CPU是双路AMD，numa数达到24，说明ccd是相互独立没有合并的，需要设置NPS=1，这样应该会变成2numa，不过你的配置估计4numa也能跑起来。 但是我没有用过AMD的平台，所以具体设置方法不太清楚，你可以参考下这篇博文 https://blog.csdn.net/guganly/article/details/149828237
bsakaguchi commented 2026-10-29 14:48:57 +00:00
Author
Owner
Copy link

@SchenbergZY commented on GitHub (Jan 16, 2026):

根据你上面的信息，大概率是numa的问题了，你的CPU是双路AMD，numa数达到24，说明ccd是相互独立没有合并的，需要设置NPS=1，这样应该会变成2numa，不过你的配置估计4numa也能跑起来。 但是我没有用过AMD的平台，所以具体设置方法不太清楚，你可以参考下这篇博文 https://blog.csdn.net/guganly/article/details/149828237

好的谢谢。因为我只能访问这个jupyterlab container，所以无法直接改NPS到1，正在尝试和服务器管理员联系看他能否改。另外我的kt-cpuinfer-threads和threadpool-count在NPS=1后该如何修改?

<!-- gh-comment-id:3760793786 --> @SchenbergZY commented on GitHub (Jan 16, 2026): > 根据你上面的信息，大概率是numa的问题了，你的CPU是双路AMD，numa数达到24，说明ccd是相互独立没有合并的，需要设置NPS=1，这样应该会变成2numa，不过你的配置估计4numa也能跑起来。 但是我没有用过AMD的平台，所以具体设置方法不太清楚，你可以参考下这篇博文 https://blog.csdn.net/guganly/article/details/149828237 好的谢谢。因为我只能访问这个jupyterlab container，所以无法直接改NPS到1，正在尝试和服务器管理员联系看他能否改。另外我的kt-cpuinfer-threads和threadpool-count在NPS=1后该如何修改?
bsakaguchi commented 2026-10-29 14:48:57 +00:00
Author
Owner
Copy link

@CodeZ-Hao commented on GitHub (Jan 16, 2026):

根据你上面的信息，大概率是numa的问题了，你的CPU是双路AMD，numa数达到24，说明ccd是相互独立没有合并的，需要设置NPS=1，这样应该会变成2numa，不过你的配置估计4numa也能跑起来。 但是我没有用过AMD的平台，所以具体设置方法不太清楚，你可以参考下这篇博文 https://blog.csdn.net/guganly/article/details/149828237

好的谢谢。因为我只能访问这个jupyterlab container，所以无法直接改NPS到1，正在尝试和服务器管理员联系看他能否改。另外我的kt-cpuinfer-threads和threadpool-count在NPS=1后该如何修改?

  • kt-cpuinfer-threads=172 （按我的经验，物理核心*0.9，这个值只影响推理速率，跑起来后可以自行调整测试找到最优值）
  • threadpool-count=numa数量 （lscpu | grep "NUMA node(s)" 查看，这里是多少就填多少，NPS=1时这个值应该是2）
<!-- gh-comment-id:3761156927 --> @CodeZ-Hao commented on GitHub (Jan 16, 2026): > > 根据你上面的信息，大概率是numa的问题了，你的CPU是双路AMD，numa数达到24，说明ccd是相互独立没有合并的，需要设置NPS=1，这样应该会变成2numa，不过你的配置估计4numa也能跑起来。 但是我没有用过AMD的平台，所以具体设置方法不太清楚，你可以参考下这篇博文 https://blog.csdn.net/guganly/article/details/149828237 > > 好的谢谢。因为我只能访问这个jupyterlab container，所以无法直接改NPS到1，正在尝试和服务器管理员联系看他能否改。另外我的kt-cpuinfer-threads和threadpool-count在NPS=1后该如何修改? - kt-cpuinfer-threads=172 （按我的经验，物理核心*0.9，这个值只影响推理速率，跑起来后可以自行调整测试找到最优值） - threadpool-count=numa数量 （lscpu | grep "NUMA node(s)" 查看，这里是多少就填多少，NPS=1时这个值应该是2）
bsakaguchi commented 2026-10-29 14:48:58 +00:00
Author
Owner
Copy link

@SchenbergZY commented on GitHub (Jan 19, 2026):

在等待我的管理员回复的时候，我有另一个疑问：

我使用之前的配置（NUMA=8）测试在--kt-weight-path 设置为空文件夹的情况下，INT8能跑起来 (虽然输出结果肯定是循环字符了）：

python -m sglang.launch_server   --host 0.0.0.0   --port 8000   --model models/DeepSeek-V3.2   --trust-remote-code   --mem-fraction-static 0.9   --chunked-prefill-size 512   --served-model-name DeepSeek-V3.2   --enable-mixed-chunk   --kt-weight-path models/DeepSeek-V3.2-INT8-N2-Empty   --kt-cpuinfer 64  --kt-threadpool-count 8 --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --kt-method AMXINT8 --attention-backend triton --cuda-graph-max-bs 1 --reasoning-parser deepseek-v3

那么我有个问题：我之前59层layer能加载40层，那如果这个加载量化weights的过程是加载一层忘记一层的话（至少不要全部加载了再全部weight update的话）是不是就不会卡死了？

<!-- gh-comment-id:3768681022 --> @SchenbergZY commented on GitHub (Jan 19, 2026): 在等待我的管理员回复的时候，我有另一个疑问： 我使用之前的配置（NUMA=8）测试在--kt-weight-path 设置为空文件夹的情况下，INT8能跑起来 (虽然输出结果肯定是循环字符了）： ``` python -m sglang.launch_server --host 0.0.0.0 --port 8000 --model models/DeepSeek-V3.2 --trust-remote-code --mem-fraction-static 0.9 --chunked-prefill-size 512 --served-model-name DeepSeek-V3.2 --enable-mixed-chunk --kt-weight-path models/DeepSeek-V3.2-INT8-N2-Empty --kt-cpuinfer 64 --kt-threadpool-count 8 --kt-num-gpu-experts 0 --disable-shared-experts-fusion --kt-max-deferred-experts-per-token 2 --tensor-parallel-size 1 --enable-p2p-check --kt-method AMXINT8 --attention-backend triton --cuda-graph-max-bs 1 --reasoning-parser deepseek-v3 ``` 那么我有个问题：我之前59层layer能加载40层，那如果这个加载量化weights的过程是加载一层忘记一层的话（至少不要全部加载了再全部weight update的话）是不是就不会卡死了？
bsakaguchi commented 2026-10-29 14:48:58 +00:00
Author
Owner
Copy link

@CodeZ-Hao commented on GitHub (Jan 21, 2026):

应该是不行的，这个权重是要加载在内存中持续运行的，并不是加载完就可以卸载了，你的说法更像是逐层量化，但是kt的权重加载不是量化过程，量化倒是确实如你所说，是逐层加载量化的

<!-- gh-comment-id:3775707401 --> @CodeZ-Hao commented on GitHub (Jan 21, 2026): 应该是不行的，这个权重是要加载在内存中持续运行的，并不是加载完就可以卸载了，你的说法更像是逐层量化，但是kt的权重加载不是量化过程，量化倒是确实如你所说，是逐层加载量化的
bsakaguchi commented 2026-10-29 14:48:58 +00:00
Author
Owner
Copy link

@SchenbergZY commented on GitHub (Jan 21, 2026):

好的那就没办法了，谢谢

<!-- gh-comment-id:3775846244 --> @SchenbergZY commented on GitHub (Jan 21, 2026): 好的那就没办法了，谢谢
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1129
No description provided.