kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #2128] AMXInt4 not supported in 2 numa machine with "Segmentation fault" #1286

New issue
Closed
opened 2026-10-29 14:50:51 +00:00 by bsakaguchi · 1 comment
bsakaguchi commented 2026-10-29 14:50:51 +00:00
Owner
Copy link

Originally created by @CodeZ-Hao on GitHub (Aug 4, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2128

Reminder

  • I have read the above rules and searched the existing issues.

System Info

安装时使用USE_NUMA=1 bash install.sh安装，

KTransformers 环境诊断
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Check ┃ Status ┃ Value ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Python 版本 │ 正常 │ 3.11.14 │
│ CUDA 可用性 │ 正常 │ 13.0 │
│ GPU 检测 │ 正常 │ 发现 1 个 GPU: NVIDIA GeForce RTX 4080 │
│ CPU │ 正常 │ Intel(R) Xeon(R) Platinum 8468 (96 核心 / 192 线程) │
│ CPU 指令集 │ 正常 │ AMX-INT8, AMX-BF16, AMX-TILE, AVX512BF16, AVX512F, AVX512BW, AVX512VL, AVX2 (+14 more) │
│ NUMA 拓扑 │ 正常 │ 2 个节点 │
│ kt-kernel │ 正常 │ v0.6.4 (AMX) │
│ 系统内存 │ 正常 │ 928.2GB 可用 / 944.5GB 总计 │
│ 磁盘空间 │ 警告 │ /models 有 38.7GB 可用空间 │
│ SGLang Source │ 正常 │ sglang-kt │
│ SGLang kt-kernel │ 正常 │ SGLang kt-kernel 支持已验证 │
│ Env: SGLANG_DSV4_2604_SUBMODE │ 警告 │ 2604B │
│ Environment Managers │ 正常 │ conda 26.3.2, mamba 2.4.0, venv builtin, docker 28.3.3 │
└───────────────────────────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────┘
✓ 所有检查通过！您的环境已就绪。

pip list
Package Version Editable project location

addict 2.4.0
aiohappyeyeballs 2.6.1
aiohttp 3.13.2
aiosignal 1.4.0
airportsdata 20250909
altair 6.1.0
annotated-doc 0.0.4
annotated-types 0.7.0
anthropic 0.75.0
anyio 4.12.0
apache-tvm-ffi 0.1.11
aria2 0.0.1b0
asttokens 3.0.1
attrs 25.4.0
av 16.1.0
black 25.12.0
blinker 1.9.0
blobfile 3.0.0
build 1.3.0
cache_dit 1.2.0
cachetools 7.1.2
certifi 2025.11.12
cffi 2.0.0
charset-normalizer 3.4.4
click 8.4.2
cloudpickle 3.1.2
compressed-tensors 0.12.2
cuda-bindings 13.2.0
cuda-pathfinder 1.3.3
cuda-python 13.2.0
cuda-tile 1.3.0
datasets 4.4.1
decorator 5.2.1
decord2 2.0.0
diffusers 0.36.0
dill 0.4.0
diskcache 5.6.3
distro 1.9.0
docstring_parser 0.17.0
einops 0.8.1
executing 2.2.1
fastapi 0.124.4
filelock 3.20.0
flashinfer-cubin 0.6.3
flashinfer-python 0.6.3
frozenlist 1.8.0
fsspec 2025.10.0
ftllm 0.1.7.1
gguf 0.17.1
gitdb 4.0.12
GitPython 3.1.50
googleapis-common-protos 1.72.0
grpcio 1.83.0
grpcio-health-checking 1.81.1
grpcio-reflection 1.81.1
grpcio-tools 1.75.1
h11 0.16.0
hf_transfer 0.1.9
hf-xet 1.6.0
httpcore 1.0.9
httptools 0.7.1
httpx 0.28.1
huggingface_hub 1.26.0
humanize 4.15.0
idna 3.11
imageio 2.36.0
imageio-ffmpeg 0.5.1
importlib_metadata 8.7.1
interegular 0.3.3
ipython 9.8.0
ipython_pygments_lexers 1.1.1
itsdangerous 2.2.0
jedi 0.19.2
Jinja2 3.1.6
jiter 0.12.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
kt-kernel 0.6.4
lark 1.3.1
llguidance 0.7.30
loguru 0.7.3
lxml 6.0.2
markdown-it-py 4.0.0
MarkupSafe 3.0.3
matplotlib-inline 0.2.1
mdurl 0.1.2
ml_dtypes 0.5.4
modelscope 1.33.0
moviepy 2.2.1
mpmath 1.3.0
msgspec 0.20.0
multidict 6.7.0
multiprocess 0.70.18
mypy_extensions 1.1.0
narwhals 2.21.2
nest-asyncio 1.6.0
networkx 3.6.1
ninja 1.13.0
numpy 2.3.5
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-crt 13.3.73
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvcc 13.2.78
nvidia-cuda-nvdisasm 13.3.73
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime 13.3.29
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cuda-tileiras 13.2.78
nvidia-cudnn-cu12 9.10.2.21
nvidia-cudnn-frontend 1.16.0
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-cutlass-dsl 4.3.5
nvidia-cutlass-dsl-libs-base 4.6.0
nvidia-cutlass-dsl-libs-core 4.6.0
nvidia-cutlass-dsl-libs-cu12 4.6.0
nvidia-ml-py 13.590.44
nvidia-nccl-cu12 2.27.5
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvshmem-cu12 3.3.20
nvidia-nvtx-cu12 12.8.90
nvidia-nvvm 13.2.78
openai 2.6.1
openai-harmony 0.0.4
opencv-python-headless 4.10.0.84
opentelemetry-api 1.39.1
opentelemetry-exporter-otlp 1.39.1
opentelemetry-exporter-otlp-proto-common 1.39.1
opentelemetry-exporter-otlp-proto-grpc 1.39.1
opentelemetry-exporter-otlp-proto-http 1.39.1
opentelemetry-proto 1.39.1
opentelemetry-sdk 1.39.1
opentelemetry-semantic-conventions 0.60b1
orjson 3.11.5
outlines 0.1.11
outlines_core 0.1.26
packaging 25.0
pandas 2.3.3
parso 0.8.5
partial-json-parser 0.2.1.1.post7
pathspec 0.12.1
pexpect 4.9.0
pillow 11.3.0
pip 25.3
platformdirs 4.5.1
proglog 0.1.12
prometheus_client 0.23.1
prompt_toolkit 3.0.52
propcache 0.4.1
protobuf 6.33.6
psutil 7.1.3
ptyprocess 0.7.0
pure_eval 0.2.3
py-spy 0.4.1
pyarrow 22.0.0
pybase64 1.4.3
pycountry 24.6.1
pycparser 2.23
pycryptodomex 3.23.0
pydantic 2.12.5
pydantic_core 2.41.5
pydeck 0.9.2
Pygments 2.19.2
pyproject_hooks 1.2.0
pyreadline3 3.5.6
python-dateutil 2.9.0.post0
python-dotenv 1.2.1
python-multipart 0.0.20
pytokens 0.3.0
pytz 2025.2
PyYAML 6.0.1
pyzmq 27.1.0
quack-kernels 0.2.4
referencing 0.37.0
regex 2025.11.3
remote-pdb 2.1.0
requests 2.32.5
rich 14.3.3
rpds-py 0.30.0
runai-model-streamer 0.15.6
safetensors 0.8.0
scipy 1.16.3
sentencepiece 0.2.1
setproctitle 1.3.7
setuptools 80.9.0
sgl-kernel 0.3.21
sglang 0.0.0.dev9743+ga864420ee /home/ktransformers/sglang/python
sglang-kt 0.0.0.dev0 /home/ktransformers/sglang/python
sglang-router 0.3.2
shellingham 1.5.4
shortuuid 1.0.13
six 1.17.0
smg-grpc-proto 0.4.14
smmap 5.0.3
sniffio 1.3.1
soundfile 0.13.1
st-attn 0.0.7
stack-data 0.6.3
starlette 0.50.0
streamlit 1.57.0
streamlit-chat 0.1.1
sympy 1.14.0
tabulate 0.9.0
tenacity 9.1.4
tiktoken 0.12.0
tilelang 0.1.12
timm 1.0.16
tokenizers 0.22.1
toml 0.10.2
torch 2.9.1
torch_c_dlpack_ext 0.1.5
torch_memory_saver 0.0.9
torchao 0.9.0
torchaudio 2.9.1
torchcodec 0.8.0
torchvision 0.24.1
tqdm 4.67.1
traitlets 5.14.3
transformers 5.14.1
transformers-kt 5.6.0.post1
triton 3.5.1
typer 0.24.1
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.3
urllib3 2.6.2
uvicorn 0.38.0
uvloop 0.22.1
vsa 0.0.4
watchdog 6.0.0
wcwidth 0.2.14
websockets 16.0
wheel 0.45.1
xgrammar 0.1.27
xxhash 3.6.0
yarl 1.22.0
z3-solver 4.15.4.0
zipp 3.23.0

Reproduction

使用--kt-cpuinfer 40 --kt-threadpool-count 1 参数可以正常运行，
执行命令以及异常打印如下：

python -m sglang.launch_server --host 0.0.0.0 --port 6398 --model /models/deepseek-ai/DeepSeek-V4-Flash-0731 --kt-weight-path /models/deepseek-ai/DeepSeek-V4-Flash-0731-AMXINT4/ --kt-method AMXINT4 --kt-num-gpu-experts 0 --kt-cpuinfer 80 --kt-threadpool-count 2 --context-length 400000 --max-total-tokens 500000 --attention-backend flashinfer --mem-fraction-static 0.9 --chunked-prefill-size 2048 --max-running-requests 4 --trust-remote-code --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --skip-server-warmup --served-model-name deepseek-v4-flash
[2026-08-04 13:49:39] WARNING hf_transformers_utils.py:236: SGLANG_APPLY_CONFIG_BACKUP=auto: checkpoint has num_hidden_layers=43, dispatching to 'small'.
[2026-08-04 13:49:39] WARNING hf_transformers_utils.py:255: SGLANG_APPLY_CONFIG_BACKUP=small: using packaged /root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/configs/config_backup_small.json instead of the checkpoint's config.json at /models/deepseek-ai/DeepSeek-V4-Flash-0731.
[2026-08-04 13:49:39] INFO _model_config_dsv4.py:117: Auto-detected routed-expert safetensors dtype=I8; SGLANG_DSV4_FP4_EXPERTS=True
[2026-08-04 13:49:39] INFO model_config.py:419: Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-08-04 13:49:39] WARNING model_config.py:1151: Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround.
[2026-08-04 13:49:39] INFO server_args.py:1407: Use compressed attention backend for DeepseekV4ForCausalLM, setting page_size to 256.
[2026-08-04 13:49:39] WARNING server_args.py:1422: Setting KV cache dtype to fp8_e4m3 for DeepseekV4ForCausalLM.
[2026-08-04 13:49:39] INFO server_args.py:1450: Setting swa_full_tokens_ratio to 0.1 for DeepseekV4ForCausalLM.
[2026-08-04 13:49:39] WARNING server_args.py:2627: KTransformers EP is enabled. --disable-shared-experts-fusion is automatically set to prevent shared experts from being offloaded to CPU.
[transformers] `BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead.
[2026-08-04 13:49:40] server_args=ServerArgs(model_path='/models/deepseek-ai/DeepSeek-V4-Flash-0731', tokenizer_path='/models/deepseek-ai/DeepSeek-V4-Flash-0731', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=400000, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=6398, fastapi_root_path='', grpc_mode=False, skip_server_warmup=True, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='fp8_e4m3', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.9, max_running_requests=4, max_queued_requests=None, max_total_tokens=500000, chunked_prefill_size=2048, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=256, swa_full_tokens_ratio=0.1, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=461787985, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='deepseek-v4-flash', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='deepseek-v4', tool_call_parser='deepseekv4', tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='compressed', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='flashinfer_cutlass', nsa_prefill_backend=None, nsa_decode_backend=None, disable_flashinfer_autotune=False, mamba_backend='triton', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, enable_aiter_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype=None, mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path='/models/deepseek-ai/DeepSeek-V4-Flash-0731-AMXINT4/', kt_method='AMXINT4', kt_cpuinfer=80, kt_threadpool_count=2, kt_numa_nodes=None, kt_num_gpu_experts=0, kt_gpu_experts_ratio=None, kt_max_deferred_experts_per_token=None, kt_gpu_prefill_token_threshold=None, record_kt_gpu_expert_distribution=False, kt_enable_dynamic_expert_update=False, kt_expert_placement_strategy='uniform', kt_lora_path=None, kt_expert_lora_path=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=24, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=2048, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='round-robin-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, enable_mm_global_cache=False, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-08-04 13:49:40] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-08-04 13:49:40] Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround.
[2026-08-04 13:49:41] No HuggingFace chat template found
[2026-08-04 13:49:41] No chat template found, defaulting to 'string' content format
[transformers] `BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead.
[2026-08-04 13:49:45] SGLANG_APPLY_CONFIG_BACKUP=auto: checkpoint has num_hidden_layers=43, dispatching to 'small'.
[2026-08-04 13:49:45] SGLANG_APPLY_CONFIG_BACKUP=small: using packaged /root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/configs/config_backup_small.json instead of the checkpoint's config.json at /models/deepseek-ai/DeepSeek-V4-Flash-0731.
[2026-08-04 13:49:45] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-08-04 13:49:45] Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround.
[transformers] `BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead.
[2026-08-04 13:49:46] Mamba selective_state_update backend initialized: triton
[2026-08-04 13:49:46] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM']
[2026-08-04 13:49:46] Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround.
[2026-08-04 13:49:46] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2026-08-04 13:49:46] Init torch distributed ends. elapsed=0.18 s, mem usage=0.04 GB
[2026-08-04 13:49:47] Load weight begin. avail mem=31.14 GB
[2026-08-04 13:49:47] Detected fp8 checkpoint.
[2026-08-04 13:49:47] Using kt_num_gpu_experts=0 per layer, total GPU experts: 0 (= 0 × 43 MoE layers)
[2026-08-04 13:49:47] Using uniform strategy for GPU expert placement
[2026-08-04 13:49:47] KT GPU experts: layer 0 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 1 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 2 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 3 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 4 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 5 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 6 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 7 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 8 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 9 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 10 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 11 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 12 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 13 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 14 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 15 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 16 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 17 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 18 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 19 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 20 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 21 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 22 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 23 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 24 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 25 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 26 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 27 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 28 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 29 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 30 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 31 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 32 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 33 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 34 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 35 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 36 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 37 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 38 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 39 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 40 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 41 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] KT GPU experts: layer 42 (MoE) has 0 GPU experts
[2026-08-04 13:49:47] Generated KT GPU experts masks using 'uniform' strategy: 43 MoE layers (out of 43 total layers) x 256 experts, total GPU experts in MoE layers = 0
[2026-08-04 13:49:47] [KT] Created shared staging buffer: 16.0 MiB (shape=torch.Size([2048, 4096]), dtype=torch.bfloat16)
CPUInfer[0x5f0f3da117f0]: Hello
WorkerPool[0x5f0f3da143c0] 2 subpools, [numa:threads][0:40] [1:40] 
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 40 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 40 threads
Loading safetensors checkpoint shards:   0% Completed | 0/48 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  10% Completed | 5/48 [00:00<00:00, 48.82it/s]
Loading safetensors checkpoint shards:  21% Completed | 10/48 [00:00<00:00, 48.23it/s]
Loading safetensors checkpoint shards:  31% Completed | 15/48 [00:00<00:01, 21.88it/s]
Loading safetensors checkpoint shards:  42% Completed | 20/48 [00:00<00:01, 27.38it/s]
Loading safetensors checkpoint shards:  52% Completed | 25/48 [00:00<00:00, 31.65it/s]
Loading safetensors checkpoint shards:  62% Completed | 30/48 [00:00<00:00, 34.85it/s]
Loading safetensors checkpoint shards:  73% Completed | 35/48 [00:01<00:00, 37.56it/s]
Loading safetensors checkpoint shards:  85% Completed | 41/48 [00:01<00:00, 41.85it/s]
Loading safetensors checkpoint shards:  96% Completed | 46/48 [00:01<00:00, 42.26it/s]
Loading safetensors checkpoint shards: 100% Completed | 48/48 [00:01<00:00, 37.17it/s]
TP MOE layer 0, pool: 0x5f0f3da143c0, expert num: 256, num_experts_per_tok: 6
Creating AMX_MOE_TP 1 at numa 1
Creating AMX_MOE_TP 0 at numa 0
TP Load from loader
Fatal Python error: Segmentation fault
Thread 0x0000755d71fff640 (most recent call first):
  File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 331 in wait
  File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 629 in wait
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
  File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 1002 in _bootstrap
Thread 0x0000758c8a9ec740 (most recent call first):
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 537 in load_weights
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 3684 in process_weights_after_loading
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 697 in load_weights_and_postprocess
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 677 in load_model
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 1060 in load_model
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 501 in initialize
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 418 in __init__
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 330 in _init_model_runner
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 247 in __init__
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 521 in init_tp_model_worker
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 563 in init_model_worker
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 367 in __init__
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3207 in run_scheduler_process
  File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/process.py", line 108 in run
  File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap
  File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/spawn.py", line 135 in _main
  File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>
Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pybase64._pybase64, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, regex._regex, yaml._yaml, markupsafe._speedups, cuda_utils, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, av.option, av.descriptor, av.format, av.utils, av.stream, av.container.streams, av.sidedata.encparams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.filter.loudnorm, av.audio.resampler, av.audio.codeccontext, av.audio.fifo, av.bitstream, av.video.codeccontext, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, msgspec._core, grpc._cython.cygrpc, google._upb._message, tilelang_cython_wrapper (total: 158)
!!!!!!! Segfault encountered !!!!!!!
  File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000758c8a64251f
  File "../../../../../libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine
  File "./nptl/pthread_create.c", line 442, in start_thread
  File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3
  File "<unknown>", line 0, in 0xffffffffffffffff
!!!!!!! Segfault encountered !!!!!!!
  File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000758c8a64251f
  File "../../../../../libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine
  File "./nptl/pthread_create.c", line 442, in start_thread
  File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3
  File "<unknown>", line 0, in 0xffffffffffffffff
[2026-08-04 13:50:15] Rank 0 scheduler is dead. Please check if there are relevant logs.
[2026-08-04 13:50:16] Exit code: -11
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/launch_server.py", line 112, in <module>
    run_server(server_args)
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/launch_server.py", line 105, in run_server
    launch_server(server_args)
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/entrypoints/http_server.py", line 1960, in launch_server
    _launch_subprocesses(
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/entrypoints/engine.py", line 1115, in _launch_subprocesses
    scheduler_infos = _wait_for_scheduler_ready(scheduler_pipe_readers, scheduler_procs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/entrypoints/engine.py", line 925, in _wait_for_scheduler_ready
    data = scheduler_pipe_readers[i].recv()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/connection.py", line 399, in _recv
    raise EOFError
EOFError

Others

No response

Originally created by @CodeZ-Hao on GitHub (Aug 4, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2128 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info 安装时使用USE_NUMA=1 bash install.sh安装， KTransformers 环境诊断 ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Check ┃ Status ┃ Value ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩ │ Python 版本 │ 正常 │ 3.11.14 │ │ CUDA 可用性 │ 正常 │ 13.0 │ │ GPU 检测 │ 正常 │ 发现 1 个 GPU: NVIDIA GeForce RTX 4080 │ │ CPU │ 正常 │ Intel(R) Xeon(R) Platinum 8468 (96 核心 / 192 线程) │ │ CPU 指令集 │ 正常 │ AMX-INT8, AMX-BF16, AMX-TILE, AVX512BF16, AVX512F, AVX512BW, AVX512VL, AVX2 (+14 more) │ │ NUMA 拓扑 │ 正常 │ 2 个节点 │ │ kt-kernel │ 正常 │ v0.6.4 (AMX) │ │ 系统内存 │ 正常 │ 928.2GB 可用 / 944.5GB 总计 │ │ 磁盘空间 │ 警告 │ /models 有 38.7GB 可用空间 │ │ SGLang Source │ 正常 │ sglang-kt │ │ SGLang kt-kernel │ 正常 │ SGLang kt-kernel 支持已验证 │ │ Env: SGLANG_DSV4_2604_SUBMODE │ 警告 │ 2604B │ │ Environment Managers │ 正常 │ conda 26.3.2, mamba 2.4.0, venv builtin, docker 28.3.3 │ └───────────────────────────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────┘ ✓ 所有检查通过！您的环境已就绪。 pip list Package Version Editable project location ---------------------------------------- ------------------------ --------------------------------- addict 2.4.0 aiohappyeyeballs 2.6.1 aiohttp 3.13.2 aiosignal 1.4.0 airportsdata 20250909 altair 6.1.0 annotated-doc 0.0.4 annotated-types 0.7.0 anthropic 0.75.0 anyio 4.12.0 apache-tvm-ffi 0.1.11 aria2 0.0.1b0 asttokens 3.0.1 attrs 25.4.0 av 16.1.0 black 25.12.0 blinker 1.9.0 blobfile 3.0.0 build 1.3.0 cache_dit 1.2.0 cachetools 7.1.2 certifi 2025.11.12 cffi 2.0.0 charset-normalizer 3.4.4 click 8.4.2 cloudpickle 3.1.2 compressed-tensors 0.12.2 cuda-bindings 13.2.0 cuda-pathfinder 1.3.3 cuda-python 13.2.0 cuda-tile 1.3.0 datasets 4.4.1 decorator 5.2.1 decord2 2.0.0 diffusers 0.36.0 dill 0.4.0 diskcache 5.6.3 distro 1.9.0 docstring_parser 0.17.0 einops 0.8.1 executing 2.2.1 fastapi 0.124.4 filelock 3.20.0 flashinfer-cubin 0.6.3 flashinfer-python 0.6.3 frozenlist 1.8.0 fsspec 2025.10.0 ftllm 0.1.7.1 gguf 0.17.1 gitdb 4.0.12 GitPython 3.1.50 googleapis-common-protos 1.72.0 grpcio 1.83.0 grpcio-health-checking 1.81.1 grpcio-reflection 1.81.1 grpcio-tools 1.75.1 h11 0.16.0 hf_transfer 0.1.9 hf-xet 1.6.0 httpcore 1.0.9 httptools 0.7.1 httpx 0.28.1 huggingface_hub 1.26.0 humanize 4.15.0 idna 3.11 imageio 2.36.0 imageio-ffmpeg 0.5.1 importlib_metadata 8.7.1 interegular 0.3.3 ipython 9.8.0 ipython_pygments_lexers 1.1.1 itsdangerous 2.2.0 jedi 0.19.2 Jinja2 3.1.6 jiter 0.12.0 jsonschema 4.25.1 jsonschema-specifications 2025.9.1 kt-kernel 0.6.4 lark 1.3.1 llguidance 0.7.30 loguru 0.7.3 lxml 6.0.2 markdown-it-py 4.0.0 MarkupSafe 3.0.3 matplotlib-inline 0.2.1 mdurl 0.1.2 ml_dtypes 0.5.4 modelscope 1.33.0 moviepy 2.2.1 mpmath 1.3.0 msgspec 0.20.0 multidict 6.7.0 multiprocess 0.70.18 mypy_extensions 1.1.0 narwhals 2.21.2 nest-asyncio 1.6.0 networkx 3.6.1 ninja 1.13.0 numpy 2.3.5 nvidia-cublas-cu12 12.8.4.1 nvidia-cuda-crt 13.3.73 nvidia-cuda-cupti-cu12 12.8.90 nvidia-cuda-nvcc 13.2.78 nvidia-cuda-nvdisasm 13.3.73 nvidia-cuda-nvrtc-cu12 12.8.93 nvidia-cuda-runtime 13.3.29 nvidia-cuda-runtime-cu12 12.8.90 nvidia-cuda-tileiras 13.2.78 nvidia-cudnn-cu12 9.10.2.21 nvidia-cudnn-frontend 1.16.0 nvidia-cufft-cu12 11.3.3.83 nvidia-cufile-cu12 1.13.1.3 nvidia-curand-cu12 10.3.9.90 nvidia-cusolver-cu12 11.7.3.90 nvidia-cusparse-cu12 12.5.8.93 nvidia-cusparselt-cu12 0.7.1 nvidia-cutlass-dsl 4.3.5 nvidia-cutlass-dsl-libs-base 4.6.0 nvidia-cutlass-dsl-libs-core 4.6.0 nvidia-cutlass-dsl-libs-cu12 4.6.0 nvidia-ml-py 13.590.44 nvidia-nccl-cu12 2.27.5 nvidia-nvjitlink-cu12 12.8.93 nvidia-nvshmem-cu12 3.3.20 nvidia-nvtx-cu12 12.8.90 nvidia-nvvm 13.2.78 openai 2.6.1 openai-harmony 0.0.4 opencv-python-headless 4.10.0.84 opentelemetry-api 1.39.1 opentelemetry-exporter-otlp 1.39.1 opentelemetry-exporter-otlp-proto-common 1.39.1 opentelemetry-exporter-otlp-proto-grpc 1.39.1 opentelemetry-exporter-otlp-proto-http 1.39.1 opentelemetry-proto 1.39.1 opentelemetry-sdk 1.39.1 opentelemetry-semantic-conventions 0.60b1 orjson 3.11.5 outlines 0.1.11 outlines_core 0.1.26 packaging 25.0 pandas 2.3.3 parso 0.8.5 partial-json-parser 0.2.1.1.post7 pathspec 0.12.1 pexpect 4.9.0 pillow 11.3.0 pip 25.3 platformdirs 4.5.1 proglog 0.1.12 prometheus_client 0.23.1 prompt_toolkit 3.0.52 propcache 0.4.1 protobuf 6.33.6 psutil 7.1.3 ptyprocess 0.7.0 pure_eval 0.2.3 py-spy 0.4.1 pyarrow 22.0.0 pybase64 1.4.3 pycountry 24.6.1 pycparser 2.23 pycryptodomex 3.23.0 pydantic 2.12.5 pydantic_core 2.41.5 pydeck 0.9.2 Pygments 2.19.2 pyproject_hooks 1.2.0 pyreadline3 3.5.6 python-dateutil 2.9.0.post0 python-dotenv 1.2.1 python-multipart 0.0.20 pytokens 0.3.0 pytz 2025.2 PyYAML 6.0.1 pyzmq 27.1.0 quack-kernels 0.2.4 referencing 0.37.0 regex 2025.11.3 remote-pdb 2.1.0 requests 2.32.5 rich 14.3.3 rpds-py 0.30.0 runai-model-streamer 0.15.6 safetensors 0.8.0 scipy 1.16.3 sentencepiece 0.2.1 setproctitle 1.3.7 setuptools 80.9.0 sgl-kernel 0.3.21 sglang 0.0.0.dev9743+ga864420ee /home/ktransformers/sglang/python sglang-kt 0.0.0.dev0 /home/ktransformers/sglang/python sglang-router 0.3.2 shellingham 1.5.4 shortuuid 1.0.13 six 1.17.0 smg-grpc-proto 0.4.14 smmap 5.0.3 sniffio 1.3.1 soundfile 0.13.1 st-attn 0.0.7 stack-data 0.6.3 starlette 0.50.0 streamlit 1.57.0 streamlit-chat 0.1.1 sympy 1.14.0 tabulate 0.9.0 tenacity 9.1.4 tiktoken 0.12.0 tilelang 0.1.12 timm 1.0.16 tokenizers 0.22.1 toml 0.10.2 torch 2.9.1 torch_c_dlpack_ext 0.1.5 torch_memory_saver 0.0.9 torchao 0.9.0 torchaudio 2.9.1 torchcodec 0.8.0 torchvision 0.24.1 tqdm 4.67.1 traitlets 5.14.3 transformers 5.14.1 transformers-kt 5.6.0.post1 triton 3.5.1 typer 0.24.1 typing_extensions 4.15.0 typing-inspection 0.4.2 tzdata 2025.3 urllib3 2.6.2 uvicorn 0.38.0 uvloop 0.22.1 vsa 0.0.4 watchdog 6.0.0 wcwidth 0.2.14 websockets 16.0 wheel 0.45.1 xgrammar 0.1.27 xxhash 3.6.0 yarl 1.22.0 z3-solver 4.15.4.0 zipp 3.23.0 ### Reproduction 使用--kt-cpuinfer 40 --kt-threadpool-count 1 参数可以正常运行， 执行命令以及异常打印如下： ```shell python -m sglang.launch_server --host 0.0.0.0 --port 6398 --model /models/deepseek-ai/DeepSeek-V4-Flash-0731 --kt-weight-path /models/deepseek-ai/DeepSeek-V4-Flash-0731-AMXINT4/ --kt-method AMXINT4 --kt-num-gpu-experts 0 --kt-cpuinfer 80 --kt-threadpool-count 2 --context-length 400000 --max-total-tokens 500000 --attention-backend flashinfer --mem-fraction-static 0.9 --chunked-prefill-size 2048 --max-running-requests 4 --trust-remote-code --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --skip-server-warmup --served-model-name deepseek-v4-flash [2026-08-04 13:49:39] WARNING hf_transformers_utils.py:236: SGLANG_APPLY_CONFIG_BACKUP=auto: checkpoint has num_hidden_layers=43, dispatching to 'small'. [2026-08-04 13:49:39] WARNING hf_transformers_utils.py:255: SGLANG_APPLY_CONFIG_BACKUP=small: using packaged /root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/configs/config_backup_small.json instead of the checkpoint's config.json at /models/deepseek-ai/DeepSeek-V4-Flash-0731. [2026-08-04 13:49:39] INFO _model_config_dsv4.py:117: Auto-detected routed-expert safetensors dtype=I8; SGLANG_DSV4_FP4_EXPERTS=True [2026-08-04 13:49:39] INFO model_config.py:419: Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] [2026-08-04 13:49:39] WARNING model_config.py:1151: Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround. [2026-08-04 13:49:39] INFO server_args.py:1407: Use compressed attention backend for DeepseekV4ForCausalLM, setting page_size to 256. [2026-08-04 13:49:39] WARNING server_args.py:1422: Setting KV cache dtype to fp8_e4m3 for DeepseekV4ForCausalLM. [2026-08-04 13:49:39] INFO server_args.py:1450: Setting swa_full_tokens_ratio to 0.1 for DeepseekV4ForCausalLM. [2026-08-04 13:49:39] WARNING server_args.py:2627: KTransformers EP is enabled. --disable-shared-experts-fusion is automatically set to prevent shared experts from being offloaded to CPU. [transformers] `BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead. [2026-08-04 13:49:40] server_args=ServerArgs(model_path='/models/deepseek-ai/DeepSeek-V4-Flash-0731', tokenizer_path='/models/deepseek-ai/DeepSeek-V4-Flash-0731', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=400000, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='0.0.0.0', port=6398, fastapi_root_path='', grpc_mode=False, skip_server_warmup=True, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='fp8_e4m3', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.9, max_running_requests=4, max_queued_requests=None, max_total_tokens=500000, chunked_prefill_size=2048, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=256, swa_full_tokens_ratio=0.1, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=461787985, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='deepseek-v4-flash', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser='deepseek-v4', tool_call_parser='deepseekv4', tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='compressed', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='flashinfer_cutlass', nsa_prefill_backend=None, nsa_decode_backend=None, disable_flashinfer_autotune=False, mamba_backend='triton', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, enable_aiter_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype=None, mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_hisparse=False, hisparse_config=None, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path='/models/deepseek-ai/DeepSeek-V4-Flash-0731-AMXINT4/', kt_method='AMXINT4', kt_cpuinfer=80, kt_threadpool_count=2, kt_numa_nodes=None, kt_num_gpu_experts=0, kt_gpu_experts_ratio=None, kt_max_deferred_experts_per_token=None, kt_gpu_prefill_token_threshold=None, record_kt_gpu_expert_distribution=False, kt_enable_dynamic_expert_update=False, kt_expert_placement_strategy='uniform', kt_lora_path=None, kt_expert_lora_path=None, dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=24, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=2048, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='round-robin-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, enable_mm_global_cache=False, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None) [2026-08-04 13:49:40] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] [2026-08-04 13:49:40] Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround. [2026-08-04 13:49:41] No HuggingFace chat template found [2026-08-04 13:49:41] No chat template found, defaulting to 'string' content format [transformers] `BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead. [2026-08-04 13:49:45] SGLANG_APPLY_CONFIG_BACKUP=auto: checkpoint has num_hidden_layers=43, dispatching to 'small'. [2026-08-04 13:49:45] SGLANG_APPLY_CONFIG_BACKUP=small: using packaged /root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/configs/config_backup_small.json instead of the checkpoint's config.json at /models/deepseek-ai/DeepSeek-V4-Flash-0731. [2026-08-04 13:49:45] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] [2026-08-04 13:49:45] Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround. [transformers] `BaseImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `BaseImageProcessor` instead. [2026-08-04 13:49:46] Mamba selective_state_update backend initialized: triton [2026-08-04 13:49:46] Hybrid swa model: self.hf_config.architectures=['DeepseekV4ForCausalLM'] [2026-08-04 13:49:46] Transformers version 5.14.1 is used for model type deepseek_v4. If you experience issues related to RoPE parameters, they may be due to incompatibilities between Transformers >=5.0.0 and some models. You can try downgrading to transformers==4.57.1 as a workaround. [2026-08-04 13:49:46] Init torch distributed begin. [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [2026-08-04 13:49:46] Init torch distributed ends. elapsed=0.18 s, mem usage=0.04 GB [2026-08-04 13:49:47] Load weight begin. avail mem=31.14 GB [2026-08-04 13:49:47] Detected fp8 checkpoint. [2026-08-04 13:49:47] Using kt_num_gpu_experts=0 per layer, total GPU experts: 0 (= 0 × 43 MoE layers) [2026-08-04 13:49:47] Using uniform strategy for GPU expert placement [2026-08-04 13:49:47] KT GPU experts: layer 0 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 1 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 2 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 3 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 4 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 5 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 6 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 7 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 8 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 9 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 10 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 11 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 12 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 13 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 14 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 15 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 16 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 17 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 18 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 19 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 20 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 21 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 22 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 23 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 24 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 25 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 26 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 27 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 28 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 29 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 30 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 31 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 32 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 33 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 34 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 35 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 36 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 37 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 38 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 39 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 40 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 41 (MoE) has 0 GPU experts [2026-08-04 13:49:47] KT GPU experts: layer 42 (MoE) has 0 GPU experts [2026-08-04 13:49:47] Generated KT GPU experts masks using 'uniform' strategy: 43 MoE layers (out of 43 total layers) x 256 experts, total GPU experts in MoE layers = 0 [2026-08-04 13:49:47] [KT] Created shared staging buffer: 16.0 MiB (shape=torch.Size([2048, 4096]), dtype=torch.bfloat16) CPUInfer[0x5f0f3da117f0]: Hello WorkerPool[0x5f0f3da143c0] 2 subpools, [numa:threads][0:40] [1:40] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 40 threads ===========In NumaPool============ In Numa Worker Pool at NUMA 1, 40 threads Loading safetensors checkpoint shards: 0% Completed | 0/48 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 10% Completed | 5/48 [00:00<00:00, 48.82it/s] Loading safetensors checkpoint shards: 21% Completed | 10/48 [00:00<00:00, 48.23it/s] Loading safetensors checkpoint shards: 31% Completed | 15/48 [00:00<00:01, 21.88it/s] Loading safetensors checkpoint shards: 42% Completed | 20/48 [00:00<00:01, 27.38it/s] Loading safetensors checkpoint shards: 52% Completed | 25/48 [00:00<00:00, 31.65it/s] Loading safetensors checkpoint shards: 62% Completed | 30/48 [00:00<00:00, 34.85it/s] Loading safetensors checkpoint shards: 73% Completed | 35/48 [00:01<00:00, 37.56it/s] Loading safetensors checkpoint shards: 85% Completed | 41/48 [00:01<00:00, 41.85it/s] Loading safetensors checkpoint shards: 96% Completed | 46/48 [00:01<00:00, 42.26it/s] Loading safetensors checkpoint shards: 100% Completed | 48/48 [00:01<00:00, 37.17it/s] TP MOE layer 0, pool: 0x5f0f3da143c0, expert num: 256, num_experts_per_tok: 6 Creating AMX_MOE_TP 1 at numa 1 Creating AMX_MOE_TP 0 at numa 0 TP Load from loader Fatal Python error: Segmentation fault Thread 0x0000755d71fff640 (most recent call first): File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 331 in wait File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 629 in wait File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 1045 in _bootstrap_inner File "/root/miniforge3/envs/kt/lib/python3.11/threading.py", line 1002 in _bootstrap Thread 0x0000758c8a9ec740 (most recent call first): File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 537 in load_weights File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 3684 in process_weights_after_loading File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 697 in load_weights_and_postprocess File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 677 in load_model File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 1060 in load_model File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 501 in initialize File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 418 in __init__ File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 330 in _init_model_runner File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 247 in __init__ File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 521 in init_tp_model_worker File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 563 in init_model_worker File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 367 in __init__ File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3207 in run_scheduler_process File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/process.py", line 108 in run File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/process.py", line 314 in _bootstrap File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/spawn.py", line 135 in _main File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/spawn.py", line 122 in spawn_main File "<string>", line 1 in <module> Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pybase64._pybase64, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, psutil._psutil_linux, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, regex._regex, yaml._yaml, markupsafe._speedups, cuda_utils, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, av.option, av.descriptor, av.format, av.utils, av.stream, av.container.streams, av.sidedata.encparams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.link, av.filter.context, av.filter.graph, av.filter.filter, av.filter.loudnorm, av.audio.resampler, av.audio.codeccontext, av.audio.fifo, av.bitstream, av.video.codeccontext, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._mt19937, numpy.random._generator, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, _cyutility, scipy._cyutility, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation, scipy.spatial.transform._rigid_transform, scipy.optimize._direct, setproctitle._setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, msgspec._core, grpc._cython.cygrpc, google._upb._message, tilelang_cython_wrapper (total: 158) !!!!!!! Segfault encountered !!!!!!! File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000758c8a64251f File "../../../../../libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine File "./nptl/pthread_create.c", line 442, in start_thread File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 File "<unknown>", line 0, in 0xffffffffffffffff !!!!!!! Segfault encountered !!!!!!! File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000758c8a64251f File "../../../../../libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine File "./nptl/pthread_create.c", line 442, in start_thread File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 81, in __clone3 File "<unknown>", line 0, in 0xffffffffffffffff [2026-08-04 13:50:15] Rank 0 scheduler is dead. Please check if there are relevant logs. [2026-08-04 13:50:16] Exit code: -11 Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/launch_server.py", line 112, in <module> run_server(server_args) File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/launch_server.py", line 105, in run_server launch_server(server_args) File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/entrypoints/http_server.py", line 1960, in launch_server _launch_subprocesses( File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/entrypoints/engine.py", line 1115, in _launch_subprocesses scheduler_infos = _wait_for_scheduler_ready(scheduler_pipe_readers, scheduler_procs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniforge3/envs/kt/lib/python3.11/site-packages/sglang/srt/entrypoints/engine.py", line 925, in _wait_for_scheduler_ready data = scheduler_pipe_readers[i].recv() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() ^^^^^^^^^^^^^^^^^^ File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/connection.py", line 430, in _recv_bytes buf = self._recv(4) ^^^^^^^^^^^^^ File "/root/miniforge3/envs/kt/lib/python3.11/multiprocessing/connection.py", line 399, in _recv raise EOFError EOFError ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:50:52 +00:00
Author
Owner
Copy link

@CodeZ-Hao commented on GitHub (Aug 5, 2026):

更新，确认是转换模型的时候没有加kt-threadpool-count=2 导致，关闭问题

<!-- gh-comment-id:5190477932 --> @CodeZ-Hao commented on GitHub (Aug 5, 2026): 更新，确认是转换模型的时候没有加kt-threadpool-count=2 导致，关闭问题
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1286
No description provided.