#1254 - [GH-ISSUE #2050] GLM5.2 on RTX 6000 Pro - Loads but segfault on warmup inference - kvcache-ai/ktransformers

bsakaguchi commented

2026-10-29 14:50:35 +00:00

Owner

Originally created by @trilog-inc on GitHub (Jun 17, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2050

Reminder

I have read the above rules and searched the existing issues.

System Info

ktransformers latest version and auto/sync-sglang.
python 3.11, conda environment
CUDA 12.8
gcc (Debian 14.2.0-19)

CPU W7-3465x, 768GB
GPU 1x RTX 6000 pro

Reproduction

With command:

SGLANG_DEBUG=1 SGLANG_ENABLE_JIT_DEEPGEMM=0 PYTORCH_ALLOC_CONF=expandable_segments:True CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path /mnt/home_extend/models/GLM52 --kt-weight-path /mnt/home_extend/models/GLM52 --kt-cpuinfer 22 --kt-threadpool-count 2 --kt-num-gpu-experts 1 --kt-method FP8 --kt-gpu-prefill-token-threshold 1024 --kt-enable-dynamic-expert-update --kt-expert-placement-strategy uniform --tp-size 1 --trust-remote-code --host 0.0.0.0 --port 8000 --mem-fraction-static 0.97 --kv-cache-dtype bf16 --max-total-tokens 4096 --max-running-requests 2 --attention-backend flashinfer --fp8-gemm-backend auto --disable-shared-experts-fusion --tool-call-parser glm47 --reasoning-parser glm45 --served-model-name GLM5.2 --watchdog-timeout 3000 --log-level debug

Server Loads and CUDA Graphs are loaded. On warmup inference:

[2026-06-17 12:56:41] Prefill batch, #new-seq: 1, #new-token: 64, #cached-token: 0, token usage: 0.02, #running-req: 0, #queue-req: 0, input throughput (token/s): 0.00, cuda graph: False
Fatal Python error: Segmentation fault

Thread 0x00007f359ffff6c0 (most recent call first):
File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/site-packages/sglang/srt/utils/watchdog.py", line 145 in _watchdog_once
File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/site-packages/sglang/srt/utils/watchdog.py", line 125 in _watchdog_thread
File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/threading.py", line 982 in run
File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/threading.py", line 1045 in _bootstrap_inner
File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/threading.py", line 1002 in _bootstrap

Has anyone gotten GLM5.2 to work on a similar setup?

Others

No response

Originally created by @trilog-inc on GitHub (Jun 17, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2050 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info ktransformers latest version and auto/sync-sglang. python 3.11, conda environment CUDA 12.8 gcc (Debian 14.2.0-19) CPU W7-3465x, 768GB GPU 1x RTX 6000 pro ### Reproduction With command: SGLANG_DEBUG=1 SGLANG_ENABLE_JIT_DEEPGEMM=0 PYTORCH_ALLOC_CONF=expandable_segments:True CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --model-path /mnt/home_extend/models/GLM52 --kt-weight-path /mnt/home_extend/models/GLM52 --kt-cpuinfer 22 --kt-threadpool-count 2 --kt-num-gpu-experts 1 --kt-method FP8 --kt-gpu-prefill-token-threshold 1024 --kt-enable-dynamic-expert-update --kt-expert-placement-strategy uniform --tp-size 1 --trust-remote-code --host 0.0.0.0 --port 8000 --mem-fraction-static 0.97 --kv-cache-dtype bf16 --max-total-tokens 4096 --max-running-requests 2 --attention-backend flashinfer --fp8-gemm-backend auto --disable-shared-experts-fusion --tool-call-parser glm47 --reasoning-parser glm45 --served-model-name GLM5.2 --watchdog-timeout 3000 --log-level debug Server Loads and CUDA Graphs are loaded. On warmup inference: [2026-06-17 12:56:41] Prefill batch, #new-seq: 1, #new-token: 64, #cached-token: 0, token usage: 0.02, #running-req: 0, #queue-req: 0, input throughput (token/s): 0.00, cuda graph: False Fatal Python error: Segmentation fault Thread 0x00007f359ffff6c0 (most recent call first): File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/site-packages/sglang/srt/utils/watchdog.py", line 145 in _watchdog_once File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/site-packages/sglang/srt/utils/watchdog.py", line 125 in _watchdog_thread File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/threading.py", line 982 in run File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/threading.py", line 1045 in _bootstrap_inner File "/home/myfrienderic/miniconda3/envs/ktglm52/lib/python3.11/threading.py", line 1002 in _bootstrap Has anyone gotten GLM5.2 to work on a similar setup? ### Others _No response_

bsakaguchi closed this issue

2026-10-29 14:50:35 +00:00

bsakaguchi commented

2026-10-29 14:50:36 +00:00

Author

Owner

@harrychk commented on GitHub (Jun 18, 2026):

I'm also looking at this problem. Finally, the error is caused by "delete args;" in "static void sync_(void* sync_args)" function in kt-kernel/cpu_backend/cpuinfer.h，which is added in the latest update. If I delete it, the error is gone.

@harrychk commented on GitHub (Jun 18, 2026): I'm also looking at this problem. Finally, the error is caused by "delete args;" in "static void sync_(void* sync_args)" function in kt-kernel/cpu_backend/cpuinfer.h，which is added in the latest update. If I delete it, the error is gone.

bsakaguchi commented

2026-10-29 14:50:36 +00:00

Author

Owner

@yyj6666667 commented on GitHub (Jun 18, 2026):

Thanks for feedback, already fixed
Bug 机制:

Graph capture 阶段: sync_with_cuda_stream() 调用一次 → new SyncArgs 堆分配 →
cudaLaunchHostFunc(stream, sync_, args) 把 args 指针记录成 graph 的 host-func 节点
Graph replay 第 1 次: callback 触发 → delete args → 内存释放
Graph replay 第 2 次: callback 用同一个 args 指针触发 → dangling pointer 解引用 → 之前 deref
args->cpuinfer 可能拿到回收复用的内存 → segfault / 数据异常

感谢反馈，目前已经修复

@yyj6666667 commented on GitHub (Jun 18, 2026): Thanks for feedback, already fixed Bug 机制: - Graph capture 阶段: sync_with_cuda_stream() 调用一次 → new SyncArgs 堆分配 → cudaLaunchHostFunc(stream, sync_, args) 把 args 指针记录成 graph 的 host-func 节点 - Graph replay 第 1 次: callback 触发 → delete args → 内存释放 - Graph replay 第 2 次: callback 用同一个 args 指针触发 → dangling pointer 解引用 → 之前 deref args->cpuinfer 可能拿到回收复用的内存 → segfault / 数据异常感谢反馈，目前已经修复

bsakaguchi commented

2026-10-29 14:50:36 +00:00

Author

Owner

@trilog-inc commented on GitHub (Jun 18, 2026):

@yyj6666667 Thanks for the update!
That helped the actual loading and I get to "The server is fired up and ready to roll!". However, the server never starts inference. Simply a " INFO: 127.0.0.1:52450 - "POST /v1/chat/completions HTTP/1.1" 200 OK " and then nothing.

Any tips on debugging this?

SGLANG_DEBUG=1 SGLANG_ENABLE_JIT_DEEPGEMM=0 PYTORCH_ALLOC_CONF=expandable_segments:True CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server
--model-path /mnt/home_extend/models/GLM52
--kt-weight-path /mnt/home_extend/models/GLM52
--kt-cpuinfer 22
--kt-threadpool-count 2
--kt-num-gpu-experts 15
--kt-method FP8
--kt-gpu-prefill-token-threshold 1024
--kt-enable-dynamic-expert-update
--kt-expert-placement-strategy uniform
--tp-size 1
--trust-remote-code
--host 0.0.0.0
--port 8000
--mem-fraction-static 0.97
--kv-cache-dtype fp8_e4m3
--max-total-tokens 4096
--max-running-requests 1
--attention-backend flashinfer
--fp8-gemm-backend triton -
-disable-shared-experts-fusion
--tool-call-parser glm47
--reasoning-parser glm45
--served-model-name GLM5.2
--watchdog-timeout 3000

@trilog-inc commented on GitHub (Jun 18, 2026): @yyj6666667 Thanks for the update! That helped the actual loading and I get to "The server is fired up and ready to roll!". However, the server never starts inference. Simply a " INFO: 127.0.0.1:52450 - "POST /v1/chat/completions HTTP/1.1" 200 OK " and then nothing. Any tips on debugging this? > SGLANG_DEBUG=1 SGLANG_ENABLE_JIT_DEEPGEMM=0 PYTORCH_ALLOC_CONF=expandable_segments:True CUDA_DEVICE_ORDER="PCI_BUS_ID" CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server > --model-path /mnt/home_extend/models/GLM52 > --kt-weight-path /mnt/home_extend/models/GLM52 > --kt-cpuinfer 22 > --kt-threadpool-count 2 > --kt-num-gpu-experts 15 > --kt-method FP8 > --kt-gpu-prefill-token-threshold 1024 > --kt-enable-dynamic-expert-update > --kt-expert-placement-strategy uniform > --tp-size 1 > --trust-remote-code > --host 0.0.0.0 > --port 8000 > --mem-fraction-static 0.97 > --kv-cache-dtype fp8_e4m3 > --max-total-tokens 4096 > --max-running-requests 1 > --attention-backend flashinfer > --fp8-gemm-backend triton - > -disable-shared-experts-fusion > --tool-call-parser glm47 > --reasoning-parser glm45 > --served-model-name GLM5.2 > --watchdog-timeout 3000

bsakaguchi commented

2026-10-29 14:50:36 +00:00

Author

Owner

@yyj6666667 commented on GitHub (Jun 24, 2026):

Does the log printing anything like '503 unreachable' or just '200ok' then nothing?

@yyj6666667 commented on GitHub (Jun 24, 2026): Does the log printing anything like '503 unreachable' or just '200ok' then nothing?

bsakaguchi commented

2026-10-29 14:50:36 +00:00

Author

Owner

@trilog-inc commented on GitHub (Jun 24, 2026):

Just the “200 OK” then nothing.

I am guessing this is probably related to the fact that the gpu is sm_120?

I see that lots of sm120 commits have been recently pushed to main line SGLang. And the latest flashinfer has fp8 gemm kernels for sm120. Does the ktransformers branch have to catch up for compatibility?

Thanks for your time!

@trilog-inc commented on GitHub (Jun 24, 2026): Just the “200 OK” then nothing. I am guessing this is probably related to the fact that the gpu is sm_120? I see that lots of sm120 commits have been recently pushed to main line SGLang. And the latest flashinfer has fp8 gemm kernels for sm120. Does the ktransformers branch have to catch up for compatibility? Thanks for your time!

bsakaguchi commented

2026-10-29 14:50:36 +00:00

Author

Owner

@yyj6666667 commented on GitHub (Jun 24, 2026):

I see. we'll continously follow the update of sglang upstream. If the problem get fixed, I'll be back and let you know.

@yyj6666667 commented on GitHub (Jun 24, 2026): I see. we'll continously follow the update of sglang upstream. If the problem get fixed, I'll be back and let you know.

bsakaguchi commented

2026-10-29 14:50:36 +00:00

Author

Owner

@trilog-inc commented on GitHub (Jun 29, 2026):

@yyj6666667 Does this help? https://github.com/sgl-project/sglang/pull/29580

@trilog-inc commented on GitHub (Jun 29, 2026): @yyj6666667 Does this help? https://github.com/sgl-project/sglang/pull/29580

Rows
Columns

[GH-ISSUE #2050] GLM5.2 on RTX 6000 Pro - Loads but segfault on warmup inference #1254

Reminder

System Info

Reproduction

Others