kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #1915] 在L20*8上部署了GLM5-FP8，感觉速度有点慢，想看看能不能再优化一下 #1191

New issue

Open

opened 2026-10-29 14:49:44 +00:00 by bsakaguchi · 6 comments

bsakaguchi commented

2026-10-29 14:49:44 +00:00

Owner

Originally created by @applezzj on GitHub (Apr 3, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1915

Originally assigned to: @ouqingliang on GitHub.

Reminder

I have read the above rules and searched the existing issues.

System Info

GPU: L20（48G）* 8
CPU: Intel(R) Xeon(R) Gold 6430 64核128线程 * 2
kt-kernel 0.5.2.post1
sglang-kt 0.5.2.post1
NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4

Reproduction

启动参数如下，目前速度单请求13-14 token/s

PYTORCH_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0 python -m sglang.launch_server \
  --model /data/GLM-5-FP8 \
  --kt-weight-path /data/GLM-5-FP8 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 72\
  --kt-method FP8 \
  --kt-gpu-prefill-token-threshold 1024 \
  --kt-enable-dynamic-expert-update \
  --kt-expert-placement-strategy uniform \
  --kt-max-deferred-experts-per-token 2 \
  --trust-remote-code \
  --mem-fraction-static 0.85 \
  --served-model-name GLM5 \
  --enable-mixed-chunk \
  --tensor-parallel-size 8 \
  --enable-p2p-check \
  --disable-shared-experts-fusion \
  --chunked-prefill-size 16384 \
  --max-running-requests 4 \
  --max-total-tokens 108000 \
  --attention-backend flashinfer \
  --fp8-gemm-backend auto \
  --kv-cache-dtype bf16 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --watchdog-timeout 3000

Others

另外想问一下，CPU是支持AMX指令的，L20应该也支持INT8计算，如果把权重转换成AMXINT8，会不会有更快的推理速度？

Originally created by @applezzj on GitHub (Apr 3, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1915 Originally assigned to: @ouqingliang on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info GPU: L20（48G）* 8 CPU: Intel(R) Xeon(R) Gold 6430 64核128线程 * 2 kt-kernel 0.5.2.post1 sglang-kt 0.5.2.post1 NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 ### Reproduction ```text 启动参数如下，目前速度单请求13-14 token/s PYTORCH_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0 python -m sglang.launch_server \ --model /data/GLM-5-FP8 \ --kt-weight-path /data/GLM-5-FP8 \ --kt-cpuinfer 64 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 72\ --kt-method FP8 \ --kt-gpu-prefill-token-threshold 1024 \ --kt-enable-dynamic-expert-update \ --kt-expert-placement-strategy uniform \ --kt-max-deferred-experts-per-token 2 \ --trust-remote-code \ --mem-fraction-static 0.85 \ --served-model-name GLM5 \ --enable-mixed-chunk \ --tensor-parallel-size 8 \ --enable-p2p-check \ --disable-shared-experts-fusion \ --chunked-prefill-size 16384 \ --max-running-requests 4 \ --max-total-tokens 108000 \ --attention-backend flashinfer \ --fp8-gemm-backend auto \ --kv-cache-dtype bf16 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --watchdog-timeout 3000 ``` ### Others 另外想问一下，CPU是支持AMX指令的，L20应该也支持INT8计算，如果把权重转换成AMXINT8，会不会有更快的推理速度？

bsakaguchi commented

2026-10-29 14:49:45 +00:00

Author

Owner

@SkqLiao commented on GitHub (Apr 7, 2026):

GLM-5原生支持MTP，会加速Decode

--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \

@SkqLiao commented on GitHub (Apr 7, 2026): GLM-5原生支持MTP，会加速Decode ``` --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ ```

bsakaguchi commented

2026-10-29 14:49:45 +00:00

Author

Owner

@applezzj commented on GitHub (Apr 15, 2026):

加了这些参数之后报错了
--speculative-algorithm EAGLE
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4 \

报错如下：
[2026-04-15 07:32:27 TP0] Scheduler hit an exception: Traceback (most recent call last):
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3118, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 363, in init
self.init_model_worker()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 560, in init_model_worker
self.maybe_init_draft_worker()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 556, in maybe_init_draft_worker
self.draft_worker = DraftWorkerClass(**draft_worker_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 142, in init
super().init(
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 247, in init
self._init_model_runner()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner
self._model_runner = ModelRunner(
^^^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 415, in init
self.initialize(min_per_gpu_memory)
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 495, in initialize
self.load_model()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 970, in load_model
self.model = self.loader.load_model(
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 677, in load_model
self.load_weights_and_postprocess(
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 697, in load_weights_and_postprocess
quant_method.process_weights_after_loading(module)
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2143, in process_weights_after_loading
self.wrapper.load_weights(physical_to_logical_map_cpu)
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 454, in load_weights
weights = self.loader.load_experts(base_key)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/loader.py", line 407, in load_experts
raise ValueError(f"No experts found for keys: {experts_prefix_candidates}")
ValueError: No experts found for keys: ['model.layers.0.mlp.experts', 'layers.0.mlp.experts']

[2026-04-15 07:32:27] Received sigquit from a child process. It usually means the child failed.

@applezzj commented on GitHub (Apr 15, 2026): 加了这些参数之后报错了 --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ 报错如下： [2026-04-15 07:32:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3118, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 363, in __init__ self.init_model_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 560, in init_model_worker self.maybe_init_draft_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 556, in maybe_init_draft_worker self.draft_worker = DraftWorkerClass(**draft_worker_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 142, in __init__ super().__init__( File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 247, in __init__ self._init_model_runner() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 415, in __init__ self.initialize(min_per_gpu_memory) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 495, in initialize self.load_model() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 970, in load_model self.model = self.loader.load_model( ^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 677, in load_model self.load_weights_and_postprocess( File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 697, in load_weights_and_postprocess quant_method.process_weights_after_loading(module) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2143, in process_weights_after_loading self.wrapper.load_weights(physical_to_logical_map_cpu) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 454, in load_weights weights = self.loader.load_experts(base_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/loader.py", line 407, in load_experts raise ValueError(f"No experts found for keys: {experts_prefix_candidates}") ValueError: No experts found for keys: ['model.layers.0.mlp.experts', 'layers.0.mlp.experts'] [2026-04-15 07:32:27] Received sigquit from a child process. It usually means the child failed.

bsakaguchi commented

2026-10-29 14:49:46 +00:00

Author

Owner

@epheien commented on GitHub (Apr 16, 2026):

有没有试过awq或者其他的int4量化? int量化理论上会快不少

@epheien commented on GitHub (Apr 16, 2026): 有没有试过awq或者其他的int4量化? int量化理论上会快不少

bsakaguchi commented

2026-10-29 14:49:46 +00:00

Author

Owner

@ErvinXie commented on GitHub (Apr 17, 2026):

@applezzj MTP 是支持 GLM-5 的，这个报错应该是参数配置的问题。我们会找时间排查一下具体原因，确认后会更新这里。

在此之前，关于当前 FP8 模式下提速的其他方向可以先试试：

增加 --kt-num-gpu-experts：你目前设了 72，L20*8 有 384GB 显存总量，可以尝试适当增大这个值，让更多 expert 常驻 GPU
确认 --kt-cpuinfer 线程数：你设了 64，CPU 是 64 核 128 线程 * 2，可以试试增大到 96-112，充分利用多核
如 @epheien 所说，INT4 量化（如 AWQ）理论上 CPU 端计算会更快，可以作为后续尝试的方向

This reply was co-authored by an AI agent and a human.

@ErvinXie commented on GitHub (Apr 17, 2026): @applezzj MTP 是支持 GLM-5 的，这个报错应该是参数配置的问题。我们会找时间排查一下具体原因，确认后会更新这里。在此之前，关于当前 FP8 模式下提速的其他方向可以先试试： 1. **增加 `--kt-num-gpu-experts`**：你目前设了 72，L20*8 有 384GB 显存总量，可以尝试适当增大这个值，让更多 expert 常驻 GPU 2. **确认 `--kt-cpuinfer` 线程数**：你设了 64，CPU 是 64 核 128 线程 * 2，可以试试增大到 96-112，充分利用多核 3. 如 @epheien 所说，INT4 量化（如 AWQ）理论上 CPU 端计算会更快，可以作为后续尝试的方向 --- *This reply was co-authored by an AI agent and a human.*

bsakaguchi commented

2026-10-29 14:49:46 +00:00

Author

Owner

@ouqingliang commented on GitHub (Apr 17, 2026):

加了这些参数之后报错了 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \

报错如下： [2026-04-15 07:32:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3118, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 363, in init self.init_model_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 560, in init_model_worker self.maybe_init_draft_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 556, in maybe_init_draft_worker self.draft_worker = DraftWorkerClass(**draft_worker_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 142, in init super().init( File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 247, in init self._init_model_runner() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 415, in init self.initialize(min_per_gpu_memory) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 495, in initialize self.load_model() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 970, in load_model self.model = self.loader.load_model( ^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 677, in load_model self.load_weights_and_postprocess( File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 697, in load_weights_and_postprocess quant_method.process_weights_after_loading(module) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2143, in process_weights_after_loading self.wrapper.load_weights(physical_to_logical_map_cpu) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 454, in load_weights weights = self.loader.load_experts(base_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/loader.py", line 407, in load_experts raise ValueError(f"No experts found for keys: {experts_prefix_candidates}") ValueError: No experts found for keys: ['model.layers.0.mlp.experts', 'layers.0.mlp.experts']

[2026-04-15 07:32:27] Received sigquit from a child process. It usually means the child failed.

使用 0.5.3 版本试试呢？kt 对 mtp 的支持是在此版本中加入的。

@ouqingliang commented on GitHub (Apr 17, 2026): > 加了这些参数之后报错了 --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ > > 报错如下： [2026-04-15 07:32:27 TP0] Scheduler hit an exception: Traceback (most recent call last): File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3118, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 363, in **init** self.init_model_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 560, in init_model_worker self.maybe_init_draft_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 556, in maybe_init_draft_worker self.draft_worker = DraftWorkerClass(**draft_worker_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 142, in **init** super().**init**( File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 247, in **init** self._init_model_runner() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 415, in **init** self.initialize(min_per_gpu_memory) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 495, in initialize self.load_model() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 970, in load_model self.model = self.loader.load_model( ^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 677, in load_model self.load_weights_and_postprocess( File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_loader/loader.py", line 697, in load_weights_and_postprocess quant_method.process_weights_after_loading(module) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/layers/moe/kt_ep_wrapper.py", line 2143, in process_weights_after_loading self.wrapper.load_weights(physical_to_logical_map_cpu) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/amx.py", line 454, in load_weights weights = self.loader.load_experts(base_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/kt_kernel/utils/loader.py", line 407, in load_experts raise ValueError(f"No experts found for keys: {experts_prefix_candidates}") ValueError: No experts found for keys: ['model.layers.0.mlp.experts', 'layers.0.mlp.experts'] > > [2026-04-15 07:32:27] Received sigquit from a child process. It usually means the child failed. 使用 0.5.3 版本试试呢？kt 对 mtp 的支持是在此版本中加入的。

bsakaguchi commented

2026-10-29 14:49:46 +00:00

Author

Owner

@applezzj commented on GitHub (May 19, 2026):

GLM-5原生支持MTP，会加速Decode

--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \

目前更新了版本
kt-kernel 0.6.2.post3
sglang-kt 0.6.2.post3
transformers-kt 5.6.0.post1
尝试使用MTP加速，但是报错

PYTORCH_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0 python -m sglang.launch_server
--model /data/GLM-5.1-FP8
--kt-weight-path /data/GLM-5.1-FP8
--kt-cpuinfer 64
--kt-threadpool-count 2
--kt-num-gpu-experts 70
--kt-method FP8
--kt-gpu-prefill-token-threshold 1024
--kt-enable-dynamic-expert-update
--kt-expert-placement-strategy uniform
--kt-max-deferred-experts-per-token 2
--trust-remote-code
--mem-fraction-static 0.85
--served-model-name GLM5
--enable-mixed-chunk
--tensor-parallel-size 8
--enable-p2p-check
--disable-shared-experts-fusion
--chunked-prefill-size 16384
--max-running-requests 4
--max-total-tokens 108000
--attention-backend flashinfer
--fp8-gemm-backend auto
--kv-cache-dtype bf16
--tool-call-parser glm47
--reasoning-parser glm45
--watchdog-timeout 3000
--speculative-algorithm EAGLE
--speculative-num-steps 3
--speculative-eagle-topk 1
--speculative-num-draft-tokens 4

报错内容：

[2026-05-19 09:02:43 TP3] Scheduler hit an exception: Traceback (most recent call last):
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3207, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 367, in init
self.init_model_worker()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 564, in init_model_worker
self.maybe_init_draft_worker()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 560, in maybe_init_draft_worker
self.draft_worker = DraftWorkerClass(**draft_worker_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 206, in init
self.init_cuda_graphs()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 252, in init_cuda_graphs
self.cuda_graph_runner = Device2DraftCudaGraphRunner[
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 164, in init
self.capture()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 214, in capture
CudaGraphRunner.capture(self)
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 723, in capture
_capture_one_stream()
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 710, in _capture_one_stream
) = self.capture_one_batch_size(bs, forward, stream_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 341, in capture_one_batch_size
self._capture_init(run_once)
File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 203, in _capture_init
self.model_runner.draft_attn_backend.on_after_cuda_graph_warmup_pass()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'FlashInferMLAMultiStepDraftBackend' object has no attribute 'on_after_cuda_graph_warmup_pass'

[2026-05-19 09:02:43] Received sigquit from a child process. It usually means the child failed.

@applezzj commented on GitHub (May 19, 2026): > GLM-5原生支持MTP，会加速Decode > > ``` > --speculative-algorithm EAGLE \ > --speculative-num-steps 3 \ > --speculative-eagle-topk 1 \ > --speculative-num-draft-tokens 4 \ > ``` ---------------------------------------------------------------------------------------------------------------- 目前更新了版本 kt-kernel 0.6.2.post3 sglang-kt 0.6.2.post3 transformers-kt 5.6.0.post1 尝试使用MTP加速，但是报错 ---------------------------------------------------------------------------------------------------------------- PYTORCH_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0 python -m sglang.launch_server \ --model /data/GLM-5.1-FP8 \ --kt-weight-path /data/GLM-5.1-FP8 \ --kt-cpuinfer 64 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 70\ --kt-method FP8 \ --kt-gpu-prefill-token-threshold 1024 \ --kt-enable-dynamic-expert-update \ --kt-expert-placement-strategy uniform \ --kt-max-deferred-experts-per-token 2 \ --trust-remote-code \ --mem-fraction-static 0.85 \ --served-model-name GLM5 \ --enable-mixed-chunk \ --tensor-parallel-size 8 \ --enable-p2p-check \ --disable-shared-experts-fusion \ --chunked-prefill-size 16384 \ --max-running-requests 4 \ --max-total-tokens 108000 \ --attention-backend flashinfer \ --fp8-gemm-backend auto \ --kv-cache-dtype bf16 \ --tool-call-parser glm47 \ --reasoning-parser glm45 \ --watchdog-timeout 3000 \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 ---------------------------------------------------------------------------------------------------------------- 报错内容： [2026-05-19 09:02:43 TP3] Scheduler hit an exception: Traceback (most recent call last): File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 3207, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 367, in __init__ self.init_model_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 564, in init_model_worker self.maybe_init_draft_worker() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 560, in maybe_init_draft_worker self.draft_worker = DraftWorkerClass(**draft_worker_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 206, in __init__ self.init_cuda_graphs() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_worker.py", line 252, in init_cuda_graphs self.cuda_graph_runner = Device2DraftCudaGraphRunner[ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 164, in __init__ self.capture() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 214, in capture CudaGraphRunner.capture(self) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 723, in capture _capture_one_stream() File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 710, in _capture_one_stream ) = self.capture_one_batch_size(bs, forward, stream_idx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 341, in capture_one_batch_size self._capture_init(run_once) File "/root/miniconda3/envs/kt/lib/python3.11/site-packages/sglang/srt/speculative/eagle_draft_cuda_graph_runner.py", line 203, in _capture_init self.model_runner.draft_attn_backend.on_after_cuda_graph_warmup_pass() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'FlashInferMLAMultiStepDraftBackend' object has no attribute 'on_after_cuda_graph_warmup_pass' [2026-05-19 09:02:43] Received sigquit from a child process. It usually means the child failed.