System Specifications

Author

Owner

@yumenoshizuku commented on GitHub (Dec 18, 2025):

bench_moe.py using CPUINFER_PARAM = 72

CPUInfer[0x3e647ee0]: Hello
WorkerPool[0x3e6aeae0] 1 subpools, [numa:threads][0:72]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 72 threads
Initializing MOEs:   0%|                                  | 0/1 [00:00<?, ?it/s]TP MOE layer 0, pool: 0x3e6aeae0, expert num: 256, num_experts_per_tok: 8
Llamafile TP splitting: intermediate_size=2048, tp_count=1, QK_K=256
  num_blocks=8, base_blocks=8, extra_blocks=0
  TP 0: intermediate_size=2048, offset=0, blocks=8
Initializing MOEs: 100%|█████████████████████████| 1/1 [04:15<00:00, 255.15s/it]Generating data...
Warming up...
Warm-up: 100%|████████████████████████████████| 100/100 [00:09<00:00, 11.08it/s]Start testing...
Testing: 100%|████████████████████████████| 10000/10000 [14:55<00:00, 11.17it/s]Quant mode: q4_k_m
Time(s): 895.418366427999
Iteration: 10000
Time(us) per iteration: 89541.8366427999
Bandwidth: 81.6452566638661 GB/s
TFLOPS: 8.058294678569345

CPUInfer[0x3e647ee0]: Goodbye

bench_moe_torch.py

Quant mode:  fp32
Time(s):  64.82605444799992
Iteration:  10000
Time(us) per iteration:  6482.605444799992
Bandwidth:  87.34621362066713 GB/s

Quant mode:  fp16
Time(s):  67.00558913100394
Iteration:  10000
Time(us) per iteration:  6700.558913100394
Bandwidth:  42.25252306139347 GB/s

Quant mode:  bf16
Time(s):  98.33891168500122
Iteration:  10000
Time(us) per iteration:  9833.891168500122
Bandwidth:  28.78977559837904 GB/s

Quant mode:  qint8
Time(s):  162.6409711400047
Iteration:  10000
Time(us) per iteration:  16264.097114000471
Bandwidth:  8.703696184778938 GB/s

Other moe benchmarks all ran into error :(

bench_moe_amx_k.py (all quantization modes)

worker_config.subpool_count = 1
worker_config.subpool_numa_map = [0]
worker_config.subpool_thread_count = [72]
CPUINFER_PARAM = 72

CPUInfer[0x28b5c670]: Hello
WorkerPool[0x2eeddef0] 1 subpools, [numa:threads][0:72]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 72 threads
TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8
From BF16 Online Quantization.
TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8
From BF16 Online Quantization.
TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8
From BF16 Online Quantization.
TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8
From BF16 Online Quantization.
Warm-up:   0%|                | 0/1000 [00:00<?, ?it/s]
Segmentation fault

bench_moe_amx.py (all quantization modes)

CPUInfer[0x45febda0]: Hello
WorkerPool[0x45fef920] 1 subpools, [numa:threads][0:72]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 72 threads
TP MOE layer 0, pool: 0x45fef920, expert num: 16, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
From BF16
  online quant from bf16
TP MOE layer 0, pool: 0x45fef920, expert num: 16, num_experts_per_tok: 8
Creating AMX_MOE_TP 0 at numa 0
From BF16
  online quant from bf16
Warm-up:   0%|                | 0/1000 [00:00<?, ?it/s]
Segmentation fault

bench_moe_kernel.py (int8)

CPUInfer[0x1d4bc3f0]: Hello
WorkerPool[0x1d55d770] 1 subpools, [numa:threads][0:72]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 72 threads
Traceback (most recent call last):
  File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 330, in <module>
    bench_moe("int8")
  File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 186, in bench_moe
    d = kt_kernel_ext.moe.tiling.get_int8()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'kt_kernel_ext.moe.tiling' has no attribute 'get_int8'
CPUInfer[0x1d4bc3f0]: Goodbye

(int4)

CPUInfer[0x446e93e0]: Hello
WorkerPool[0x4478a760] 1 subpools, [numa:threads][0:72]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 72 threads
Traceback (most recent call last):
  File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 331, in <module>
    bench_moe("int4")
  File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 200, in bench_moe
    moe = kt_kernel_ext.moe.Int4_KERNEL_MOE(config)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'kt_kernel_ext.moe' has no attribute 'Int4_KERNEL_MOE'
CPUInfer[0x446e93e0]: Goodbye

bench_moe_kernel_tiling.py --quant int8 --threads 72

Traceback (most recent call last):
  File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 232, in <module>
    main()
  File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 65, in main
    print("[tiling] default int8:", ce.moe.tiling.get_int8())
                                    ^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'kt_kernel_ext.moe.tiling' has no attribute 'get_int8'

bench_moe_kernel_tiling.py --quant int4 --threads 72

[note] Ensure your selected tiling parameters are compatible with hidden/intermediate sizes and blocking.
CPUInfer[0x20265570]: Hello
WorkerPool[0x202e95c0] 1 subpools, [numa:threads][0:72]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 72 threads
Traceback (most recent call last):
  File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 232, in <module>
    main()
  File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 121, in main
    raise RuntimeError("Int4 kernel binding 'Int4_KERNEL_MOE' not available on this platform.")
RuntimeError: Int4 kernel binding 'Int4_KERNEL_MOE' not available on this platform.
CPUInfer[0x20265570]: Goodbye

bench_moe_kml.py

CPUInfer[0x9efe610]: Hello
WorkerPool[0x9f65210] 1 subpools, [numa:threads][0:72]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 72 threads
Traceback (most recent call last):
  File "ktransformers/kt-kernel/bench/bench_moe_kml.py", line 300, in <module>
    bench_moe("int4")
  File "ktransformers/kt-kernel/bench/bench_moe_kml.py", line 178, in bench_moe
    moe = kt_kernel_ext.moe.KMLInt4_MOE(config)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'kt_kernel_ext.moe' has no attribute 'KMLInt4_MOE'. Did you mean: 'AMXInt4_MOE'?
CPUInfer[0x9efe610]: Goodbye

@yumenoshizuku commented on GitHub (Dec 18, 2025): `bench_moe.py` using `CPUINFER_PARAM = 72` ``` CPUInfer[0x3e647ee0]: Hello WorkerPool[0x3e6aeae0] 1 subpools, [numa:threads][0:72] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 72 threads Initializing MOEs: 0%| | 0/1 [00:00<?, ?it/s]TP MOE layer 0, pool: 0x3e6aeae0, expert num: 256, num_experts_per_tok: 8 Llamafile TP splitting: intermediate_size=2048, tp_count=1, QK_K=256 num_blocks=8, base_blocks=8, extra_blocks=0 TP 0: intermediate_size=2048, offset=0, blocks=8 Initializing MOEs: 100%|█████████████████████████| 1/1 [04:15<00:00, 255.15s/it]Generating data... Warming up... Warm-up: 100%|████████████████████████████████| 100/100 [00:09<00:00, 11.08it/s]Start testing... Testing: 100%|████████████████████████████| 10000/10000 [14:55<00:00, 11.17it/s]Quant mode: q4_k_m Time(s): 895.418366427999 Iteration: 10000 Time(us) per iteration: 89541.8366427999 Bandwidth: 81.6452566638661 GB/s TFLOPS: 8.058294678569345 CPUInfer[0x3e647ee0]: Goodbye ``` `bench_moe_torch.py` ``` Quant mode: fp32 Time(s): 64.82605444799992 Iteration: 10000 Time(us) per iteration: 6482.605444799992 Bandwidth: 87.34621362066713 GB/s Quant mode: fp16 Time(s): 67.00558913100394 Iteration: 10000 Time(us) per iteration: 6700.558913100394 Bandwidth: 42.25252306139347 GB/s Quant mode: bf16 Time(s): 98.33891168500122 Iteration: 10000 Time(us) per iteration: 9833.891168500122 Bandwidth: 28.78977559837904 GB/s Quant mode: qint8 Time(s): 162.6409711400047 Iteration: 10000 Time(us) per iteration: 16264.097114000471 Bandwidth: 8.703696184778938 GB/s ``` Other moe benchmarks all ran into error :( `bench_moe_amx_k.py` (all quantization modes) ``` worker_config.subpool_count = 1 worker_config.subpool_numa_map = [0] worker_config.subpool_thread_count = [72] CPUINFER_PARAM = 72 ``` ``` CPUInfer[0x28b5c670]: Hello WorkerPool[0x2eeddef0] 1 subpools, [numa:threads][0:72] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 72 threads TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8 From BF16 Online Quantization. TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8 From BF16 Online Quantization. TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8 From BF16 Online Quantization. TP MOE layer 0, pool: 0x2eeddef0, expert num: 256, num_experts_per_tok: 8 From BF16 Online Quantization. Warm-up: 0%| | 0/1000 [00:00<?, ?it/s] Segmentation fault ``` `bench_moe_amx.py` (all quantization modes) ``` CPUInfer[0x45febda0]: Hello WorkerPool[0x45fef920] 1 subpools, [numa:threads][0:72] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 72 threads TP MOE layer 0, pool: 0x45fef920, expert num: 16, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 From BF16 online quant from bf16 TP MOE layer 0, pool: 0x45fef920, expert num: 16, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 From BF16 online quant from bf16 Warm-up: 0%| | 0/1000 [00:00<?, ?it/s] Segmentation fault ``` `bench_moe_kernel.py` (int8) ``` CPUInfer[0x1d4bc3f0]: Hello WorkerPool[0x1d55d770] 1 subpools, [numa:threads][0:72] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 72 threads Traceback (most recent call last): File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 330, in <module> bench_moe("int8") File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 186, in bench_moe d = kt_kernel_ext.moe.tiling.get_int8() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'kt_kernel_ext.moe.tiling' has no attribute 'get_int8' CPUInfer[0x1d4bc3f0]: Goodbye ``` (int4) ``` CPUInfer[0x446e93e0]: Hello WorkerPool[0x4478a760] 1 subpools, [numa:threads][0:72] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 72 threads Traceback (most recent call last): File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 331, in <module> bench_moe("int4") File "ktransformers/kt-kernel/bench/bench_moe_kernel.py", line 200, in bench_moe moe = kt_kernel_ext.moe.Int4_KERNEL_MOE(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'kt_kernel_ext.moe' has no attribute 'Int4_KERNEL_MOE' CPUInfer[0x446e93e0]: Goodbye ``` `bench_moe_kernel_tiling.py --quant int8 --threads 72` ``` Traceback (most recent call last): File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 232, in <module> main() File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 65, in main print("[tiling] default int8:", ce.moe.tiling.get_int8()) ^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'kt_kernel_ext.moe.tiling' has no attribute 'get_int8' ``` `bench_moe_kernel_tiling.py --quant int4 --threads 72` ``` [note] Ensure your selected tiling parameters are compatible with hidden/intermediate sizes and blocking. CPUInfer[0x20265570]: Hello WorkerPool[0x202e95c0] 1 subpools, [numa:threads][0:72] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 72 threads Traceback (most recent call last): File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 232, in <module> main() File "ktransformers/kt-kernel/bench/bench_moe_kernel_tiling.py", line 121, in main raise RuntimeError("Int4 kernel binding 'Int4_KERNEL_MOE' not available on this platform.") RuntimeError: Int4 kernel binding 'Int4_KERNEL_MOE' not available on this platform. CPUInfer[0x20265570]: Goodbye ``` `bench_moe_kml.py` ``` CPUInfer[0x9efe610]: Hello WorkerPool[0x9f65210] 1 subpools, [numa:threads][0:72] ===========In NumaPool============ In Numa Worker Pool at NUMA 0, 72 threads Traceback (most recent call last): File "ktransformers/kt-kernel/bench/bench_moe_kml.py", line 300, in <module> bench_moe("int4") File "ktransformers/kt-kernel/bench/bench_moe_kml.py", line 178, in bench_moe moe = kt_kernel_ext.moe.KMLInt4_MOE(config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: module 'kt_kernel_ext.moe' has no attribute 'KMLInt4_MOE'. Did you mean: 'AMXInt4_MOE'? CPUInfer[0x9efe610]: Goodbye ```

bsakaguchi commented

Author

Owner

@ErvinXie commented on GitHub (Dec 19, 2025):

@yumenoshizuku try this param in bench_moe_amx.py, change CPUINFER_PARAM to your number of cores. Segfault is not an expected results. The expected bandwidth is about 70% of total memory bandwidth.

# 测试参数设置
expert_num = 256
hidden_size = 7168
intermediate_size = 2048
max_len = 25600
num_experts_per_tok = 8
layer_num = 5

qlen = 1
warm_up_iter = 1000
test_iter = 2000
physical_to_logical_map = torch.tensor(data=range(expert_num), device="cpu", dtype=torch.int64).contiguous()

# 将 CPUInfer 参数设为变量
# CPUINFER_PARAM = 257
# CPUInfer = kt_kernel_ext.CPUInfer(CPUINFER_PARAM)

worker_config = kt_kernel_ext.WorkerPoolConfig()
worker_config.subpool_count = 2
worker_config.subpool_numa_map = [0, 1]
worker_config.subpool_thread_count = [80, 80]
CPUINFER_PARAM = 96
CPUInfer = kt_kernel_ext.CPUInfer(worker_config)

@ErvinXie commented on GitHub (Dec 19, 2025): @yumenoshizuku try this param in bench_moe_amx.py, change CPUINFER_PARAM to your number of cores. Segfault is not an expected results. The expected bandwidth is about 70% of total memory bandwidth. ``` # 测试参数设置 expert_num = 256 hidden_size = 7168 intermediate_size = 2048 max_len = 25600 num_experts_per_tok = 8 layer_num = 5 qlen = 1 warm_up_iter = 1000 test_iter = 2000 physical_to_logical_map = torch.tensor(data=range(expert_num), device="cpu", dtype=torch.int64).contiguous() # 将 CPUInfer 参数设为变量 # CPUINFER_PARAM = 257 # CPUInfer = kt_kernel_ext.CPUInfer(CPUINFER_PARAM) worker_config = kt_kernel_ext.WorkerPoolConfig() worker_config.subpool_count = 2 worker_config.subpool_numa_map = [0, 1] worker_config.subpool_thread_count = [80, 80] CPUINFER_PARAM = 96 CPUInfer = kt_kernel_ext.CPUInfer(worker_config) ```

bsakaguchi commented

Author

Owner

@ErvinXie commented on GitHub (Dec 19, 2025):

Maybe you did not install the AMX version?

@ErvinXie commented on GitHub (Dec 19, 2025): Maybe you did not install the AMX version?

bsakaguchi commented

Author

Owner

@yumenoshizuku commented on GitHub (Dec 19, 2025):

Still shows segmentation fault when at the warm-up stage using the following config:

worker_config.subpool_count = 1
worker_config.subpool_numa_map = [0]
worker_config.subpool_thread_count = [72]
CPUINFER_PARAM = 72

Looks like AMX is supported:

>>> import kt_kernel
>>> print(f"CPU variant: {kt_kernel.__cpu_variant__}")
CPU variant: amx

@yumenoshizuku commented on GitHub (Dec 19, 2025): Still shows segmentation fault when at the warm-up stage using the following config: ``` worker_config.subpool_count = 1 worker_config.subpool_numa_map = [0] worker_config.subpool_thread_count = [72] CPUINFER_PARAM = 72 ``` Looks like AMX is supported: ``` >>> import kt_kernel >>> print(f"CPU variant: {kt_kernel.__cpu_variant__}") CPU variant: amx ```

bsakaguchi commented

Author

Owner

@ErvinXie commented on GitHub (Dec 19, 2025):

This is not ok, did you test your memory bandwidth using STREAM? We currently do not have intel XEON6, you can provide us your environment if you don't mind. We can ssh to your environment and fix this problem.

@ErvinXie commented on GitHub (Dec 19, 2025): This is not ok, did you test your memory bandwidth using STREAM? We currently do not have intel XEON6, you can provide us your environment if you don't mind. We can ssh to your environment and fix this problem.

bsakaguchi commented

Author

Owner

@yumenoshizuku commented on GitHub (Dec 19, 2025):

Here are the STREAM benchmark results:
Multi-thread:

-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 1000000000 (elements), Offset = 0 (elements)
Memory per array = 7629.4 MiB (= 7.5 GiB).
Total memory required = 22888.2 MiB (= 22.4 GiB).
Each kernel will be executed 10 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 72
Number of Threads counted = 72
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 80834 microseconds.
   (= 80834 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:          292622.5     0.061026     0.054678     0.088830
Scale:         288672.6     0.056092     0.055426     0.057651
Add:           344976.0     0.073514     0.069570     0.093317
Triad:         343894.1     0.077766     0.069789     0.100870
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------

Single-thread:

-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 1000000000 (elements), Offset = 0 (elements)
Memory per array = 7629.4 MiB (= 7.5 GiB).
Total memory required = 22888.2 MiB (= 22.4 GiB).
Each kernel will be executed 10 times.
 The *best* time for each kernel (excluding the first iteration)
 will be used to compute the reported bandwidth.
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 2559092 microseconds.
   (= 2559092 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           13756.2     1.173411     1.163113     1.184663
Scale:          13636.2     1.183886     1.173351     1.197547
Add:            13633.5     1.776060     1.760373     1.779994
Triad:          13655.0     1.761865     1.757592     1.780411
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------

Regarding the hardware setup, we are currently working on balancing the RAM between the two NUMA nodes. I will let you know once that is ready, as it might affect the memory bandwidth utilization.
Regarding environment access: It would be great if you can look into this. Would it work if we host an AnyDesk session? I can open a terminal with SSH access inside the remote desktop session for you to operate.

@yumenoshizuku commented on GitHub (Dec 19, 2025): Here are the STREAM benchmark results: Multi-thread: ``` ------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 1000000000 (elements), Offset = 0 (elements) Memory per array = 7629.4 MiB (= 7.5 GiB). Total memory required = 22888.2 MiB (= 22.4 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Number of Threads requested = 72 Number of Threads counted = 72 ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 80834 microseconds. (= 80834 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 292622.5 0.061026 0.054678 0.088830 Scale: 288672.6 0.056092 0.055426 0.057651 Add: 344976.0 0.073514 0.069570 0.093317 Triad: 343894.1 0.077766 0.069789 0.100870 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays ------------------------------------------------------------- ``` Single-thread: ``` ------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 1000000000 (elements), Offset = 0 (elements) Memory per array = 7629.4 MiB (= 7.5 GiB). Total memory required = 22888.2 MiB (= 22.4 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 2559092 microseconds. (= 2559092 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 13756.2 1.173411 1.163113 1.184663 Scale: 13636.2 1.183886 1.173351 1.197547 Add: 13633.5 1.776060 1.760373 1.779994 Triad: 13655.0 1.761865 1.757592 1.780411 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays ------------------------------------------------------------- ``` Regarding the hardware setup, we are currently working on balancing the RAM between the two NUMA nodes. I will let you know once that is ready, as it might affect the memory bandwidth utilization. Regarding environment access: It would be great if you can look into this. Would it work if we host an _AnyDesk_ session? I can open a terminal with SSH access inside the remote desktop session for you to operate.

bsakaguchi commented

Author

Owner

@ErvinXie commented on GitHub (Dec 20, 2025):

@yumenoshizuku It is estimated that your env should achieve at least 15 token/s, with 350GB/s for one single numa and 700GB/s for 2 numas. You can email me ervinxie@qq.com your remote desktop, we will find a time to operate on your env next week.

@ErvinXie commented on GitHub (Dec 20, 2025): @yumenoshizuku It is estimated that your env should achieve at least 15 token/s, with 350GB/s for one single numa and 700GB/s for 2 numas. You can email me ervinxie@qq.com your remote desktop, we will find a time to operate on your env next week.

bsakaguchi commented

Author

Owner

@shrould8888 commented on GitHub (Dec 20, 2025):

I got near 15 t/s with single xeon 5 64 cores cpu + 4090 24G x 2. My config only uses 1 NUMA node

@shrould8888 commented on GitHub (Dec 20, 2025): I got near 15 t/s with single xeon 5 64 cores cpu + 4090 24G x 2. My config only uses 1 NUMA node

bsakaguchi commented

Author

Owner

@mtcl commented on GitHub (Dec 22, 2025):

Do you think i can run this model on my configuration?

System Specifications

OS Overview

OS: Ubuntu 24.04.3 LTS
Kernel: Linux 6.14.0-37-generic

Motherboard Specifications

Model: ASUS Pro WS W790E-SAGE SE

Platform: Intel W790 Chipset
Socket: LGA4677 (supports Intel Xeon W-3000 series)
Memory Slots: 8x DDR5 DIMM slots
Maximum Memory Capacity: Supports up to 2TB DDR5
Memory Speed: Support for DDR5-4800+ (OC capable)
PCIe Slots: 7 PCIe 5.0 x16 slots for dual GPU setup
Form Factor: Extended ATX

CPU Specifications

Processor: Intel Xeon w9-3495X

Cores: 56 physical cores / 112 threads
Base Frequency: 800 MHz
Max Frequency: 4.8 GHz
AVX-512 Support: Full support
AMX (Advanced Matrix Extensions): Full support
- AMX BF16: 112 cores support (Brain Float16 matrix ops)
- AMX INT8: 112 cores support (INT8 matrix ops)
- AMX Tile: 112 cores support (Tile matrix unit)
NUMA Nodes: 1

GPU Specifications

Graphics Cards: 2x NVIDIA RTX PRO 6000 Black Edition

VRAM per GPU: 96 GB
Driver Version: 580.95.05
CUDA Version: 13.0
Power Consumption: 600W max each

Memory Configuration

Total RAM: 512 GB
Memory Type: DDR5 (8x DIMM slots populated)
Memory Speed: DDR5-4800+ (OC capable)
OC Supported: 6000 OC Stable
Swap: 0 GB (disabled)
Memory Architecture: 8-channel DDR5 configuration
Available for Inference: 503 GB total capacity for model loading

@mtcl commented on GitHub (Dec 22, 2025): Do you think i can run this model on my configuration? # System Specifications ## OS Overview **OS:** Ubuntu 24.04.3 LTS **Kernel:** Linux 6.14.0-37-generic --- ## Motherboard Specifications **Model:** ASUS Pro WS W790E-SAGE SE - **Platform:** Intel W790 Chipset - **Socket:** LGA4677 (supports Intel Xeon W-3000 series) - **Memory Slots:** 8x DDR5 DIMM slots - **Maximum Memory Capacity:** Supports up to 2TB DDR5 - **Memory Speed:** Support for DDR5-4800+ (OC capable) - **PCIe Slots:** 7 PCIe 5.0 x16 slots for dual GPU setup - **Form Factor:** Extended ATX --- ## CPU Specifications **Processor:** Intel Xeon w9-3495X - **Cores:** 56 physical cores / 112 threads - **Base Frequency:** 800 MHz - **Max Frequency:** 4.8 GHz - **AVX-512 Support:** Full support - **AMX (Advanced Matrix Extensions):** Full support - AMX BF16: 112 cores support (Brain Float16 matrix ops) - AMX INT8: 112 cores support (INT8 matrix ops) - AMX Tile: 112 cores support (Tile matrix unit) - **NUMA Nodes:** 1 --- ## GPU Specifications **Graphics Cards:** 2x NVIDIA RTX PRO 6000 Black Edition - **VRAM per GPU:** 96 GB - **Driver Version:** 580.95.05 - **CUDA Version:** 13.0 - **Power Consumption:** 600W max each --- ## Memory Configuration **Total RAM:** 512 GB **Memory Type:** DDR5 (8x DIMM slots populated) **Memory Speed:** DDR5-4800+ (OC capable) **OC Supported:** 6000 OC Stable **Swap:** 0 GB (disabled) **Memory Architecture:** 8-channel DDR5 configuration **Available for Inference:** 503 GB total capacity for model loading ---

bsakaguchi commented

Author

Owner

@ErvinXie commented on GitHub (Dec 23, 2025):

@mtcl it is not enough for kimi k2 thinking. But we are going to release our support for Minimax m2 later, then you can try it.

@ErvinXie commented on GitHub (Dec 23, 2025): @mtcl it is not enough for kimi k2 thinking. But we are going to release our support for Minimax m2 later, then you can try it.

bsakaguchi commented

Author

Owner

@mtcl commented on GitHub (Dec 23, 2025):

@mtcl it is not enough for kimi k2 thinking. But we are going to release our support for Minimax m2 later, then you can try it.

Can I run glm 4.6 or 4.7 with CPU GPU hybrid? I'm able to do it in llama.cp right now with glm 4.6 GG UF. But I would like to run it using SGlang and kt-kernel/ktransformers. When you had your independent ktransformers I used to run deepseek r1 using that. I bought my AMX enabled Intel Xeon processor because of ktransformers.

@mtcl commented on GitHub (Dec 23, 2025): > @mtcl it is not enough for kimi k2 thinking. But we are going to release our support for Minimax m2 later, then you can try it. Can I run glm 4.6 or 4.7 with CPU GPU hybrid? I'm able to do it in llama.cp right now with glm 4.6 GG UF. But I would like to run it using SGlang and kt-kernel/ktransformers. When you had your independent ktransformers I used to run deepseek r1 using that. I bought my AMX enabled Intel Xeon processor because of ktransformers.

bsakaguchi commented

Author

Owner

@yumenoshizuku commented on GitHub (Dec 24, 2025):

Update: After fully populating both CPUs with 12 * 96 GB MRDIMM-8800 per socket, decoding performance increased a little bit:

Hyper-threading ON: ~12.8 tokens/s
Hyper-threading OFF: ~11 tokens/s

I also tried different settings of SNC in the BIOS:

~~SNC Disabled: The speed drops to 0.5 tokens/s.~~
SNC Enabled: This results in 6 NUMA nodes total. But only 4 of them can be utilized due to the MoE intermediate size being 2048 (unable to evenly distribute across 6 nodes).

Are there specific BIOS options that I can change to optimize the memory topology?
For reference, the BIOS options for our server can be found in the user manual.

@yumenoshizuku commented on GitHub (Dec 24, 2025): Update: After fully populating both CPUs with 12 * 96 GB MRDIMM-8800 per socket, decoding performance increased a little bit: Hyper-threading **ON**: ~12.8 tokens/s Hyper-threading **OFF**: ~11 tokens/s I also tried different settings of SNC in the BIOS: ~~SNC **Disabled**: The speed drops to **0.5** tokens/s.~~ SNC **Enabled**: This results in **6 NUMA nodes** total. But only 4 of them can be utilized due to the MoE intermediate size being 2048 (unable to evenly distribute across 6 nodes). Are there specific BIOS options that I can change to optimize the memory topology? For reference, the BIOS options for our server can be found in the [user manual](https://www.gigabyte.com/us/Enterprise/Rack-Server/XV24-AX0-AAJ1#Support-Manual).

bsakaguchi commented

Author

Owner

@ErvinXie commented on GitHub (Dec 24, 2025):

We recommend using 2 NUMA, multiple NUMA is not friendly to tensor parallel on cpu. How about the total memory bandwidth now?

@ErvinXie commented on GitHub (Dec 24, 2025): We recommend using 2 NUMA, multiple NUMA is not friendly to tensor parallel on cpu. How about the total memory bandwidth now?

bsakaguchi commented

Author

Owner

@yumenoshizuku commented on GitHub (Dec 24, 2025):

The 6-node topology is due to SNC3 (Sub-NUMA Clustering), which is default on Xeon 6 (also discussed in Issue #812).

I tested disabling SNC to achieve 2 NUMA nodes (HEX mode). While raw bandwidth more than doubled, inference speed dropped catastrophically:

Mode	NUMA Nodes	STREAM Triad	Decode Speed
SNC Enabled (Default)	6	~380 GB/s	12.8 t/s
SNC Disabled (HEX)	2	~840 GB/s	0.5 t/s

The anomaly: In HEX mode, using both sockets yields 0.5 t/s, but restricting it to one socket yields 9.5 t/s. This suggests a severe bottleneck/synchronization issue between the two HEX nodes, despite the much higher bandwidth. Is there a specific configuration to make the 2-node HEX mode work correctly?

Update & Correction:

I re-tested the SNC Disabled (HEX mode/2 NUMA) configuration. The 0.5 t/s was caused by disabling Hyper-Threading and using all physical cores.

With Hyper-Threading ON, the 2-NUMA topology performed better because it allows utilizing all cores (unlike SNC Enabled, which limited usage to 96/144 cores):
SNC Enabled (4 out of 6 NUMA): ~12.8 t/s (96 threads)
SNC Disabled (2 NUMA): 14.5 t/s (144 threads)

However, I attempted to upgrade to v0.5.0 to see if performance could be improved further, but it now crashes on launch.
I attempted setting --mem-fraction-static 0.8, but the error persists.
It appears the kt cli does not expose --cuda-graph-max-bs for adjustment.

v0.5.0 Crash Log:

[2025-12-24 23:43:06] Using KV cache dtype: torch.bfloat16
[2025-12-24 23:43:06] The available memory for KV cache is 40.78 GB.
[2025-12-24 23:43:06] KV Cache is allocated. #tokens: 100000, KV size: 6.54 GB
[2025-12-24 23:43:06] Memory pool end. avail mem=53.11 GB
[2025-12-24 23:43:06] Capture cuda graph begin. This can take up to several minutes. avail mem=52.72 GB
[2025-12-24 23:43:06] Capture cuda graph bs [1, 2, 4, 8, 12, 16]
Capturing batches (bs=16 avail_mem=52.70 GB):   0%|              | 0/6 [00:02<?, ?it/s]
[2025-12-24 23:43:09] Scheduler hit an exception: Traceback (most recent call last):
  File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 358, in __init__
    self.capture()
  File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 513, in capture
    _capture_one_stream()
  File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 500, in _capture_one_stream
    ) = self.capture_one_batch_size(bs, forward, stream_idx)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 719, in capture_one_batch_size
    run_once()
  File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 706, in run_once
    logits_output_or_pp_proxy_tensors = forward(
                                        ^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 3462, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 3213, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 2925, in forward
    hidden_states = self.mlp(
                    ^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 805, in forward
    return self.forward_normal_dual_stream(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 839, in forward_normal_dual_stream
    final_hidden_states = self.experts(hidden_states, topk_output)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 908, in forward
    return self.forward_impl(hidden_states, topk_output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 927, in forward_impl
    combine_input = self.run_moe_core(
                    ^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 948, in run_moe_core
    return self.quant_method.apply(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 980, in apply
    gpu_combine_input = self.gpu_method.apply(layer, masked_dispatch_output)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors_moe.py", line 1142, in apply
    output = torch.ops.sglang.fused_marlin_moe(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_marlin_moe.py", line 211, in fused_marlin_moe
    moe_sum_reduce(
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/sgl_kernel/moe.py", line 88, in moe_sum_reduce
    torch.ops.sgl_kernel.moe_sum_reduce.default(
  File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/_ops.py", line 841, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: moe_sum_reduce CUDA kernel (small-token) launch failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/kt050/sglang/python/sglang/srt/managers/scheduler.py", line 2833, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/managers/scheduler.py", line 314, in __init__
    self.init_model_worker()
  File "/kt050/sglang/python/sglang/srt/managers/scheduler.py", line 452, in init_model_worker
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/managers/tp_worker.py", line 253, in __init__
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/model_executor/model_runner.py", line 385, in __init__
    self.initialize(min_per_gpu_memory)
  File "/kt050/sglang/python/sglang/srt/model_executor/model_runner.py", line 584, in initialize
    self.init_device_graphs()
  File "/kt050/sglang/python/sglang/srt/model_executor/model_runner.py", line 2670, in init_device_graphs
    self.graph_runner = graph_runners[self.device](self)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 360, in __init__
    raise Exception(
Exception: Capture cuda graph failed: moe_sum_reduce CUDA kernel (small-token) launch failed
Possible solutions:
1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose


[2025-12-24 23:43:09] Received sigquit from a child process. It usually means the child failed.

@yumenoshizuku commented on GitHub (Dec 24, 2025): The 6-node topology is due to SNC3 (Sub-NUMA Clustering), which is default on Xeon 6 (also discussed in [Issue #812](https://github.com/kvcache-ai/ktransformers/issues/812#issuecomment-2707030445)). I tested disabling SNC to achieve 2 NUMA nodes (HEX mode). While raw bandwidth more than doubled, inference speed dropped catastrophically: | Mode | NUMA Nodes | STREAM Triad | Decode Speed | | :--- | :--- | :--- | :--- | | **SNC Enabled** (Default) | 6 | ~380 GB/s | **12.8 t/s** | | **SNC Disabled** (HEX) | 2 | **~840 GB/s** | 0.5 t/s | The anomaly: In HEX mode, using **both** sockets yields 0.5 t/s, but restricting it to **one** socket yields 9.5 t/s. This suggests a severe bottleneck/synchronization issue between the two HEX nodes, despite the much higher bandwidth. Is there a specific configuration to make the 2-node HEX mode work correctly? --- Update & Correction: I re-tested the SNC Disabled (HEX mode/2 NUMA) configuration. The 0.5 t/s was caused by disabling Hyper-Threading and using all physical cores. With Hyper-Threading **ON**, the 2-NUMA topology performed better because it allows utilizing all cores (unlike SNC Enabled, which limited usage to 96/144 cores): SNC Enabled (4 out of 6 NUMA): ~12.8 t/s (96 threads) SNC Disabled (2 NUMA): 14.5 t/s (144 threads) However, I attempted to upgrade to v0.5.0 to see if performance could be improved further, but it now crashes on launch. I attempted setting `--mem-fraction-static 0.8`, but the error persists. It appears the kt cli does not expose `--cuda-graph-max-bs` for adjustment. v0.5.0 Crash Log: ``` [2025-12-24 23:43:06] Using KV cache dtype: torch.bfloat16 [2025-12-24 23:43:06] The available memory for KV cache is 40.78 GB. [2025-12-24 23:43:06] KV Cache is allocated. #tokens: 100000, KV size: 6.54 GB [2025-12-24 23:43:06] Memory pool end. avail mem=53.11 GB [2025-12-24 23:43:06] Capture cuda graph begin. This can take up to several minutes. avail mem=52.72 GB [2025-12-24 23:43:06] Capture cuda graph bs [1, 2, 4, 8, 12, 16] Capturing batches (bs=16 avail_mem=52.70 GB): 0%| | 0/6 [00:02<?, ?it/s] [2025-12-24 23:43:09] Scheduler hit an exception: Traceback (most recent call last): File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 358, in __init__ self.capture() File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 513, in capture _capture_one_stream() File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 500, in _capture_one_stream ) = self.capture_one_batch_size(bs, forward, stream_idx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 719, in capture_one_batch_size run_once() File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 706, in run_once logits_output_or_pp_proxy_tensors = forward( ^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 3462, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 3213, in forward hidden_states, residual = layer( ^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 2925, in forward hidden_states = self.mlp( ^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 805, in forward return self.forward_normal_dual_stream( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/models/deepseek_v2.py", line 839, in forward_normal_dual_stream final_hidden_states = self.experts(hidden_states, topk_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 908, in forward return self.forward_impl(hidden_states, topk_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 927, in forward_impl combine_input = self.run_moe_core( ^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 948, in run_moe_core return self.quant_method.apply( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 980, in apply gpu_combine_input = self.gpu_method.apply(layer, masked_dispatch_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors_moe.py", line 1142, in apply output = torch.ops.sglang.fused_marlin_moe( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/_ops.py", line 1255, in __call__ return self._op(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_marlin_moe.py", line 211, in fused_marlin_moe moe_sum_reduce( File "/miniconda3/envs/kt050/lib/python3.11/site-packages/sgl_kernel/moe.py", line 88, in moe_sum_reduce torch.ops.sgl_kernel.moe_sum_reduce.default( File "/miniconda3/envs/kt050/lib/python3.11/site-packages/torch/_ops.py", line 841, in __call__ return self._op(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: moe_sum_reduce CUDA kernel (small-token) launch failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/kt050/sglang/python/sglang/srt/managers/scheduler.py", line 2833, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/managers/scheduler.py", line 314, in __init__ self.init_model_worker() File "/kt050/sglang/python/sglang/srt/managers/scheduler.py", line 452, in init_model_worker self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/managers/tp_worker.py", line 253, in __init__ self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/model_executor/model_runner.py", line 385, in __init__ self.initialize(min_per_gpu_memory) File "/kt050/sglang/python/sglang/srt/model_executor/model_runner.py", line 584, in initialize self.init_device_graphs() File "/kt050/sglang/python/sglang/srt/model_executor/model_runner.py", line 2670, in init_device_graphs self.graph_runner = graph_runners[self.device](self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/kt050/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 360, in __init__ raise Exception( Exception: Capture cuda graph failed: moe_sum_reduce CUDA kernel (small-token) launch failed Possible solutions: 1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7) 2. set --cuda-graph-max-bs to a smaller value (e.g., 16) 3. disable torch compile by not using --enable-torch-compile 4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss) Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose [2025-12-24 23:43:09] Received sigquit from a child process. It usually means the child failed. ```

bsakaguchi commented

Author

Owner

@yumenoshizuku commented on GitHub (Dec 25, 2025):

I did more testing with MiniMax-M2 using v0.5.0. Here are the detailed performance results under different BIOS configurations (SNC Enabled vs Disabled) and Hyper-Threading settings.

Summary:

Best Performance: 33 t/s (SNC Enabled, HT On, 138-144 threads).
Observation: Performance drops dramatically when saturating all cores (144) with HT Off. Leaving a few cores free or enabling HT stabilizes performance.

Detailed Benchmarks

1. SNC Enabled (6 NUMA Nodes)

Hyper-Threading: OFF
STREAM Triad: 658,201.1 MB/s

Arguments	Speed	Note
`--cpu-threads 144 --numa-nodes 6`	0.01 t/s	Saturation?
`--cpu-threads 142 --numa-nodes 6`	10 t/s
`--cpu-threads 140 --numa-nodes 6`	12-24 t/s	Unstable
`--cpu-threads 138 --numa-nodes 6`	25 t/s	Peak for HT Off
`--cpu-threads 136 --numa-nodes 6`	25 t/s
`--cpu-threads 132 --numa-nodes 6`	19 t/s
`--cpu-threads 96 --numa-nodes 6`	19 t/s
`--cpu-threads 96 --numa-nodes 4`	20 t/s
`--cpu-threads 72 --numa-nodes 3`	19 t/s

Hyper-Threading: ON
STREAM Triad: 648,842.0 MB/s

Arguments	Speed	Note
`--cpu-threads 150 --numa-nodes 6`	30 t/s
`--cpu-threads 144 --numa-nodes 6`	33 t/s	Overall Peak
`--cpu-threads 138 --numa-nodes 6`	33 t/s	Overall Peak
`--cpu-threads 96 --numa-nodes 6`	18 t/s

2. SNC Disabled (2 NUMA Nodes / HEX Mode)

Higher bandwidth, lower decode speed.

Hyper-Threading: ON
STREAM Triad: 770,911.4 MB/s

Arguments	Speed	Note
`--cpu-threads 150 --numa-nodes 2`	26 t/s
`--cpu-threads 146 --numa-nodes 2`	28 t/s
`--cpu-threads 144 --numa-nodes 2`	30 t/s	Peak for SNC Disabled
`--cpu-threads 142 --numa-nodes 2`	30 t/s
`--cpu-threads 140 --numa-nodes 2`	26 t/s
`--cpu-threads 138 --numa-nodes 2`	24 t/s

Hyper-Threading: OFF
STREAM Triad: 695,106.9 MB/s

Arguments	Speed	Note
`--cpu-threads 144 --numa-nodes 2`	0.4 t/s	Saturation>
`--cpu-threads 142 --numa-nodes 2`	9 t/s
`--cpu-threads 140 --numa-nodes 2`	18 t/s
`--cpu-threads 138 --numa-nodes 2`	23 t/s	Peak for HT Off
`--cpu-threads 132 --numa-nodes 2`	17 t/s
`--cpu-threads 96 --numa-nodes 2`	15 t/s
`--cpu-threads 72 --numa-nodes 1`	17 t/s

However, when trying to run DeepSeek-V3.2, it only outputs garbled characters.

@yumenoshizuku commented on GitHub (Dec 25, 2025): I did more testing with MiniMax-M2 using v0.5.0. Here are the detailed performance results under different BIOS configurations (SNC Enabled vs Disabled) and Hyper-Threading settings. **Summary:** * Best Performance: **33 t/s** (SNC Enabled, HT On, 138-144 threads). * Observation: Performance drops dramatically when saturating all cores (144) with HT Off. Leaving a few cores free or enabling HT stabilizes performance. ### Detailed Benchmarks #### 1. SNC Enabled (6 NUMA Nodes) Hyper-Threading: **OFF** STREAM Triad: 658,201.1 MB/s | Arguments | Speed | Note | | :--- | :--- | :--- | | `--cpu-threads 144 --numa-nodes 6` | 0.01 t/s | Saturation? | | `--cpu-threads 142 --numa-nodes 6` | 10 t/s | | | `--cpu-threads 140 --numa-nodes 6` | 12-24 t/s | Unstable | | `--cpu-threads 138 --numa-nodes 6` | **25 t/s** | Peak for HT Off | | `--cpu-threads 136 --numa-nodes 6` | **25 t/s** | | | `--cpu-threads 132 --numa-nodes 6` | 19 t/s | | | `--cpu-threads 96 --numa-nodes 6` | 19 t/s | | | `--cpu-threads 96 --numa-nodes 4` | 20 t/s | | | `--cpu-threads 72 --numa-nodes 3` | 19 t/s | | Hyper-Threading: **ON** STREAM Triad: 648,842.0 MB/s | Arguments | Speed | Note | | :--- | :--- | :--- | | `--cpu-threads 150 --numa-nodes 6` | 30 t/s | | | `--cpu-threads 144 --numa-nodes 6` | **33 t/s** | Overall Peak | | `--cpu-threads 138 --numa-nodes 6` | **33 t/s** | Overall Peak | | `--cpu-threads 96 --numa-nodes 6` | 18 t/s | | --- #### 2. SNC Disabled (2 NUMA Nodes / HEX Mode) Higher bandwidth, lower decode speed. Hyper-Threading: **ON** STREAM Triad: 770,911.4 MB/s | Arguments | Speed | Note | | :--- | :--- | :--- | | `--cpu-threads 150 --numa-nodes 2` | 26 t/s | | | `--cpu-threads 146 --numa-nodes 2` | 28 t/s | | | `--cpu-threads 144 --numa-nodes 2` | **30 t/s** | Peak for SNC Disabled | | `--cpu-threads 142 --numa-nodes 2` | **30 t/s** | | | `--cpu-threads 140 --numa-nodes 2` | 26 t/s | | | `--cpu-threads 138 --numa-nodes 2` | 24 t/s | | Hyper-Threading: **OFF** STREAM Triad: 695,106.9 MB/s | Arguments | Speed | Note | | :--- | :--- | :--- | | `--cpu-threads 144 --numa-nodes 2` | **0.4 t/s** | Saturation> | | `--cpu-threads 142 --numa-nodes 2` | 9 t/s | | | `--cpu-threads 140 --numa-nodes 2` | 18 t/s | | | `--cpu-threads 138 --numa-nodes 2` | **23 t/s** | Peak for HT Off | | `--cpu-threads 132 --numa-nodes 2` | 17 t/s | | | `--cpu-threads 96 --numa-nodes 2` | 15 t/s | | | `--cpu-threads 72 --numa-nodes 1` | 17 t/s | | However, when trying to run **DeepSeek-V3.2**, it only outputs garbled characters.

bsakaguchi commented

Author

Owner

@ErvinXie commented on GitHub (Dec 25, 2025):

I think the results for M2 is reasonable, but for deepseek it is not expected. A community user of us provided his results for reference.

@ErvinXie commented on GitHub (Dec 25, 2025): I think the results for M2 is reasonable, but for deepseek it is not expected. A community user of us provided his results for reference. <img width="1000" height="1924" alt="Image" src="https://github.com/user-attachments/assets/c0ad2802-91c3-4828-8858-a91a891069bb" />

bsakaguchi commented

Author

Owner

@KMSorSMS commented on GitHub (Dec 25, 2025):

I did more testing with MiniMax-M2 using v0.5.0. Here are the detailed performance results under different BIOS configurations (SNC Enabled vs Disabled) and Hyper-Threading settings.

Summary:

Best Performance: 33 t/s (SNC Enabled, HT On, 138-144 threads).

Observation: Performance drops dramatically when saturating all cores (144) with HT Off. Leaving a few cores free or enabling HT stabilizes performance.

Detailed Benchmarks

1. SNC Enabled (6 NUMA Nodes)

Hyper-Threading: OFF STREAM Triad: 658,201.1 MB/s

Arguments Speed Note
--cpu-threads 144 --numa-nodes 6 0.01 t/s Saturation?
--cpu-threads 142 --numa-nodes 6 10 t/s
--cpu-threads 140 --numa-nodes 6 12-24 t/s Unstable
--cpu-threads 138 --numa-nodes 6 25 t/s Peak for HT Off
--cpu-threads 136 --numa-nodes 6 25 t/s
--cpu-threads 132 --numa-nodes 6 19 t/s
--cpu-threads 96 --numa-nodes 6 19 t/s
--cpu-threads 96 --numa-nodes 4 20 t/s
--cpu-threads 72 --numa-nodes 3 19 t/s
Hyper-Threading: ON STREAM Triad: 648,842.0 MB/s

Arguments Speed Note
--cpu-threads 150 --numa-nodes 6 30 t/s
--cpu-threads 144 --numa-nodes 6 33 t/s Overall Peak
--cpu-threads 138 --numa-nodes 6 33 t/s Overall Peak
--cpu-threads 96 --numa-nodes 6 18 t/s

2. SNC Disabled (2 NUMA Nodes / HEX Mode)

Higher bandwidth, lower decode speed.

Hyper-Threading: ON STREAM Triad: 770,911.4 MB/s

Arguments Speed Note
--cpu-threads 150 --numa-nodes 2 26 t/s
--cpu-threads 146 --numa-nodes 2 28 t/s
--cpu-threads 144 --numa-nodes 2 30 t/s Peak for SNC Disabled
--cpu-threads 142 --numa-nodes 2 30 t/s
--cpu-threads 140 --numa-nodes 2 26 t/s
--cpu-threads 138 --numa-nodes 2 24 t/s
Hyper-Threading: OFF STREAM Triad: 695,106.9 MB/s

Arguments Speed Note
--cpu-threads 144 --numa-nodes 2 0.4 t/s Saturation>
--cpu-threads 142 --numa-nodes 2 9 t/s
--cpu-threads 140 --numa-nodes 2 18 t/s
--cpu-threads 138 --numa-nodes 2 23 t/s Peak for HT Off
--cpu-threads 132 --numa-nodes 2 17 t/s
--cpu-threads 96 --numa-nodes 2 15 t/s
--cpu-threads 72 --numa-nodes 1 17 t/s
However, when trying to run DeepSeek-V3.2, it only outputs garbled characters.

You need to add a parameter : --kv-cache-dtype bf16

@KMSorSMS commented on GitHub (Dec 25, 2025): > I did more testing with MiniMax-M2 using v0.5.0. Here are the detailed performance results under different BIOS configurations (SNC Enabled vs Disabled) and Hyper-Threading settings. > > **Summary:** > > * Best Performance: **33 t/s** (SNC Enabled, HT On, 138-144 threads). > * Observation: Performance drops dramatically when saturating all cores (144) with HT Off. Leaving a few cores free or enabling HT stabilizes performance. > > ### Detailed Benchmarks > #### 1. SNC Enabled (6 NUMA Nodes) > Hyper-Threading: **OFF** STREAM Triad: 658,201.1 MB/s > > Arguments Speed Note > `--cpu-threads 144 --numa-nodes 6` 0.01 t/s Saturation? > `--cpu-threads 142 --numa-nodes 6` 10 t/s > `--cpu-threads 140 --numa-nodes 6` 12-24 t/s Unstable > `--cpu-threads 138 --numa-nodes 6` **25 t/s** Peak for HT Off > `--cpu-threads 136 --numa-nodes 6` **25 t/s** > `--cpu-threads 132 --numa-nodes 6` 19 t/s > `--cpu-threads 96 --numa-nodes 6` 19 t/s > `--cpu-threads 96 --numa-nodes 4` 20 t/s > `--cpu-threads 72 --numa-nodes 3` 19 t/s > Hyper-Threading: **ON** STREAM Triad: 648,842.0 MB/s > > Arguments Speed Note > `--cpu-threads 150 --numa-nodes 6` 30 t/s > `--cpu-threads 144 --numa-nodes 6` **33 t/s** Overall Peak > `--cpu-threads 138 --numa-nodes 6` **33 t/s** Overall Peak > `--cpu-threads 96 --numa-nodes 6` 18 t/s > #### 2. SNC Disabled (2 NUMA Nodes / HEX Mode) > Higher bandwidth, lower decode speed. > > Hyper-Threading: **ON** STREAM Triad: 770,911.4 MB/s > > Arguments Speed Note > `--cpu-threads 150 --numa-nodes 2` 26 t/s > `--cpu-threads 146 --numa-nodes 2` 28 t/s > `--cpu-threads 144 --numa-nodes 2` **30 t/s** Peak for SNC Disabled > `--cpu-threads 142 --numa-nodes 2` **30 t/s** > `--cpu-threads 140 --numa-nodes 2` 26 t/s > `--cpu-threads 138 --numa-nodes 2` 24 t/s > Hyper-Threading: **OFF** STREAM Triad: 695,106.9 MB/s > > Arguments Speed Note > `--cpu-threads 144 --numa-nodes 2` **0.4 t/s** Saturation> > `--cpu-threads 142 --numa-nodes 2` 9 t/s > `--cpu-threads 140 --numa-nodes 2` 18 t/s > `--cpu-threads 138 --numa-nodes 2` **23 t/s** Peak for HT Off > `--cpu-threads 132 --numa-nodes 2` 17 t/s > `--cpu-threads 96 --numa-nodes 2` 15 t/s > `--cpu-threads 72 --numa-nodes 1` 17 t/s > However, when trying to run **DeepSeek-V3.2**, it only outputs garbled characters. You need to add a parameter : `--kv-cache-dtype bf16`

bsakaguchi commented

Author

Owner

@yumenoshizuku commented on GitHub (Dec 25, 2025):

Thanks for the suggestion. It runs correctly now.

However, the decode speed for DeepSeek V3.2 is only ~13.5 tokens/s with SNC Disabled, which is still much lower than the result posted in the earlier comment. Does it look like the bottleneck is on the CPU?

CPU Usage: ~14600%
Volatile GPU-Util: ~45%
pqos total MBL: ~250 GB/s

With SNC Enabled, decode speed is around 11.8 t/s:

CPU Usage: ~9800%
Volatile GPU-Util: ~40%
pqos total MBL: ~335 GB/s

Launch Command:

SGL_ENABLE_JIT_DEEPGEMM=0 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python -m sglang.launch_server \
 --host 0.0.0.0 \
 --port 8000 \
 --model /mnt/m2b/models/deepseek/DeepSeek-V3.2 \
 --kt-weight-path /mnt/m2b/models/deepseek/DeepSeek-V3.2 \
 --kt-cpuinfer 144/96 \
 --kt-threadpool-count 2/4 \
 --kt-num-gpu-experts 9 \
 --kt-method FP8 \
 --kt-gpu-prefill-token-threshold 4096 \
 --attention-backend flashinfer \
 --trust-remote-code \
 --mem-fraction-static 0.8 \
 --chunked-prefill-size 32768 \
 --max-running-requests 2 \
 --max-total-tokens 65536 \
 --watchdog-timeout 3000 \
 --enable-mixed-chunk \
 --tensor-parallel-size 1 \
 --enable-p2p-check \
 --served-model-name DeepSeek-V3.2 \
 --disable-shared-experts-fusion \
 --fp8-gemm-backend triton \
 --kv-cache-dtype bf16

@yumenoshizuku commented on GitHub (Dec 25, 2025): Thanks for the suggestion. It runs correctly now. However, the decode speed for DeepSeek V3.2 is only **~13.5 tokens/s** with SNC Disabled, which is still much lower than the result posted in the earlier comment. Does it look like the bottleneck is on the CPU? * CPU Usage: ~14600% * Volatile GPU-Util: ~45% * pqos total MBL: ~250 GB/s With SNC Enabled, decode speed is around 11.8 t/s: * CPU Usage: ~9800% * Volatile GPU-Util: ~40% * pqos total MBL: ~335 GB/s Launch Command: ``` SGL_ENABLE_JIT_DEEPGEMM=0 \ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ python -m sglang.launch_server \ --host 0.0.0.0 \ --port 8000 \ --model /mnt/m2b/models/deepseek/DeepSeek-V3.2 \ --kt-weight-path /mnt/m2b/models/deepseek/DeepSeek-V3.2 \ --kt-cpuinfer 144/96 \ --kt-threadpool-count 2/4 \ --kt-num-gpu-experts 9 \ --kt-method FP8 \ --kt-gpu-prefill-token-threshold 4096 \ --attention-backend flashinfer \ --trust-remote-code \ --mem-fraction-static 0.8 \ --chunked-prefill-size 32768 \ --max-running-requests 2 \ --max-total-tokens 65536 \ --watchdog-timeout 3000 \ --enable-mixed-chunk \ --tensor-parallel-size 1 \ --enable-p2p-check \ --served-model-name DeepSeek-V3.2 \ --disable-shared-experts-fusion \ --fp8-gemm-backend triton \ --kv-cache-dtype bf16 ```

bsakaguchi commented

Author

Owner

@mtcl commented on GitHub (Dec 26, 2025):

would you know for a similar issue, even if i add or dont add --kv-cache-dtype bf16 i keep on getting garbled characters.

CUDA_VISIBLE_DEVICES=0 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0 \
python3 -m sglang.launch_server \
  --host 0.0.0.0 --port 10002 \
  --model /media/mukul/data/models/zai-org/GLM-4.7 \
  --trust-remote-code \
  --tool-call-parser glm --reasoning-parser glm45 \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 4096 \
  --enable-mixed-chunk \
  --attention-backend triton \
  --served-model-name jarvis-thinker \
  --kv-cache-dtype bf16 \
  --kt-method AMXINT4 \
  --kt-weight-path /media/mukul/data/models/zai-org/GLM-4.7-AMXINT4 \
  --kt-cpuinfer 56 \
  --kt-threadpool-count 1 \
  --kt-num-gpu-experts 4 \
  --kt-max-deferred-experts-per-token 1

@mtcl commented on GitHub (Dec 26, 2025): would you know for a similar issue, even if i add or dont add ` --kv-cache-dtype bf16` i keep on getting garbled characters. ```bash CUDA_VISIBLE_DEVICES=0 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True SGLANG_ENABLE_JIT_DEEPGEMM=0 \ python3 -m sglang.launch_server \ --host 0.0.0.0 --port 10002 \ --model /media/mukul/data/models/zai-org/GLM-4.7 \ --trust-remote-code \ --tool-call-parser glm --reasoning-parser glm45 \ --mem-fraction-static 0.85 \ --chunked-prefill-size 4096 \ --enable-mixed-chunk \ --attention-backend triton \ --served-model-name jarvis-thinker \ --kv-cache-dtype bf16 \ --kt-method AMXINT4 \ --kt-weight-path /media/mukul/data/models/zai-org/GLM-4.7-AMXINT4 \ --kt-cpuinfer 56 \ --kt-threadpool-count 1 \ --kt-num-gpu-experts 4 \ --kt-max-deferred-experts-per-token 1 ```

bsakaguchi commented

Author

Owner

@mtcl commented on GitHub (Dec 26, 2025):

So I was able to solve the garbled test issue with below command, but I get this new error now on a bigger prompt.

SGL_ENABLE_JIT_DEEPGEMM=0 \
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
python -m sglang.launch_server \
	--host 0.0.0.0 \
	--port 10002 \
	--model /media/mukul/data/models/zai-org/GLM-4.7 \
	--kt-weight-path /media/mukul/data/models/zai-org/GLM-4.7-AMXINT4 \
	--kt-cpuinfer 56 \
	--kt-threadpool-count 1 \
	--kt-num-gpu-experts 4 \
	--kt-method AMXINT4 \
	--attention-backend flashinfer \
	--trust-remote-code \
	--tool-call-parser glm \
	--reasoning-parser glm45 \
	--mem-fraction-static 0.8 \
	--chunked-prefill-size 32768 \
	--max-running-requests 2 \
	--max-total-tokens 65536 \
	--watchdog-timeout 3000 \
	--enable-mixed-chunk \
	--tensor-parallel-size 1 \
	--enable-p2p-check \
	--served-model-name jarvis-thinker \
	--disable-shared-experts-fusion \
	--fp8-gemm-backend triton \
	--kv-cache-dtype bf16

[2025-12-25 20:49:17] INFO:     127.0.0.1:50774 - "GET /model_info HTTP/1.1" 200 OK
[2025-12-25 20:49:17] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-12-25 20:49:19] INFO:     127.0.0.1:50784 - "POST /generate HTTP/1.1" 200 OK
[2025-12-25 20:49:19] The server is fired up and ready to roll!
[2025-12-25 20:49:24] INFO:     172.18.0.3:39462 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-25 20:49:24] Prefill batch, #new-seq: 1, #new-token: 11, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-12-25 20:49:26] Decode batch, #running-req: 1, #token: 44, token usage: 0.00, cuda graph: True, gen throughput (token/s): 1.16, #queue-req: 0, 
[2025-12-25 20:49:29] Decode batch, #running-req: 1, #token: 84, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.96, #queue-req: 0, 
[2025-12-25 20:49:32] Decode batch, #running-req: 1, #token: 124, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.90, #queue-req: 0, 
[2025-12-25 20:49:34] Decode batch, #running-req: 1, #token: 164, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.81, #queue-req: 0, 
[2025-12-25 20:49:37] Decode batch, #running-req: 1, #token: 204, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.91, #queue-req: 0, 
[2025-12-25 20:49:40] Decode batch, #running-req: 1, #token: 244, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.70, #queue-req: 0, 
[2025-12-25 20:49:42] Decode batch, #running-req: 1, #token: 284, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.84, #queue-req: 0, 
[2025-12-25 20:49:45] Decode batch, #running-req: 1, #token: 324, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.74, #queue-req: 0, 
[2025-12-25 20:49:48] Decode batch, #running-req: 1, #token: 364, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.88, #queue-req: 0, 
[2025-12-25 20:49:51] Decode batch, #running-req: 1, #token: 404, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.83, #queue-req: 0, 
[2025-12-25 20:49:53] Decode batch, #running-req: 1, #token: 444, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.88, #queue-req: 0, 
[2025-12-25 20:49:56] Decode batch, #running-req: 1, #token: 484, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.85, #queue-req: 0, 
[2025-12-25 20:49:59] Decode batch, #running-req: 1, #token: 524, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.90, #queue-req: 0, 
[2025-12-25 20:50:01] Decode batch, #running-req: 1, #token: 564, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.83, #queue-req: 0, 
[2025-12-25 20:50:04] Decode batch, #running-req: 1, #token: 604, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.70, #queue-req: 0, 
[2025-12-25 20:50:07] Decode batch, #running-req: 1, #token: 644, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.85, #queue-req: 0, 
[2025-12-25 20:51:02] INFO:     172.18.0.3:50562 - "POST /v1/chat/completions HTTP/1.1" 200 OK
[2025-12-25 20:51:02] Prefill batch, #new-seq: 1, #new-token: 32768, #cached-token: 3, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-12-25 20:51:04] Scheduler hit an exception: Traceback (most recent call last):
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2890, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1111, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2172, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 421, in forward_batch_generation
    out = self.model_runner.forward(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2889, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2967, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2834, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 1076, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 967, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 784, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 322, in forward
    return self.forward_core(s)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 307, in forward_core
    attn_output = self.attn(*inner_state)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 123, in forward
    return forward_batch.attn_backend.forward(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 113, in forward
    return self.forward_extend(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/attention/flashinfer_backend.py", line 840, in forward_extend
    o, _ = merge_state(o1, s1, o2, s2)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/cascade.py", line 86, in merge_state
    get_cascade_module().merge_state(v_a, s_a, v_b, s_b, v_merged, s_merged)
  File "python/tvm_ffi/cython/function.pxi", line 923, in tvm_ffi.core.Function.__call__
  File "<unknown>", line 0, in __tvm_ffi_merge_state
  File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/data/csrc/cascade.cu", line 53, in merge_state(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView)::<lambda()>::<lambda()>
    TVM_FFI_ICHECK(status == cudaSuccess)
    
RuntimeError: Check failed: (status == cudaSuccess) is false: MergeState kernel launch failed: invalid argument

[2025-12-25 20:51:04] SIGQUIT received. signum=None, frame=None. It usually means one child failed.
Killed
(kt-kernel) mukul@jarvis:~/dev-ai/ik_llama.cpp$

@mtcl commented on GitHub (Dec 26, 2025): So I was able to solve the garbled test issue with below command, but I get this new error now on a bigger prompt. ```bash SGL_ENABLE_JIT_DEEPGEMM=0 \ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ python -m sglang.launch_server \ --host 0.0.0.0 \ --port 10002 \ --model /media/mukul/data/models/zai-org/GLM-4.7 \ --kt-weight-path /media/mukul/data/models/zai-org/GLM-4.7-AMXINT4 \ --kt-cpuinfer 56 \ --kt-threadpool-count 1 \ --kt-num-gpu-experts 4 \ --kt-method AMXINT4 \ --attention-backend flashinfer \ --trust-remote-code \ --tool-call-parser glm \ --reasoning-parser glm45 \ --mem-fraction-static 0.8 \ --chunked-prefill-size 32768 \ --max-running-requests 2 \ --max-total-tokens 65536 \ --watchdog-timeout 3000 \ --enable-mixed-chunk \ --tensor-parallel-size 1 \ --enable-p2p-check \ --served-model-name jarvis-thinker \ --disable-shared-experts-fusion \ --fp8-gemm-backend triton \ --kv-cache-dtype bf16 ``` ``` [2025-12-25 20:49:17] INFO: 127.0.0.1:50774 - "GET /model_info HTTP/1.1" 200 OK [2025-12-25 20:49:17] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-12-25 20:49:19] INFO: 127.0.0.1:50784 - "POST /generate HTTP/1.1" 200 OK [2025-12-25 20:49:19] The server is fired up and ready to roll! [2025-12-25 20:49:24] INFO: 172.18.0.3:39462 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2025-12-25 20:49:24] Prefill batch, #new-seq: 1, #new-token: 11, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-12-25 20:49:26] Decode batch, #running-req: 1, #token: 44, token usage: 0.00, cuda graph: True, gen throughput (token/s): 1.16, #queue-req: 0, [2025-12-25 20:49:29] Decode batch, #running-req: 1, #token: 84, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.96, #queue-req: 0, [2025-12-25 20:49:32] Decode batch, #running-req: 1, #token: 124, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.90, #queue-req: 0, [2025-12-25 20:49:34] Decode batch, #running-req: 1, #token: 164, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.81, #queue-req: 0, [2025-12-25 20:49:37] Decode batch, #running-req: 1, #token: 204, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.91, #queue-req: 0, [2025-12-25 20:49:40] Decode batch, #running-req: 1, #token: 244, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.70, #queue-req: 0, [2025-12-25 20:49:42] Decode batch, #running-req: 1, #token: 284, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.84, #queue-req: 0, [2025-12-25 20:49:45] Decode batch, #running-req: 1, #token: 324, token usage: 0.00, cuda graph: True, gen throughput (token/s): 14.74, #queue-req: 0, [2025-12-25 20:49:48] Decode batch, #running-req: 1, #token: 364, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.88, #queue-req: 0, [2025-12-25 20:49:51] Decode batch, #running-req: 1, #token: 404, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.83, #queue-req: 0, [2025-12-25 20:49:53] Decode batch, #running-req: 1, #token: 444, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.88, #queue-req: 0, [2025-12-25 20:49:56] Decode batch, #running-req: 1, #token: 484, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.85, #queue-req: 0, [2025-12-25 20:49:59] Decode batch, #running-req: 1, #token: 524, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.90, #queue-req: 0, [2025-12-25 20:50:01] Decode batch, #running-req: 1, #token: 564, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.83, #queue-req: 0, [2025-12-25 20:50:04] Decode batch, #running-req: 1, #token: 604, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.70, #queue-req: 0, [2025-12-25 20:50:07] Decode batch, #running-req: 1, #token: 644, token usage: 0.01, cuda graph: True, gen throughput (token/s): 14.85, #queue-req: 0, [2025-12-25 20:51:02] INFO: 172.18.0.3:50562 - "POST /v1/chat/completions HTTP/1.1" 200 OK [2025-12-25 20:51:02] Prefill batch, #new-seq: 1, #new-token: 32768, #cached-token: 3, token usage: 0.00, #running-req: 0, #queue-req: 0, [2025-12-25 20:51:04] Scheduler hit an exception: Traceback (most recent call last): File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2890, in run_scheduler_process scheduler.event_loop_overlap() File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 1111, in event_loop_overlap batch_result = self.run_batch(batch) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/scheduler.py", line 2172, in run_batch batch_result = self.model_worker.forward_batch_generation( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/managers/tp_worker.py", line 421, in forward_batch_generation out = self.model_runner.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2889, in forward output = self._forward_raw( ^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2967, in _forward_raw ret = self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/model_executor/model_runner.py", line 2834, in forward_extend return self.model.forward( ^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 1076, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 967, in forward hidden_states, residual = layer( ^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 784, in forward hidden_states = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 322, in forward return self.forward_core(s) ^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/models/glm4_moe.py", line 307, in forward_core attn_output = self.attn(*inner_state) ^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/radix_attention.py", line 123, in forward return forward_batch.attn_backend.forward( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/attention/base_attn_backend.py", line 113, in forward return self.forward_extend( ^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/sglang/srt/layers/attention/flashinfer_backend.py", line 840, in forward_extend o, _ = merge_state(o1, s1, o2, s2) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/cascade.py", line 86, in merge_state get_cascade_module().merge_state(v_a, s_a, v_b, s_b, v_merged, s_merged) File "python/tvm_ffi/cython/function.pxi", line 923, in tvm_ffi.core.Function.__call__ File "<unknown>", line 0, in __tvm_ffi_merge_state File "/home/mukul/miniconda3/envs/kt-kernel/lib/python3.11/site-packages/flashinfer/data/csrc/cascade.cu", line 53, in merge_state(tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView, tvm::ffi::TensorView)::<lambda()>::<lambda()> TVM_FFI_ICHECK(status == cudaSuccess) RuntimeError: Check failed: (status == cudaSuccess) is false: MergeState kernel launch failed: invalid argument [2025-12-25 20:51:04] SIGQUIT received. signum=None, frame=None. It usually means one child failed. Killed (kt-kernel) mukul@jarvis:~/dev-ai/ik_llama.cpp$ ```

bsakaguchi commented