kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1914] [Benchmark] AVX2 perf on RTX5090 #1192

New issue
Closed
opened 2026-10-29 14:49:45 +00:00 by bsakaguchi · 0 comments
bsakaguchi commented 2026-10-29 14:49:45 +00:00
Owner
Copy link

Originally created by @yyj6666667 on GitHub (Apr 3, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1914

Benchmark Submission

Hardware

  • GPU: 8x NVIDIA GeForce RTX 5090 32GB
  • CPU: AMD EPYC 9355 32-Core Processor (128 logical / 64 physical cores)
  • RAM: 1511 GB total / 1240 GB available
  • OS: Ubuntu 22.04.5 LTS (Kernel 5.15.0-168-generic)
  • 存储: 2TB DDR5
  • PCIe: PCIe 5.0

Software

  • KTransformers Version: 0.5.2
  • Backend: kt-kernel + SGLang

Model & Precision

Model Precision TP GPU Experts CPU Infer Threadpool
Qwen3-30B-A3B-Instruct-2507 BF16 1 28 16 1
Qwen3.5-122B-A10B FP8 1 24 16 1
Qwen3.5-35B-A3B FP8 1 64 16 1
Qwen3-Coder-Next BF16 1 12 16 1
Qwen3-Coder-Next FP8 1 20 16 1
MiniMax-M2.5 FP8 1 8 16 1

Performance

Qwen3-30B-A3B-Instruct-2507 (BF16, 28 GPU Experts)

Metric hi (2) 1K (969) 2K (1930) 4K (3846) 8K (7678)
Prefill (tok/s) 14.8 1179.6 2966.6 3079.9 2971.2
Decode (tok/s) 44.4 52.5 51.6 50.2 45.8
TTFT (ms) 68 872 692 1332 2759
TOPT (ms/tok) 22.5 19.1 19.4 19.9 21.9

Qwen3.5-122B-A10B-FP8 (FP8, 24 GPU Experts)

Metric hi (2) 1K (969) 2K (1930) 4K (3846) 8K (7678)
Prefill (tok/s) 13.6 278.2 457.1 1111.6 FAIL
Decode (tok/s) 22.1 24.4 24.3 23.9 FAIL
TTFT (ms) 73 3699 4492 3689 FAIL
TOPT (ms/tok) 45.3 40.9 41.1 41.8 FAIL

Qwen3.5-35B-A3B-FP8 (FP8, 64 GPU Experts)

Metric hi (2) 1K (969) 2K (1930) 4K (3846) 8K (7678)
Prefill (tok/s) 15.9 526.2 1963.3 2392.2 6789.6
Decode (tok/s) 57.8 60.9 60.7 59.6 57.4
TTFT (ms) 63 1956 1046 1714 1207
TOPT (ms/tok) 17.3 16.4 16.5 16.8 17.4

Qwen3-Coder-Next (BF16, 12 GPU Experts)

Metric hi (2) 1K (969) 2K (1930) 4K (3846) 8K (7678)
Prefill (tok/s) 17.4 985.2 271.3 1004.2 1931.7
Decode (tok/s) 41.1 45.4 70.3 58.1 54.5
TTFT (ms) 57 1044 7568 4084 4244
TOPT (ms/tok) 24.3 22.0 14.2 17.2 18.4

Qwen3-Coder-Next-FP8 (FP8, 20 GPU Experts)

Metric hi (2) 1K (969) 2K (1930) 4K (3846) 8K (7678)
Prefill (tok/s) 13.5 533.8 361.5 1344.0 2613.8
Decode (tok/s) 37.7 44.0 50.8 71.0 55.9
TTFT (ms) 74 1928 5680 3051 3136
TOPT (ms/tok) 26.5 22.7 19.7 14.1 17.9

MiniMax-M2.5 (FP8, 8 GPU Experts)

Metric hi (2) 1K (969) 2K (1930) 4K (3846) 8K (7678)
Prefill (tok/s) 6.8 145.7 203.9 317.0 317.9
Decode (tok/s) 15.3 15.9 15.9 15.8 15.6
TTFT (ms) 147 7062 10067 12939 25781
TOPT (ms/tok) 65.4 62.9 62.9 63.3 64.0

Launch Command

/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30002 --model /mnt/data/models/Qwen3-30B-A3B-Instruct-2507 --kt-weight-path /mnt/data/models/Qwen3-30B-A3B-Instruct-2507 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 28 --kt-method BF16 --attention-backend triton --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion

/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30003 --model /mnt/data/models/Qwen3.5-122B-A10B-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-122B-A10B-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 24 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion

/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30004 --model /mnt/data/models/Qwen3.5-35B-A3B-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-35B-A3B-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 64 --kt-method FP8 --kt-gpu-prefill-token-threshold 400 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion

/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30005 --model /mnt/data/models/Qwen3.5-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 8 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion

/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30006 --model /mnt/data/models/Qwen3-Coder-Next --kt-weight-path /mnt/data/models/Qwen3-Coder-Next --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 12 --kt-method BF16 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --trust-remote-code --mem-fraction-static 0.8 --chunked-prefill-size 16384 --max-running-requests 1 --max-total-tokens 64000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion --kt-enable-dynamic-expert-update

/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30007 --model /mnt/data/models/Qwen3-Coder-Next-FP8 --kt-weight-path /mnt/data/models/Qwen3-Coder-Next-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 20 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.8 --chunked-prefill-size 16384 --max-running-requests 1 --max-total-tokens 64000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion --kt-enable-dynamic-expert-update

/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30008 --model /mnt/data/models/MiniMax-M2.5 --kt-weight-path /mnt/data/models/MiniMax-M2.5 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 8 --kt-method FP8 --kt-max-deferred-experts-per-token 0 --kt-expert-placement-strategy uniform --attention-backend flashinfer --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion
Originally created by @yyj6666667 on GitHub (Apr 3, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1914 ## Benchmark Submission ### Hardware - **GPU**: 8x NVIDIA GeForce RTX 5090 32GB - **CPU**: AMD EPYC 9355 32-Core Processor (128 logical / 64 physical cores) - **RAM**: 1511 GB total / 1240 GB available - **OS**: Ubuntu 22.04.5 LTS (Kernel 5.15.0-168-generic) - **存储**: 2TB DDR5 - **PCIe**: PCIe 5.0 ### Software - **KTransformers Version**: 0.5.2 - **Backend**: kt-kernel + SGLang ### Model & Precision | Model | Precision | TP | GPU Experts | CPU Infer | Threadpool | |---|---|---|---|---|---| | Qwen3-30B-A3B-Instruct-2507 | BF16 | 1 | 28 | 16 | 1 | | Qwen3.5-122B-A10B | FP8 | 1 | 24 | 16 | 1 | | Qwen3.5-35B-A3B | FP8 | 1 | 64 | 16 | 1 | | Qwen3-Coder-Next | BF16 | 1 | 12 | 16 | 1 | | Qwen3-Coder-Next | FP8 | 1 | 20 | 16 | 1 | | MiniMax-M2.5 | FP8 | 1 | 8 | 16 | 1 | ### Performance **Qwen3-30B-A3B-Instruct-2507 (BF16, 28 GPU Experts)** | Metric | hi (2) | 1K (969) | 2K (1930) | 4K (3846) | 8K (7678) | |---|---|---|---|---|---| | Prefill (tok/s) | 14.8 | 1179.6 | 2966.6 | 3079.9 | 2971.2 | | Decode (tok/s) | 44.4 | 52.5 | 51.6 | 50.2 | 45.8 | | TTFT (ms) | 68 | 872 | 692 | 1332 | 2759 | | TOPT (ms/tok) | 22.5 | 19.1 | 19.4 | 19.9 | 21.9 | **Qwen3.5-122B-A10B-FP8 (FP8, 24 GPU Experts)** | Metric | hi (2) | 1K (969) | 2K (1930) | 4K (3846) | 8K (7678) | |---|---|---|---|---|---| | Prefill (tok/s) | 13.6 | 278.2 | 457.1 | 1111.6 | FAIL | | Decode (tok/s) | 22.1 | 24.4 | 24.3 | 23.9 | FAIL | | TTFT (ms) | 73 | 3699 | 4492 | 3689 | FAIL | | TOPT (ms/tok) | 45.3 | 40.9 | 41.1 | 41.8 | FAIL | **Qwen3.5-35B-A3B-FP8 (FP8, 64 GPU Experts)** | Metric | hi (2) | 1K (969) | 2K (1930) | 4K (3846) | 8K (7678) | |---|---|---|---|---|---| | Prefill (tok/s) | 15.9 | 526.2 | 1963.3 | 2392.2 | 6789.6 | | Decode (tok/s) | 57.8 | 60.9 | 60.7 | 59.6 | 57.4 | | TTFT (ms) | 63 | 1956 | 1046 | 1714 | 1207 | | TOPT (ms/tok) | 17.3 | 16.4 | 16.5 | 16.8 | 17.4 | **Qwen3-Coder-Next (BF16, 12 GPU Experts)** | Metric | hi (2) | 1K (969) | 2K (1930) | 4K (3846) | 8K (7678) | |---|---|---|---|---|---| | Prefill (tok/s) | 17.4 | 985.2 | 271.3 | 1004.2 | 1931.7 | | Decode (tok/s) | 41.1 | 45.4 | 70.3 | 58.1 | 54.5 | | TTFT (ms) | 57 | 1044 | 7568 | 4084 | 4244 | | TOPT (ms/tok) | 24.3 | 22.0 | 14.2 | 17.2 | 18.4 | **Qwen3-Coder-Next-FP8 (FP8, 20 GPU Experts)** | Metric | hi (2) | 1K (969) | 2K (1930) | 4K (3846) | 8K (7678) | |---|---|---|---|---|---| | Prefill (tok/s) | 13.5 | 533.8 | 361.5 | 1344.0 | 2613.8 | | Decode (tok/s) | 37.7 | 44.0 | 50.8 | 71.0 | 55.9 | | TTFT (ms) | 74 | 1928 | 5680 | 3051 | 3136 | | TOPT (ms/tok) | 26.5 | 22.7 | 19.7 | 14.1 | 17.9 | **MiniMax-M2.5 (FP8, 8 GPU Experts)** | Metric | hi (2) | 1K (969) | 2K (1930) | 4K (3846) | 8K (7678) | |---|---|---|---|---|---| | Prefill (tok/s) | 6.8 | 145.7 | 203.9 | 317.0 | 317.9 | | Decode (tok/s) | 15.3 | 15.9 | 15.9 | 15.8 | 15.6 | | TTFT (ms) | 147 | 7062 | 10067 | 12939 | 25781 | | TOPT (ms/tok) | 65.4 | 62.9 | 62.9 | 63.3 | 64.0 | ### Launch Command ```bash /home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30002 --model /mnt/data/models/Qwen3-30B-A3B-Instruct-2507 --kt-weight-path /mnt/data/models/Qwen3-30B-A3B-Instruct-2507 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 28 --kt-method BF16 --attention-backend triton --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion /home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30003 --model /mnt/data/models/Qwen3.5-122B-A10B-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-122B-A10B-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 24 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion /home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30004 --model /mnt/data/models/Qwen3.5-35B-A3B-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-35B-A3B-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 64 --kt-method FP8 --kt-gpu-prefill-token-threshold 400 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion /home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30005 --model /mnt/data/models/Qwen3.5-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 8 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion /home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30006 --model /mnt/data/models/Qwen3-Coder-Next --kt-weight-path /mnt/data/models/Qwen3-Coder-Next --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 12 --kt-method BF16 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --trust-remote-code --mem-fraction-static 0.8 --chunked-prefill-size 16384 --max-running-requests 1 --max-total-tokens 64000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion --kt-enable-dynamic-expert-update /home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30007 --model /mnt/data/models/Qwen3-Coder-Next-FP8 --kt-weight-path /mnt/data/models/Qwen3-Coder-Next-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 20 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.8 --chunked-prefill-size 16384 --max-running-requests 1 --max-total-tokens 64000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion --kt-enable-dynamic-expert-update /home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30008 --model /mnt/data/models/MiniMax-M2.5 --kt-weight-path /mnt/data/models/MiniMax-M2.5 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 8 --kt-method FP8 --kt-max-deferred-experts-per-token 0 --kt-expert-placement-strategy uniform --attention-backend flashinfer --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1192
No description provided.