deepseek-ai/DeepGEMM

Fork 0

mirror of https://github.com/deepseek-ai/DeepGEMM.git synced 2026-11-05 00:47:26 +00:00

[PR #25] [CLOSED] add fp8 block-wise gemm benchmark #209

New issue

Closed

opened 2026-10-26 21:14:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-26 21:14:53 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/deepseek-ai/DeepGEMM/pull/25
Author: @BBuf
Created: 2/27/2025
Status: ❌ Closed

Base: main ← Head: main

📝 Commits (1)

8933678 upd

📊 Changes

2 files changed (+271 additions, -0 deletions)

View changed files

➕ benchmark/README.md (+6 -0)
➕ benchmark/benchmark_deepgemm_fp8_gemm.py (+265 -0)

📄 Description

The script benchmark all FP8 w8a8 block-wise matrix multiplications involved in DeepSeek V3/R1 under the current tensor parallelism (TP) setting with DeepGemm and vLLM/SGLang.

Conclusion : When both m and k increase, DeepGEMM demonstrates advantages over VLLM/SGLang's triton FP8 block-wise GEMM kernel.

BUG : When TP==16 and N==7168, the DeepGEMM kernel would cause the error:

/bbuf/DeepGEMM/deep_gemm/jit/../include/deep_gemm/fp8_gemm.cuh(103): error: static assertion failed with "Misaligned barriers"
      static_assert(not kMustUseUniformedScaleB or SHAPE_K_SCALES % (sizeof(Barrier) / sizeof(float)) == 0, "Misaligned barriers");
      ^
          detected during:
            instantiation of "void deep_gemm::fp8_gemm_kernel<SHAPE_N,SHAPE_K,BLOCK_M,BLOCK_N,BLOCK_K,kNumGroups,kNumStages,kNumTMAThreads,kNumMathThreadsPerGroup,kNumTMAMulticast,kGemmType>(__nv_bfloat16 *, float *, int *, uint32_t, CUtensorMap, CUtensorMap, CUtensorMap, CUtensorMap) [with SHAPE_N=7168U, SHAPE_K=1152U, BLOCK_M=128U, BLOCK_N=128U, BLOCK_K=128U, kNumGroups=1U, kNumStages=5U, kNumTMAThreads=128U, kNumMathThreadsPerGroup=128U, kNumTMAMulticast=2U, kGemmType=deep_gemm::GemmType::Normal]" at line 368
            instantiation of "void deep_gemm::Gemm<SHAPE_N, SHAPE_K, BLOCK_M, BLOCK_N, BLOCK_K, kNumGroups, kNumStages, kNumTMAMulticast, kGemmType>::run(__nv_bfloat16 *, float *, int *, uint32_t, const CUtensorMap &, const CUtensorMap &, const CUtensorMap &, const CUtensorMap &, cudaStream_t, int, uint32_t) [with SHAPE_N=7168U, SHAPE_K=1152U, BLOCK_M=128U, BLOCK_N=128U, BLOCK_K=128U, kNumGroups=1U, kNumStages=5U, kNumTMAMulticast=2U, kGemmType=deep_gemm::GemmType::Normal]" at line 37 of /root/.deep_gemm/cache/kernel.gemm_fp8_fp8_bf16_nt.0928a42d0949/kernel.cu

1 error detected in the compilation of "/root/.deep_gemm/cache/kernel.gemm_fp8_fp8_bf16_nt.0928a42d0949/kernel.cu".

In h100:

run correctness test

➜  DeepGEMM python3 /opt/dlami/nvme/bbuf/sglang/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm.py --tp_size 8 --run_correctness
INFO 02-27 01:48:01 __init__.py:190] Automatically detected platform cuda.
Running correctness tests...
Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=512,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json
WARNING 02-27 01:48:05 fp8_utils.py:441] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=512,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128,128].json
Shape m=64, n=512, k=7168:
DeepGEMM output: tensor([-21.3750,  41.2500, -16.3750, -60.0000, -72.0000], device='cuda:0',
       dtype=torch.bfloat16)
SGLang output: tensor([-21.3750,  41.2500, -16.3750, -60.0000, -72.0000], device='cuda:0',
       dtype=torch.bfloat16)
vLLM output: tensor([-21.3750,  41.2500, -16.3750, -60.0000, -72.0000], device='cuda:0',
       dtype=torch.bfloat16)
Mean absolute difference (SGLang-DeepGEMM): 1.4901161193847656e-07
Mean absolute difference (vLLM-DeepGEMM): 1.4901161193847656e-07
Mean absolute difference (vLLM-SGLang): 0.0
✅ All implementations match

INFO 02-27 01:48:05 fp8_utils.py:432] Using configuration from /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=16384,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel.
Shape m=64, n=7168, k=16384:
DeepGEMM output: tensor([ 91.0000, 161.0000, -49.2500,  46.2500,  98.5000], device='cuda:0',
       dtype=torch.bfloat16)
SGLang output: tensor([ 91.0000, 161.0000, -49.2500,  46.2500,  98.5000], device='cuda:0',
       dtype=torch.bfloat16)
vLLM output: tensor([ 91.0000, 161.0000, -49.2500,  46.2500,  98.5000], device='cuda:0',
       dtype=torch.bfloat16)
Mean absolute difference (SGLang-DeepGEMM): 2.7418136596679688e-06
Mean absolute difference (vLLM-DeepGEMM): 2.7418136596679688e-06
Mean absolute difference (vLLM-SGLang): 0.0
✅ All implementations match

Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=18432,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json
WARNING 02-27 01:48:06 fp8_utils.py:441] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=18432,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128,128].json
Shape m=64, n=18432, k=7168:
DeepGEMM output: tensor([118.5000,  28.3750, -58.5000, 180.0000,  11.1875], device='cuda:0',
       dtype=torch.bfloat16)
SGLang output: tensor([118.5000,  28.3750, -58.5000, 180.0000,  11.1875], device='cuda:0',
       dtype=torch.bfloat16)
vLLM output: tensor([118.5000,  28.3750, -58.5000, 180.0000,  11.1875], device='cuda:0',
       dtype=torch.bfloat16)
Mean absolute difference (SGLang-DeepGEMM): 4.410743713378906e-06
Mean absolute difference (vLLM-DeepGEMM): 4.410743713378906e-06
Mean absolute difference (vLLM-SGLang): 0.0
✅ All implementations match

run benchmark

The results below represent the time taken, with the unit in milliseconds (ms).

tp-size=1

fp8-gemm-performance-comparison-tp1:
          m        n        k  tp_size     DeepGEMM        SGLang          vLLM
0       8.0    576.0   7168.0      1.0   455.583990    360.992014    351.743996
1      16.0    576.0   7168.0      1.0   456.880003    360.320002    365.888000
2      32.0    576.0   7168.0      1.0   457.616001    365.728021    366.479993
3      64.0    576.0   7168.0      1.0   457.055986    364.064008    373.152018
4     128.0    576.0   7168.0      1.0   458.335996    370.288014    372.224003
5     256.0    576.0   7168.0      1.0   461.808026    360.431999    361.247987
6    1024.0    576.0   7168.0      1.0   449.440002    369.567990    361.775994
7    2048.0    576.0   7168.0      1.0   454.703987    428.544015    428.768009
8    4096.0    576.0   7168.0      1.0   629.408002    676.703990    676.496029
9       8.0  24576.0   7168.0      1.0  3127.487898   3158.031940   3160.160065
10     16.0  24576.0   7168.0      1.0  3146.368027   3151.232004   3151.199818
11     32.0  24576.0   7168.0      1.0  3146.368027   3152.672052   3153.632164
12     64.0  24576.0   7168.0      1.0  3147.648096   3154.031992   3156.320095
13    128.0  24576.0   7168.0      1.0  3157.472134   3226.464033   3228.352070
14    256.0  24576.0   7168.0      1.0  3185.744047   3346.208096   3345.504045
15   1024.0  24576.0   7168.0      1.0  3460.016012   4129.600048   4130.527973
16   2048.0  24576.0   7168.0      1.0  3830.528021   5256.735802   5187.120438
17   4096.0  24576.0   7168.0      1.0  4569.952011   7373.280048   7221.727848
18      8.0  32768.0    512.0      1.0   430.703998    388.063997    386.720002
19     16.0  32768.0    512.0      1.0   424.960017    385.519981    385.567993
20     32.0  32768.0    512.0      1.0   422.656000    384.416014    383.071989
21     64.0  32768.0    512.0      1.0   423.648000    386.527985    386.783987
22    128.0  32768.0    512.0      1.0   428.799987    392.960012    392.767996
23    256.0  32768.0    512.0      1.0   429.584026    413.695991    410.560012
24   1024.0  32768.0    512.0      1.0   450.271994    507.103980    506.399989
25   2048.0  32768.0    512.0      1.0   509.855986    633.855999    627.135992
26   4096.0  32768.0    512.0      1.0   627.359986    884.288013    870.975971
27      8.0   7168.0  16384.0      1.0  2122.080088   2153.632164   2154.927969
28     16.0   7168.0  16384.0      1.0  2140.799999   2157.056093   2154.128075
29     32.0   7168.0  16384.0      1.0  2137.248039   2159.327984   2160.431862
30     64.0   7168.0  16384.0      1.0  2139.647961   2167.824030   2166.416168
31    128.0   7168.0  16384.0      1.0  2153.232098   2209.856033   2210.880041
32    256.0   7168.0  16384.0      1.0  2186.304092   2311.104059   2308.095932
33   1024.0   7168.0  16384.0      1.0  2534.976006   3021.296024   3023.072004
34   2048.0   7168.0  16384.0      1.0  2903.007984   3808.959961   3800.287962
35   4096.0   7168.0  16384.0      1.0  3751.104116   5592.415810   5521.567822
36      8.0   7168.0  18432.0      1.0  2383.855820   2414.400101   2414.911985
37     16.0   7168.0  18432.0      1.0  2396.592140   2417.104006   2417.216063
38     32.0   7168.0  18432.0      1.0  2394.592047   2422.816038   2421.472073
39     64.0   7168.0  18432.0      1.0  2397.119999   2429.728031   2429.759979
40    128.0   7168.0  18432.0      1.0  2410.912037   2477.855921   2477.695942
41    256.0   7168.0  18432.0      1.0  2450.176001   2587.296009   2588.608027
42   1024.0   7168.0  18432.0      1.0  2840.303898   3396.895885   3397.215843
43   2048.0   7168.0  18432.0      1.0  3241.231918   4308.351994   4266.223907
44   4096.0   7168.0  18432.0      1.0  4199.776173   6211.935997   6199.744225
45      8.0  36864.0   7168.0      1.0  4614.079952   4690.591812   4690.383911
46     16.0  36864.0   7168.0      1.0  4653.952122   4678.495884   4678.495884
47     32.0  36864.0   7168.0      1.0  4652.991772   4677.343845   4677.279949
48     64.0  36864.0   7168.0      1.0  4652.863979   4679.615974   4679.423809
49    128.0  36864.0   7168.0      1.0  4663.584232   4769.167900   4771.056175
50    256.0  36864.0   7168.0      1.0  4708.703995   4982.272148   4981.056213
51   1024.0  36864.0   7168.0      1.0  5085.599899   6090.288162   6090.255737
52   2048.0  36864.0   7168.0      1.0  5570.047855   7588.031769   7567.200184
53   4096.0  36864.0   7168.0      1.0  6662.960052  10634.336472  10543.328285
54      8.0  24576.0   7168.0      1.0  3093.280077   3141.407967   3148.831844
55     16.0  24576.0   7168.0      1.0  3138.816118   3142.496109   3144.223928
56     32.0  24576.0   7168.0      1.0  3138.240099   3145.920038   3142.047882
57     64.0  24576.0   7168.0      1.0  3139.487982   3143.903971   3145.087957
58    128.0  24576.0   7168.0      1.0  3147.583961   3214.895964   3217.087984
59    256.0  24576.0   7168.0      1.0  3176.976204   3331.615925   3333.760023
60   1024.0  24576.0   7168.0      1.0  3449.584007   4130.335808   4130.943775
61   2048.0  24576.0   7168.0      1.0  3838.495970   5242.464066   5184.463978
62   4096.0  24576.0   7168.0      1.0  4572.256088   7338.687897   7222.464085
63      8.0  32768.0    512.0      1.0   457.520008    387.807995    388.159990
64     16.0  32768.0    512.0      1.0   446.079999    385.392010    385.648012
65     32.0  32768.0    512.0      1.0   448.255986    385.504007    384.368002
66     64.0  32768.0    512.0      1.0   448.927999    384.063989    384.128004
67    128.0  32768.0    512.0      1.0   449.328005    395.056009    393.983990
68    256.0  32768.0    512.0      1.0   452.223986    411.296010    410.879999
69   1024.0  32768.0    512.0      1.0   444.207996    505.568027    503.679991
70   2048.0  32768.0    512.0      1.0   508.704007    631.919980    626.048028
71   4096.0  32768.0    512.0      1.0   626.976013    871.392012    864.928007
72      8.0  24576.0   1536.0      1.0   744.848013    751.055956    759.680033
73     16.0  24576.0   1536.0      1.0   754.432023    754.495978    760.703981
74     32.0  24576.0   1536.0      1.0   753.952026    753.055990    759.263992
75     64.0  24576.0   1536.0      1.0   756.896019    755.487978    762.112021
76    128.0  24576.0   1536.0      1.0   758.047998    775.327981    785.600007
77    256.0  24576.0   1536.0      1.0   767.903984    796.864033    809.103966
78   1024.0  24576.0   1536.0      1.0   840.304017    979.663968   1007.007957
79   2048.0  24576.0   1536.0      1.0   935.743988   1220.864058   1255.679965
80   4096.0  24576.0   1536.0      1.0  1124.351978   1696.559906   1769.183993
81      8.0   4096.0   7168.0      1.0   612.128019    646.687984    680.431962
82     16.0   4096.0   7168.0      1.0   616.927981    650.480032    684.224010
83     32.0   4096.0   7168.0      1.0   616.415977    649.151981    685.711980
84     64.0   4096.0   7168.0      1.0   611.808002    652.304053    686.367989
85    128.0   4096.0   7168.0      1.0   619.199991    656.127989    689.119995
86    256.0   4096.0   7168.0      1.0   634.976029    667.104006    702.080011
87   1024.0   4096.0   7168.0      1.0   740.127981    850.624025    889.280021
88   2048.0   4096.0   7168.0      1.0   901.008010   1127.759933   1185.872078
89   4096.0   4096.0   7168.0      1.0  1187.872052   1645.135999   1771.263957
90      8.0   7168.0  18432.0      1.0  2385.871887   2410.048008   2415.632010
91     16.0   7168.0  18432.0      1.0  2396.800041   2417.520046   2414.864063
92     32.0   7168.0  18432.0      1.0  2395.088196   2422.528028   2422.368050
93     64.0   7168.0  18432.0      1.0  2396.640062   2431.616068   2431.551933
94    128.0   7168.0  18432.0      1.0  2412.575960   2480.351925   2480.799913
95    256.0   7168.0  18432.0      1.0  2452.095985   2589.312077   2589.375973
96   1024.0   7168.0  18432.0      1.0  2840.111971   3395.808220   3397.488117
97   2048.0   7168.0  18432.0      1.0  3251.120090   4284.751892   4273.216248
98   4096.0   7168.0  18432.0      1.0  4167.967796   6282.048225   6208.511829
99      8.0   7168.0  16384.0      1.0  2117.791891   2148.927927   2153.151989
100    16.0   7168.0  16384.0      1.0  2140.000105   2153.392076   2155.071974
101    32.0   7168.0  16384.0      1.0  2136.320114   2159.840107   2159.984112
102    64.0   7168.0  16384.0      1.0  2139.744043   2165.967941   2168.031931
103   128.0   7168.0  16384.0      1.0  2153.872013   2212.512016   2210.815907
104   256.0   7168.0  16384.0      1.0  2187.183857   2310.112000   2311.167955
105  1024.0   7168.0  16384.0      1.0  2536.432028   3023.056030   3024.911880
106  2048.0   7168.0  16384.0      1.0  2897.183895   3837.183952   3806.976080
107  4096.0   7168.0  16384.0      1.0  3717.247963   5539.231777   5519.584179
108     8.0   7168.0   2048.0      1.0   450.495988    366.656005    361.135989
109    16.0   7168.0   2048.0      1.0   448.736012    365.727991    359.120011
110    32.0   7168.0   2048.0      1.0   448.480010    360.895991    357.823998
111    64.0   7168.0   2048.0      1.0   448.127985    357.888013    361.600012
112   128.0   7168.0   2048.0      1.0   451.008022    358.047992    360.927999
113   256.0   7168.0   2048.0      1.0   451.151997    371.039987    371.407986
114  1024.0   7168.0   2048.0      1.0   455.040008    459.968001    459.679991
115  2048.0   7168.0   2048.0      1.0   450.432003    561.951995    563.103974
116  4096.0   7168.0   2048.0      1.0   578.320026    807.775974    797.504008

tp-size=8

fp8-gemm-performance-comparison-tp8:
          m        n        k  tp_size     DeepGEMM       SGLang         vLLM
0       8.0    576.0   7168.0      8.0   478.496015   384.191990   378.912002
1      16.0    576.0   7168.0      8.0   475.744009   384.415984   384.191990
2      32.0    576.0   7168.0      8.0   476.415992   384.736001   380.015999
3      64.0    576.0   7168.0      8.0   473.583996   380.144000   381.359994
4     128.0    576.0   7168.0      8.0   477.231979   386.304021   383.279979
5     256.0    576.0   7168.0      8.0   475.295991   383.904010   380.720019
6    1024.0    576.0   7168.0      8.0   474.943995   385.168016   385.663986
7    2048.0    576.0   7168.0      8.0   478.287995   428.768009   429.152012
8    4096.0    576.0   7168.0      8.0   631.232023   681.855977   683.744013
9       8.0  24576.0   7168.0      8.0  3135.999918  3153.343916  3149.183989
10     16.0  24576.0   7168.0      8.0  3136.607885  3140.959978  3143.840075
11     32.0  24576.0   7168.0      8.0  3137.120008  3145.215988  3141.488075
12     64.0  24576.0   7168.0      8.0  3141.472101  3146.080017  3147.007942
13    128.0  24576.0   7168.0      8.0  3149.119854  3218.351841  3218.495846
14    256.0  24576.0   7168.0      8.0  3180.943966  3338.752031  3338.848114
15   1024.0  24576.0   7168.0      8.0  3455.071926  4132.575989  4132.031918
16   2048.0  24576.0   7168.0      8.0  3837.728024  5242.000103  5186.160088
17   4096.0  24576.0   7168.0      8.0  4577.280045  7338.047981  7265.632153
18      8.0  32768.0    512.0      8.0   470.207989   387.616009   386.687994
19     16.0  32768.0    512.0      8.0   471.487999   384.351999   379.696012
20     32.0  32768.0    512.0      8.0   472.671986   384.864002   385.136008
21     64.0  32768.0    512.0      8.0   473.327994   386.335999   388.704002
22    128.0  32768.0    512.0      8.0   474.624008   396.800011   393.952012
23    256.0  32768.0    512.0      8.0   470.799983   411.743999   409.247994
24   1024.0  32768.0    512.0      8.0   470.560014   509.296000   508.080006
25   2048.0  32768.0    512.0      8.0   509.151995   629.055977   628.351986
26   4096.0  32768.0    512.0      8.0   629.760027   886.111975   868.864000
27      8.0   7168.0  16384.0      8.0  2112.319946  2149.775982  2158.063889
28     16.0   7168.0  16384.0      8.0  2146.751881  2162.544012  2161.551952
29     32.0   7168.0  16384.0      8.0  2142.751932  2165.823936  2167.504072
30     64.0   7168.0  16384.0      8.0  2144.144058  2171.008110  2173.488140
31    128.0   7168.0  16384.0      8.0  2161.600113  2218.240023  2220.191956
32    256.0   7168.0  16384.0      8.0  2195.359945  2319.648027  2322.224140
33   1024.0   7168.0  16384.0      8.0  2540.944099  3041.167974  3032.768011
34   2048.0   7168.0  16384.0      8.0  2914.079905  3853.408098  3813.920021
35   4096.0   7168.0  16384.0      8.0  3748.720169  5609.439850  5589.791775
36      8.0   7168.0  18432.0      8.0  2388.911963  2412.415981  2414.047956
37     16.0   7168.0  18432.0      8.0  2394.703865  2411.087990  2419.296026
38     32.0   7168.0  18432.0      8.0  2399.872065  2429.023981  2426.431894
39     64.0   7168.0  18432.0      8.0  2402.192116  2438.463926  2437.727928
40    128.0   7168.0  18432.0      8.0  2421.983957  2489.536047  2489.903927
41    256.0   7168.0  18432.0      8.0  2459.872007  2600.800037  2600.032091
42   1024.0   7168.0  18432.0      8.0  2841.232061  3394.383907  3390.416145
43   2048.0   7168.0  18432.0      8.0  3259.903908  4320.672035  4282.527924
44   4096.0   7168.0  18432.0      8.0  4199.071884  6304.543972  6224.607944
45      8.0   4608.0   7168.0      8.0   680.768013   689.055979   689.872026
46     16.0   4608.0   7168.0      8.0   684.480011   693.567991   692.095995
47     32.0   4608.0   7168.0      8.0   682.879984   693.104029   693.359971
48     64.0   4608.0   7168.0      8.0   681.952000   696.287990   696.928024
49    128.0   4608.0   7168.0      8.0   688.480020   715.071976   714.655995
50    256.0   4608.0   7168.0      8.0   703.552008   746.656001   743.903995
51   1024.0   4608.0   7168.0      8.0   820.864022   958.368003   955.376029
52   2048.0   4608.0   7168.0      8.0  1001.935959  1281.056046  1268.656015
53   4096.0   4608.0   7168.0      8.0  1275.552034  1807.456017  1790.607929
54      8.0   3072.0   7168.0      8.0   496.223986   499.055982   500.383973
55     16.0   3072.0   7168.0      8.0   496.623993   498.111993   501.631975
56     32.0   3072.0   7168.0      8.0   494.623989   503.407955   501.536012
57     64.0   3072.0   7168.0      8.0   492.671996   504.320025   504.224002
58    128.0   3072.0   7168.0      8.0   497.759998   513.023973   512.000024
59    256.0   3072.0   7168.0      8.0   512.000024   539.232016   538.319945
60   1024.0   3072.0   7168.0      8.0   613.759995   703.296006   703.136027
61   2048.0   3072.0   7168.0      8.0   755.584002   924.704015   922.288001
62   4096.0   3072.0   7168.0      8.0  1022.896051  1363.968015  1364.223957
63      8.0   4096.0    512.0      8.0   473.327994   363.263994   376.224011
64     16.0   4096.0    512.0      8.0   471.455991   362.240016   362.271994
65     32.0   4096.0    512.0      8.0   470.048010   360.799998   361.248016
66     64.0   4096.0    512.0      8.0   466.655999   360.064000   357.984006
67    128.0   4096.0    512.0      8.0   470.256001   376.320004   376.367986
68    256.0   4096.0    512.0      8.0   469.983995   382.400006   378.544003
69   1024.0   4096.0    512.0      8.0   476.639986   379.168004   380.095989
70   2048.0   4096.0    512.0      8.0   472.975999   382.959992   383.776009
71   4096.0   4096.0    512.0      8.0   474.496007   385.951996   385.744005
72      8.0   3072.0   1536.0      8.0   473.536015   374.480009   359.696001
73     16.0   3072.0   1536.0      8.0   472.223997   362.176001   358.047992
74     32.0   3072.0   1536.0      8.0   476.191998   370.415986   369.664013
75     64.0   3072.0   1536.0      8.0   476.352006   365.568012   373.360008
76    128.0   3072.0   1536.0      8.0   488.207996   379.648000   367.935985
77    256.0   3072.0   1536.0      8.0   479.360014   365.040004   378.688008
78   1024.0   3072.0   1536.0      8.0   475.152016   390.864015   383.616000
79   2048.0   3072.0   1536.0      8.0   481.391996   392.351985   391.968012
80   4096.0   3072.0   1536.0      8.0   487.088025   388.448000   386.976004
81      8.0    512.0   7168.0      8.0   478.944004   383.424014   372.047991
82     16.0    512.0   7168.0      8.0   475.232005   382.719994   377.552003
83     32.0    512.0   7168.0      8.0   475.695997   377.407998   385.679990
84     64.0    512.0   7168.0      8.0   476.368010   382.975996   377.631992
85    128.0    512.0   7168.0      8.0   482.656002   380.928010   384.496003
86    256.0    512.0   7168.0      8.0   476.159990   386.047989   382.192016
87   1024.0    512.0   7168.0      8.0   480.176002   385.199994   387.551993
88   2048.0    512.0   7168.0      8.0   481.743991   400.319993   426.144004
89   4096.0    512.0   7168.0      8.0   584.336042   629.440010   646.975994
90      8.0   7168.0   2304.0      8.0   479.312003   382.560015   461.791992
91     16.0   7168.0   2304.0      8.0   482.912004   383.935988   382.016003
92     32.0   7168.0   2304.0      8.0   479.360014   379.999995   380.335987
93     64.0   7168.0   2304.0      8.0   476.624012   380.383998   380.255997
94    128.0   7168.0   2304.0      8.0   479.743987   389.472008   393.599987
95    256.0   7168.0   2304.0      8.0   478.336006   407.375991   408.335984
96   1024.0   7168.0   2304.0      8.0   483.168006   504.688025   504.607975
97   2048.0   7168.0   2304.0      8.0   501.504004   623.488009   625.760019
98   4096.0   7168.0   2304.0      8.0   635.775983   886.687994   896.160007
99      8.0   7168.0   2048.0      8.0   480.463982   390.592009   392.560005
100    16.0   7168.0   2048.0      8.0   480.991989   392.336011   385.424018
101    32.0   7168.0   2048.0      8.0   478.703976   390.751988   386.896014
102    64.0   7168.0   2048.0      8.0   481.503993   390.959978   397.664011
103   128.0   7168.0   2048.0      8.0   484.239995   394.591987   398.111999
104   256.0   7168.0   2048.0      8.0   500.415981   408.383995   404.127985
105  1024.0   7168.0   2048.0      8.0   501.375973   462.399989   461.088002
106  2048.0   7168.0   2048.0      8.0   507.647991   566.383958   567.407966
107  4096.0   7168.0   2048.0      8.0   578.320026   805.904031   798.207998
108     8.0   7168.0    256.0      8.0   505.280018   397.215992   392.672002
109    16.0   7168.0    256.0      8.0   494.751990   395.040005   392.416000
110    32.0   7168.0    256.0      8.0   496.767998   393.279999   394.751996
111    64.0   7168.0    256.0      8.0   497.664005   393.984020   391.488016
112   128.0   7168.0    256.0      8.0   504.159987   395.904005   397.920012
113   256.0   7168.0    256.0      8.0   497.824013   399.711996   393.599987
114  1024.0   7168.0    256.0      8.0   499.552011   398.847997   375.871986
115  2048.0   7168.0    256.0      8.0   474.063993   369.343996   369.055986
116  4096.0   7168.0    256.0      8.0   478.015989   372.720003   370.527983

In h200:

tp=8

fp8-gemm-performance-comparison-tp8:
          m        n        k  tp_size     DeepGEMM       SGLang         vLLM
0       8.0    576.0   7168.0      8.0   224.127993   170.111999   172.575995
1      16.0    576.0   7168.0      8.0   222.240001   171.008006   171.519995
2      32.0    576.0   7168.0      8.0   222.624004   171.455994   171.039999
3      64.0    576.0   7168.0      8.0   221.855998   170.368001   169.919997
4     128.0    576.0   7168.0      8.0   221.231997   179.104000   179.168001
5     256.0    576.0   7168.0      8.0   220.479995   190.975994   189.472005
6    1024.0    576.0   7168.0      8.0   253.311992   281.823993   281.599998
7    2048.0    576.0   7168.0      8.0   370.368004   405.503988   407.328010
8    4096.0    576.0   7168.0      8.0   579.487979   642.207980   639.567971
9       8.0  24576.0   7168.0      8.0  2901.792049  2917.887926  2919.071913
10     16.0  24576.0   7168.0      8.0  2891.839981  2910.655975  2909.183979
11     32.0  24576.0   7168.0      8.0  2891.007900  2910.207987  2910.239935
12     64.0  24576.0   7168.0      8.0  2889.663935  2909.984112  2911.231995
13    128.0  24576.0   7168.0      8.0  2898.240089  2984.416008  2986.080170
14    256.0  24576.0   7168.0      8.0  2934.623957  3105.184078  3103.264093
15   1024.0  24576.0   7168.0      8.0  3208.287954  3917.695999  3919.359922
16   2048.0  24576.0   7168.0      8.0  3597.023964  5002.848148  5005.407810
17   4096.0  24576.0   7168.0      8.0  4326.303959  7080.800056  7089.375973
18      8.0  32768.0    512.0      8.0   359.344006   354.431987   354.431987
19     16.0  32768.0    512.0      8.0   359.679997   356.543988   356.207997
20     32.0  32768.0    512.0      8.0   360.368013   354.672015   354.752004
21     64.0  32768.0    512.0      8.0   360.480011   356.943995   356.976002
22    128.0  32768.0    512.0      8.0   363.456011   366.207987   365.568012
23    256.0  32768.0    512.0      8.0   370.624006   382.048011   382.463992
24   1024.0  32768.0    512.0      8.0   420.320004   479.584008   479.167998
25   2048.0  32768.0    512.0      8.0   482.176006   602.815986   602.976024
26   4096.0  32768.0    512.0      8.0   607.743979   854.240000   852.096021
27      8.0   7168.0  16384.0      8.0  1983.152032  1996.079922  1996.160030
28     16.0   7168.0  16384.0      8.0  1981.760025  1998.687983  1998.208046
29     32.0   7168.0  16384.0      8.0  1976.079941  2001.136065  2002.592087
30     64.0   7168.0  16384.0      8.0  1968.703985  2006.432056  2007.040024
31    128.0   7168.0  16384.0      8.0  1986.239910  2054.944038  2054.175854
32    256.0   7168.0  16384.0      8.0  2024.544001  2151.535988  2153.055906
33   1024.0   7168.0  16384.0      8.0  2354.367971  2859.087944  2861.583948
34   2048.0   7168.0  16384.0      8.0  2713.248014  3651.167870  3646.016121
35   4096.0   7168.0  16384.0      8.0  3503.871918  5345.327854  5352.416039
36      8.0   7168.0  18432.0      8.0  2219.264030  2236.991882  2236.704111
37     16.0   7168.0  18432.0      8.0  2211.999893  2240.128040  2237.535954
38     32.0   7168.0  18432.0      8.0  2207.520008  2240.832090  2240.704060
39     64.0   7168.0  18432.0      8.0  2207.551956  2250.319958  2249.792099
40    128.0   7168.0  18432.0      8.0  2224.832058  2301.199913  2302.623987
41    256.0   7168.0  18432.0      8.0  2267.424107  2410.992146  2410.416126
42   1024.0   7168.0  18432.0      8.0  2636.623859  3206.624031  3205.568075
43   2048.0   7168.0  18432.0      8.0  3037.007809  4082.544327  4087.632179
44   4096.0   7168.0  18432.0      8.0  3923.712015  6007.535934  6005.424023
45      8.0   4608.0   7168.0      8.0   635.136008   637.983978   638.128042
46     16.0   4608.0   7168.0      8.0   630.079985   638.848007   638.912022
47     32.0   4608.0   7168.0      8.0   629.279971   640.511990   641.439974
48     64.0   4608.0   7168.0      8.0   626.655996   643.263996   644.320011
49    128.0   4608.0   7168.0      8.0   633.872032   661.311984   662.016034
50    256.0   4608.0   7168.0      8.0   647.871971   694.176018   693.343997
51   1024.0   4608.0   7168.0      8.0   761.568010   902.032018   902.480006
52   2048.0   4608.0   7168.0      8.0   931.904018  1210.719943  1212.543964
53   4096.0   4608.0   7168.0      8.0  1198.224068  1722.336054  1723.551989
54      8.0   3072.0   7168.0      8.0   463.663995   461.504012   462.336004
55     16.0   3072.0   7168.0      8.0   457.376003   463.808000   461.551994
56     32.0   3072.0   7168.0      8.0   456.831992   466.176003   466.048002
57     64.0   3072.0   7168.0      8.0   453.520000   468.511999   467.967987
58    128.0   3072.0   7168.0      8.0   460.960001   477.456003   477.551997
59    256.0   3072.0   7168.0      8.0   473.567992   502.879977   501.520038
60   1024.0   3072.0   7168.0      8.0   569.696009   662.495971   662.591994
61   2048.0   3072.0   7168.0      8.0   703.439951   875.584006   876.991987
62   4096.0   3072.0   7168.0      8.0   958.591998  1310.176015  1310.783982
63      8.0   4096.0    512.0      8.0   227.903992   170.368001   168.704003
64     16.0   4096.0    512.0      8.0   224.848002   170.527995   170.111999
65     32.0   4096.0    512.0      8.0   225.375995   170.320004   169.263989
66     64.0   4096.0    512.0      8.0   224.672005   170.175999   169.568002
67    128.0   4096.0    512.0      8.0   227.008000   171.488002   170.911998
68    256.0   4096.0    512.0      8.0   227.039993   172.031999   170.607999
69   1024.0   4096.0    512.0      8.0   227.520004   172.864005   172.800004
70   2048.0   4096.0    512.0      8.0   227.039993   175.264001   174.976006
71   4096.0   4096.0    512.0      8.0   227.487996   185.471997   186.751992
72      8.0   3072.0   1536.0      8.0   227.295995   170.847997   170.016006
73     16.0   3072.0   1536.0      8.0   226.480007   171.136007   170.240000
74     32.0   3072.0   1536.0      8.0   225.567997   171.296000   170.767993
75     64.0   3072.0   1536.0      8.0   226.815999   171.391994   170.975998
76    128.0   3072.0   1536.0      8.0   227.648005   171.519995   170.144007
77    256.0   3072.0   1536.0      8.0   225.295991   168.640003   168.768004
78   1024.0   3072.0   1536.0      8.0   224.240005   191.520005   193.279997
79   2048.0   3072.0   1536.0      8.0   228.992000   238.368005   239.071995
80   4096.0   3072.0   1536.0      8.0   270.399988   336.863995   340.063989
81      8.0    512.0   7168.0      8.0   226.159990   175.871998   175.056010
82     16.0    512.0   7168.0      8.0   228.720009   175.200000   173.951998
83     32.0    512.0   7168.0      8.0   228.799999   177.279994   176.159993
84     64.0    512.0   7168.0      8.0   229.552001   177.215993   177.088007
85    128.0    512.0   7168.0      8.0   228.480011   175.487995   175.136000
86    256.0    512.0   7168.0      8.0   228.031993   176.768005   176.512003
87   1024.0    512.0   7168.0      8.0   235.072002   251.040012   250.079989
88   2048.0    512.0   7168.0      8.0   354.752004   371.616006   374.112010
89   4096.0    512.0   7168.0      8.0   541.599989   589.119971   589.887977
90      8.0   7168.0   2304.0      8.0   357.951999   358.128011   357.919991
91     16.0   7168.0   2304.0      8.0   357.551992   356.992006   357.248008
92     32.0   7168.0   2304.0      8.0   357.823998   358.720005   358.687997
93     64.0   7168.0   2304.0      8.0   357.088000   358.080000   358.271986
94    128.0   7168.0   2304.0      8.0   358.880013   365.359992   364.111990
95    256.0   7168.0   2304.0      8.0   365.247995   377.247989   377.664000
96   1024.0   7168.0   2304.0      8.0   411.231995   476.767987   477.120012
97   2048.0   7168.0   2304.0      8.0   468.607992   599.168003   599.135995
98   4096.0   7168.0   2304.0      8.0   596.768022   854.687989   856.064022
99      8.0   7168.0   2048.0      8.0   324.864000   321.056008   320.767999
100    16.0   7168.0   2048.0      8.0   325.888008   323.552012   324.384004
101    32.0   7168.0   2048.0      8.0   324.784011   323.583990   324.256003
102    64.0   7168.0   2048.0      8.0   325.152010   325.152010   325.760007
103   128.0   7168.0   2048.0      8.0   326.687992   329.407990   331.328005
104   256.0   7168.0   2048.0      8.0   333.824009   343.952000   345.376015
105  1024.0   7168.0   2048.0      8.0   375.088006   434.080005   434.175998
106  2048.0   7168.0   2048.0      8.0   427.807987   537.984014   540.000021
107  4096.0   7168.0   2048.0      8.0   542.783976   768.224001   769.135952
108     8.0   7168.0    256.0      8.0   231.728002   174.879998   173.311993
109    16.0   7168.0    256.0      8.0   230.591998   173.087999   173.792005
110    32.0   7168.0    256.0      8.0   232.352003   176.512003   175.487995
111    64.0   7168.0    256.0      8.0   230.368003   175.648004   175.807998
112   128.0   7168.0    256.0      8.0   231.440008   175.280005   174.559996
113   256.0   7168.0    256.0      8.0   231.488004   175.280005   174.559996
114  1024.0   7168.0    256.0      8.0   230.031997   178.080007   178.640008
115  2048.0   7168.0    256.0      8.0   234.463990   180.160001   180.928007
116  4096.0   7168.0    256.0      8.0   231.840000   180.895999   180.99200

tp=16

fp8-gemm-performance-comparison-tp16:
         m        n       k  tp_size     DeepGEMM       SGLang         vLLM
0      8.0    576.0  7168.0     16.0   234.512001   179.104000   179.232001
1     16.0    576.0  7168.0     16.0   229.312003   179.376006   176.927999
2     32.0    576.0  7168.0     16.0   231.056005   179.872006   180.800006
3     64.0    576.0  7168.0     16.0   228.399992   178.479999   177.568004
4    128.0    576.0  7168.0     16.0   226.768002   177.151993   176.512003
5    256.0    576.0  7168.0     16.0   228.351995   189.088002   190.559998
6   1024.0    576.0  7168.0     16.0   250.800014   280.223995   280.128002
7   2048.0    576.0  7168.0     16.0   369.536012   404.352009   407.231987
8   4096.0    576.0  7168.0     16.0   578.256011   639.840007   638.656020
9      8.0  24576.0  7168.0     16.0  2900.063992  2914.688110  2917.311907
10    16.0  24576.0  7168.0     16.0  2890.399933  2907.840014  2908.159971
11    32.0  24576.0  7168.0     16.0  2889.823914  2907.648087  2907.007933
12    64.0  24576.0  7168.0     16.0  2889.568090  2909.696102  2908.351898
13   128.0  24576.0  7168.0     16.0  2897.088051  2983.824015  2984.224081
14   256.0  24576.0  7168.0     16.0  2934.079885  3103.647947  3103.552103
15  1024.0  24576.0  7168.0     16.0  3206.416130  3919.104099  3918.879986
16  2048.0  24576.0  7168.0     16.0  3596.384048  5003.456116  5001.311779
17  4096.0  24576.0  7168.0     16.0  4326.864243  7081.632137  7091.807842
18     8.0  32768.0   512.0     16.0   357.712001   353.280008   353.376001
19    16.0  32768.0   512.0     16.0   359.232008   354.416013   354.207993
20    32.0  32768.0   512.0     16.0   358.783990   353.408009   353.408009
21    64.0  32768.0   512.0     16.0   358.848006   353.599995   353.664011
22   128.0  32768.0   512.0     16.0   361.279994   364.800006   362.415999
23   256.0  32768.0   512.0     16.0   368.719995   379.487991   381.632000
24  1024.0  32768.0   512.0     16.0   418.352008   477.824003   477.631986
25  2048.0  32768.0   512.0     16.0   480.544001   603.519976   601.087987
26  4096.0  32768.0   512.0     16.0   605.823994   852.064013   851.391971
27     8.0   2304.0  7168.0     16.0   368.192017   370.463997   371.551991
28    16.0   2304.0  7168.0     16.0   364.127994   371.151984   373.600006
29    32.0   2304.0  7168.0     16.0   363.040000   374.464005   375.216007
30    64.0   2304.0  7168.0     16.0   362.848014   378.095984   377.664000
31   128.0   2304.0  7168.0     16.0   371.039987   386.079997   386.112005
32   256.0   2304.0  7168.0     16.0   384.095997   413.727999   414.463997
33  1024.0   2304.0  7168.0     16.0   474.624008   548.031986   549.983978
34  2048.0   2304.0  7168.0     16.0   606.912017   746.208012   748.799980
35  4096.0   2304.0  7168.0     16.0   864.160001  1134.799957  1137.199998
36     8.0   1536.0  7168.0     16.0   272.255987   278.816015   278.943986
37    16.0   1536.0  7168.0     16.0   269.760013   279.040009   281.280011
38    32.0   1536.0  7168.0     16.0   268.415987   283.199996   283.199996
39    64.0   1536.0  7168.0     16.0   267.967999   284.976006   285.376012
40   128.0   1536.0  7168.0     16.0   275.792003   292.127997   289.407998
41   256.0   1536.0  7168.0     16.0   289.887995   304.991990   305.023998
42  1024.0   1536.0  7168.0     16.0   369.311988   409.568012   411.455989
43  2048.0   1536.0  7168.0     16.0   494.271994   586.943984   587.296009
44  4096.0   1536.0  7168.0     16.0   708.384037   878.383994   879.487991
45     8.0   2048.0   512.0     16.0   229.983985   176.223993   174.303994
46    16.0   2048.0   512.0     16.0   228.095993   174.079999   173.472002
47    32.0   2048.0   512.0     16.0   227.968007   172.928005   173.615992
48    64.0   2048.0   512.0     16.0   229.248002   174.336001   174.048007
49   128.0   2048.0   512.0     16.0   230.463997   175.136000   177.199990
50   256.0   2048.0   512.0     16.0   229.279995   177.487999   175.999999
51  1024.0   2048.0   512.0     16.0   230.463997   173.375994   174.208000
52  2048.0   2048.0   512.0     16.0   228.735998   176.448002   175.840005
53  4096.0   2048.0   512.0     16.0   228.848010   180.127993   178.368002
54     8.0   1536.0  1536.0     16.0   227.712005   170.144007   172.160000
55    16.0   1536.0  1536.0     16.0   230.720013   171.792001   171.583995
56    32.0   1536.0  1536.0     16.0   229.727998   171.232000   170.767993
57    64.0   1536.0  1536.0     16.0   228.624001   171.488002   171.327993
58   128.0   1536.0  1536.0     16.0   229.376003   170.736000   171.263993
59   256.0   1536.0  1536.0     16.0   230.704010   171.232000   171.568006
60  1024.0   1536.0  1536.0     16.0   228.479996   173.503995   173.600003
61  2048.0   1536.0  1536.0     16.0   228.607997   181.600004   184.064001
62  4096.0   1536.0  1536.0     16.0   229.535997   245.407999   250.272006
63     8.0    256.0  7168.0     16.0   230.048001   173.759997   173.600003
64    16.0    256.0  7168.0     16.0   228.992000   173.951998   173.087999
65    32.0    256.0  7168.0     16.0   230.399996   173.728004   173.664004
66    64.0    256.0  7168.0     16.0   228.848010   174.848005   173.951998
67   128.0    256.0  7168.0     16.0   230.976000   173.216000   173.887998
68   256.0    256.0  7168.0     16.0   231.616005   174.383998   185.791999
69  1024.0    256.0  7168.0     16.0   229.824007   241.632000   269.632012
70  2048.0    256.0  7168.0     16.0   312.032014   348.639995   376.320004
71  4096.0    256.0  7168.0     16.0   517.184019   536.768019   565.999985

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/deepseek-ai/DeepGEMM/pull/25 **Author:** [@BBuf](https://github.com/BBuf) **Created:** 2/27/2025 **Status:** ❌ Closed **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (1) - [`8933678`](https://github.com/deepseek-ai/DeepGEMM/commit/8933678ee566b801e7e7250e32937d8ed03f9483) upd ### 📊 Changes **2 files changed** (+271 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `benchmark/README.md` (+6 -0) ➕ `benchmark/benchmark_deepgemm_fp8_gemm.py` (+265 -0) </details> ### 📄 Description The script benchmark all FP8 w8a8 block-wise matrix multiplications involved in DeepSeek V3/R1 under the current tensor parallelism (TP) setting with [DeepGemm](https://github.com/deepseek-ai/DeepGEMM) and vLLM/SGLang. **Conclusion** : When both m and k increase, DeepGEMM demonstrates advantages over VLLM/SGLang's triton FP8 block-wise GEMM kernel. **BUG** : When `TP==16` and `N==7168`, the DeepGEMM kernel would cause the error: ```shell /bbuf/DeepGEMM/deep_gemm/jit/../include/deep_gemm/fp8_gemm.cuh(103): error: static assertion failed with "Misaligned barriers" static_assert(not kMustUseUniformedScaleB or SHAPE_K_SCALES % (sizeof(Barrier) / sizeof(float)) == 0, "Misaligned barriers"); ^ detected during: instantiation of "void deep_gemm::fp8_gemm_kernel<SHAPE_N,SHAPE_K,BLOCK_M,BLOCK_N,BLOCK_K,kNumGroups,kNumStages,kNumTMAThreads,kNumMathThreadsPerGroup,kNumTMAMulticast,kGemmType>(__nv_bfloat16 *, float *, int *, uint32_t, CUtensorMap, CUtensorMap, CUtensorMap, CUtensorMap) [with SHAPE_N=7168U, SHAPE_K=1152U, BLOCK_M=128U, BLOCK_N=128U, BLOCK_K=128U, kNumGroups=1U, kNumStages=5U, kNumTMAThreads=128U, kNumMathThreadsPerGroup=128U, kNumTMAMulticast=2U, kGemmType=deep_gemm::GemmType::Normal]" at line 368 instantiation of "void deep_gemm::Gemm<SHAPE_N, SHAPE_K, BLOCK_M, BLOCK_N, BLOCK_K, kNumGroups, kNumStages, kNumTMAMulticast, kGemmType>::run(__nv_bfloat16 *, float *, int *, uint32_t, const CUtensorMap &, const CUtensorMap &, const CUtensorMap &, const CUtensorMap &, cudaStream_t, int, uint32_t) [with SHAPE_N=7168U, SHAPE_K=1152U, BLOCK_M=128U, BLOCK_N=128U, BLOCK_K=128U, kNumGroups=1U, kNumStages=5U, kNumTMAMulticast=2U, kGemmType=deep_gemm::GemmType::Normal]" at line 37 of /root/.deep_gemm/cache/kernel.gemm_fp8_fp8_bf16_nt.0928a42d0949/kernel.cu 1 error detected in the compilation of "/root/.deep_gemm/cache/kernel.gemm_fp8_fp8_bf16_nt.0928a42d0949/kernel.cu". ``` In h100: ### run correctness test ```shell ➜ DeepGEMM python3 /opt/dlami/nvme/bbuf/sglang/benchmark/kernels/deepseek/benchmark_deepgemm_fp8_gemm.py --tp_size 8 --run_correctness INFO 02-27 01:48:01 __init__.py:190] Automatically detected platform cuda. Running correctness tests... Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=512,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json WARNING 02-27 01:48:05 fp8_utils.py:441] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=512,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128,128].json Shape m=64, n=512, k=7168: DeepGEMM output: tensor([-21.3750, 41.2500, -16.3750, -60.0000, -72.0000], device='cuda:0', dtype=torch.bfloat16) SGLang output: tensor([-21.3750, 41.2500, -16.3750, -60.0000, -72.0000], device='cuda:0', dtype=torch.bfloat16) vLLM output: tensor([-21.3750, 41.2500, -16.3750, -60.0000, -72.0000], device='cuda:0', dtype=torch.bfloat16) Mean absolute difference (SGLang-DeepGEMM): 1.4901161193847656e-07 Mean absolute difference (vLLM-DeepGEMM): 1.4901161193847656e-07 Mean absolute difference (vLLM-SGLang): 0.0 ✅ All implementations match INFO 02-27 01:48:05 fp8_utils.py:432] Using configuration from /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=7168,K=16384,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128,128].json for W8A8 Block FP8 kernel. Shape m=64, n=7168, k=16384: DeepGEMM output: tensor([ 91.0000, 161.0000, -49.2500, 46.2500, 98.5000], device='cuda:0', dtype=torch.bfloat16) SGLang output: tensor([ 91.0000, 161.0000, -49.2500, 46.2500, 98.5000], device='cuda:0', dtype=torch.bfloat16) vLLM output: tensor([ 91.0000, 161.0000, -49.2500, 46.2500, 98.5000], device='cuda:0', dtype=torch.bfloat16) Mean absolute difference (SGLang-DeepGEMM): 2.7418136596679688e-06 Mean absolute difference (vLLM-DeepGEMM): 2.7418136596679688e-06 Mean absolute difference (vLLM-SGLang): 0.0 ✅ All implementations match Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /sgl-workspace/sglang/python/sglang/srt/layers/quantization/configs/N=18432,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128, 128].json WARNING 02-27 01:48:06 fp8_utils.py:441] Using default W8A8 Block FP8 kernel config. Performance might be sub-optimal! Config file not found at /usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/quantization/utils/configs/N=18432,K=7168,device_name=NVIDIA_H100_80GB_HBM3,dtype=fp8_w8a8,block_shape=[128,128].json Shape m=64, n=18432, k=7168: DeepGEMM output: tensor([118.5000, 28.3750, -58.5000, 180.0000, 11.1875], device='cuda:0', dtype=torch.bfloat16) SGLang output: tensor([118.5000, 28.3750, -58.5000, 180.0000, 11.1875], device='cuda:0', dtype=torch.bfloat16) vLLM output: tensor([118.5000, 28.3750, -58.5000, 180.0000, 11.1875], device='cuda:0', dtype=torch.bfloat16) Mean absolute difference (SGLang-DeepGEMM): 4.410743713378906e-06 Mean absolute difference (vLLM-DeepGEMM): 4.410743713378906e-06 Mean absolute difference (vLLM-SGLang): 0.0 ✅ All implementations match ``` ### run benchmark The results below represent the time taken, with the unit in milliseconds (ms). - tp-size=1 ```shell fp8-gemm-performance-comparison-tp1: m n k tp_size DeepGEMM SGLang vLLM 0 8.0 576.0 7168.0 1.0 455.583990 360.992014 351.743996 1 16.0 576.0 7168.0 1.0 456.880003 360.320002 365.888000 2 32.0 576.0 7168.0 1.0 457.616001 365.728021 366.479993 3 64.0 576.0 7168.0 1.0 457.055986 364.064008 373.152018 4 128.0 576.0 7168.0 1.0 458.335996 370.288014 372.224003 5 256.0 576.0 7168.0 1.0 461.808026 360.431999 361.247987 6 1024.0 576.0 7168.0 1.0 449.440002 369.567990 361.775994 7 2048.0 576.0 7168.0 1.0 454.703987 428.544015 428.768009 8 4096.0 576.0 7168.0 1.0 629.408002 676.703990 676.496029 9 8.0 24576.0 7168.0 1.0 3127.487898 3158.031940 3160.160065 10 16.0 24576.0 7168.0 1.0 3146.368027 3151.232004 3151.199818 11 32.0 24576.0 7168.0 1.0 3146.368027 3152.672052 3153.632164 12 64.0 24576.0 7168.0 1.0 3147.648096 3154.031992 3156.320095 13 128.0 24576.0 7168.0 1.0 3157.472134 3226.464033 3228.352070 14 256.0 24576.0 7168.0 1.0 3185.744047 3346.208096 3345.504045 15 1024.0 24576.0 7168.0 1.0 3460.016012 4129.600048 4130.527973 16 2048.0 24576.0 7168.0 1.0 3830.528021 5256.735802 5187.120438 17 4096.0 24576.0 7168.0 1.0 4569.952011 7373.280048 7221.727848 18 8.0 32768.0 512.0 1.0 430.703998 388.063997 386.720002 19 16.0 32768.0 512.0 1.0 424.960017 385.519981 385.567993 20 32.0 32768.0 512.0 1.0 422.656000 384.416014 383.071989 21 64.0 32768.0 512.0 1.0 423.648000 386.527985 386.783987 22 128.0 32768.0 512.0 1.0 428.799987 392.960012 392.767996 23 256.0 32768.0 512.0 1.0 429.584026 413.695991 410.560012 24 1024.0 32768.0 512.0 1.0 450.271994 507.103980 506.399989 25 2048.0 32768.0 512.0 1.0 509.855986 633.855999 627.135992 26 4096.0 32768.0 512.0 1.0 627.359986 884.288013 870.975971 27 8.0 7168.0 16384.0 1.0 2122.080088 2153.632164 2154.927969 28 16.0 7168.0 16384.0 1.0 2140.799999 2157.056093 2154.128075 29 32.0 7168.0 16384.0 1.0 2137.248039 2159.327984 2160.431862 30 64.0 7168.0 16384.0 1.0 2139.647961 2167.824030 2166.416168 31 128.0 7168.0 16384.0 1.0 2153.232098 2209.856033 2210.880041 32 256.0 7168.0 16384.0 1.0 2186.304092 2311.104059 2308.095932 33 1024.0 7168.0 16384.0 1.0 2534.976006 3021.296024 3023.072004 34 2048.0 7168.0 16384.0 1.0 2903.007984 3808.959961 3800.287962 35 4096.0 7168.0 16384.0 1.0 3751.104116 5592.415810 5521.567822 36 8.0 7168.0 18432.0 1.0 2383.855820 2414.400101 2414.911985 37 16.0 7168.0 18432.0 1.0 2396.592140 2417.104006 2417.216063 38 32.0 7168.0 18432.0 1.0 2394.592047 2422.816038 2421.472073 39 64.0 7168.0 18432.0 1.0 2397.119999 2429.728031 2429.759979 40 128.0 7168.0 18432.0 1.0 2410.912037 2477.855921 2477.695942 41 256.0 7168.0 18432.0 1.0 2450.176001 2587.296009 2588.608027 42 1024.0 7168.0 18432.0 1.0 2840.303898 3396.895885 3397.215843 43 2048.0 7168.0 18432.0 1.0 3241.231918 4308.351994 4266.223907 44 4096.0 7168.0 18432.0 1.0 4199.776173 6211.935997 6199.744225 45 8.0 36864.0 7168.0 1.0 4614.079952 4690.591812 4690.383911 46 16.0 36864.0 7168.0 1.0 4653.952122 4678.495884 4678.495884 47 32.0 36864.0 7168.0 1.0 4652.991772 4677.343845 4677.279949 48 64.0 36864.0 7168.0 1.0 4652.863979 4679.615974 4679.423809 49 128.0 36864.0 7168.0 1.0 4663.584232 4769.167900 4771.056175 50 256.0 36864.0 7168.0 1.0 4708.703995 4982.272148 4981.056213 51 1024.0 36864.0 7168.0 1.0 5085.599899 6090.288162 6090.255737 52 2048.0 36864.0 7168.0 1.0 5570.047855 7588.031769 7567.200184 53 4096.0 36864.0 7168.0 1.0 6662.960052 10634.336472 10543.328285 54 8.0 24576.0 7168.0 1.0 3093.280077 3141.407967 3148.831844 55 16.0 24576.0 7168.0 1.0 3138.816118 3142.496109 3144.223928 56 32.0 24576.0 7168.0 1.0 3138.240099 3145.920038 3142.047882 57 64.0 24576.0 7168.0 1.0 3139.487982 3143.903971 3145.087957 58 128.0 24576.0 7168.0 1.0 3147.583961 3214.895964 3217.087984 59 256.0 24576.0 7168.0 1.0 3176.976204 3331.615925 3333.760023 60 1024.0 24576.0 7168.0 1.0 3449.584007 4130.335808 4130.943775 61 2048.0 24576.0 7168.0 1.0 3838.495970 5242.464066 5184.463978 62 4096.0 24576.0 7168.0 1.0 4572.256088 7338.687897 7222.464085 63 8.0 32768.0 512.0 1.0 457.520008 387.807995 388.159990 64 16.0 32768.0 512.0 1.0 446.079999 385.392010 385.648012 65 32.0 32768.0 512.0 1.0 448.255986 385.504007 384.368002 66 64.0 32768.0 512.0 1.0 448.927999 384.063989 384.128004 67 128.0 32768.0 512.0 1.0 449.328005 395.056009 393.983990 68 256.0 32768.0 512.0 1.0 452.223986 411.296010 410.879999 69 1024.0 32768.0 512.0 1.0 444.207996 505.568027 503.679991 70 2048.0 32768.0 512.0 1.0 508.704007 631.919980 626.048028 71 4096.0 32768.0 512.0 1.0 626.976013 871.392012 864.928007 72 8.0 24576.0 1536.0 1.0 744.848013 751.055956 759.680033 73 16.0 24576.0 1536.0 1.0 754.432023 754.495978 760.703981 74 32.0 24576.0 1536.0 1.0 753.952026 753.055990 759.263992 75 64.0 24576.0 1536.0 1.0 756.896019 755.487978 762.112021 76 128.0 24576.0 1536.0 1.0 758.047998 775.327981 785.600007 77 256.0 24576.0 1536.0 1.0 767.903984 796.864033 809.103966 78 1024.0 24576.0 1536.0 1.0 840.304017 979.663968 1007.007957 79 2048.0 24576.0 1536.0 1.0 935.743988 1220.864058 1255.679965 80 4096.0 24576.0 1536.0 1.0 1124.351978 1696.559906 1769.183993 81 8.0 4096.0 7168.0 1.0 612.128019 646.687984 680.431962 82 16.0 4096.0 7168.0 1.0 616.927981 650.480032 684.224010 83 32.0 4096.0 7168.0 1.0 616.415977 649.151981 685.711980 84 64.0 4096.0 7168.0 1.0 611.808002 652.304053 686.367989 85 128.0 4096.0 7168.0 1.0 619.199991 656.127989 689.119995 86 256.0 4096.0 7168.0 1.0 634.976029 667.104006 702.080011 87 1024.0 4096.0 7168.0 1.0 740.127981 850.624025 889.280021 88 2048.0 4096.0 7168.0 1.0 901.008010 1127.759933 1185.872078 89 4096.0 4096.0 7168.0 1.0 1187.872052 1645.135999 1771.263957 90 8.0 7168.0 18432.0 1.0 2385.871887 2410.048008 2415.632010 91 16.0 7168.0 18432.0 1.0 2396.800041 2417.520046 2414.864063 92 32.0 7168.0 18432.0 1.0 2395.088196 2422.528028 2422.368050 93 64.0 7168.0 18432.0 1.0 2396.640062 2431.616068 2431.551933 94 128.0 7168.0 18432.0 1.0 2412.575960 2480.351925 2480.799913 95 256.0 7168.0 18432.0 1.0 2452.095985 2589.312077 2589.375973 96 1024.0 7168.0 18432.0 1.0 2840.111971 3395.808220 3397.488117 97 2048.0 7168.0 18432.0 1.0 3251.120090 4284.751892 4273.216248 98 4096.0 7168.0 18432.0 1.0 4167.967796 6282.048225 6208.511829 99 8.0 7168.0 16384.0 1.0 2117.791891 2148.927927 2153.151989 100 16.0 7168.0 16384.0 1.0 2140.000105 2153.392076 2155.071974 101 32.0 7168.0 16384.0 1.0 2136.320114 2159.840107 2159.984112 102 64.0 7168.0 16384.0 1.0 2139.744043 2165.967941 2168.031931 103 128.0 7168.0 16384.0 1.0 2153.872013 2212.512016 2210.815907 104 256.0 7168.0 16384.0 1.0 2187.183857 2310.112000 2311.167955 105 1024.0 7168.0 16384.0 1.0 2536.432028 3023.056030 3024.911880 106 2048.0 7168.0 16384.0 1.0 2897.183895 3837.183952 3806.976080 107 4096.0 7168.0 16384.0 1.0 3717.247963 5539.231777 5519.584179 108 8.0 7168.0 2048.0 1.0 450.495988 366.656005 361.135989 109 16.0 7168.0 2048.0 1.0 448.736012 365.727991 359.120011 110 32.0 7168.0 2048.0 1.0 448.480010 360.895991 357.823998 111 64.0 7168.0 2048.0 1.0 448.127985 357.888013 361.600012 112 128.0 7168.0 2048.0 1.0 451.008022 358.047992 360.927999 113 256.0 7168.0 2048.0 1.0 451.151997 371.039987 371.407986 114 1024.0 7168.0 2048.0 1.0 455.040008 459.968001 459.679991 115 2048.0 7168.0 2048.0 1.0 450.432003 561.951995 563.103974 116 4096.0 7168.0 2048.0 1.0 578.320026 807.775974 797.504008 ``` - tp-size=8 ```shell fp8-gemm-performance-comparison-tp8: m n k tp_size DeepGEMM SGLang vLLM 0 8.0 576.0 7168.0 8.0 478.496015 384.191990 378.912002 1 16.0 576.0 7168.0 8.0 475.744009 384.415984 384.191990 2 32.0 576.0 7168.0 8.0 476.415992 384.736001 380.015999 3 64.0 576.0 7168.0 8.0 473.583996 380.144000 381.359994 4 128.0 576.0 7168.0 8.0 477.231979 386.304021 383.279979 5 256.0 576.0 7168.0 8.0 475.295991 383.904010 380.720019 6 1024.0 576.0 7168.0 8.0 474.943995 385.168016 385.663986 7 2048.0 576.0 7168.0 8.0 478.287995 428.768009 429.152012 8 4096.0 576.0 7168.0 8.0 631.232023 681.855977 683.744013 9 8.0 24576.0 7168.0 8.0 3135.999918 3153.343916 3149.183989 10 16.0 24576.0 7168.0 8.0 3136.607885 3140.959978 3143.840075 11 32.0 24576.0 7168.0 8.0 3137.120008 3145.215988 3141.488075 12 64.0 24576.0 7168.0 8.0 3141.472101 3146.080017 3147.007942 13 128.0 24576.0 7168.0 8.0 3149.119854 3218.351841 3218.495846 14 256.0 24576.0 7168.0 8.0 3180.943966 3338.752031 3338.848114 15 1024.0 24576.0 7168.0 8.0 3455.071926 4132.575989 4132.031918 16 2048.0 24576.0 7168.0 8.0 3837.728024 5242.000103 5186.160088 17 4096.0 24576.0 7168.0 8.0 4577.280045 7338.047981 7265.632153 18 8.0 32768.0 512.0 8.0 470.207989 387.616009 386.687994 19 16.0 32768.0 512.0 8.0 471.487999 384.351999 379.696012 20 32.0 32768.0 512.0 8.0 472.671986 384.864002 385.136008 21 64.0 32768.0 512.0 8.0 473.327994 386.335999 388.704002 22 128.0 32768.0 512.0 8.0 474.624008 396.800011 393.952012 23 256.0 32768.0 512.0 8.0 470.799983 411.743999 409.247994 24 1024.0 32768.0 512.0 8.0 470.560014 509.296000 508.080006 25 2048.0 32768.0 512.0 8.0 509.151995 629.055977 628.351986 26 4096.0 32768.0 512.0 8.0 629.760027 886.111975 868.864000 27 8.0 7168.0 16384.0 8.0 2112.319946 2149.775982 2158.063889 28 16.0 7168.0 16384.0 8.0 2146.751881 2162.544012 2161.551952 29 32.0 7168.0 16384.0 8.0 2142.751932 2165.823936 2167.504072 30 64.0 7168.0 16384.0 8.0 2144.144058 2171.008110 2173.488140 31 128.0 7168.0 16384.0 8.0 2161.600113 2218.240023 2220.191956 32 256.0 7168.0 16384.0 8.0 2195.359945 2319.648027 2322.224140 33 1024.0 7168.0 16384.0 8.0 2540.944099 3041.167974 3032.768011 34 2048.0 7168.0 16384.0 8.0 2914.079905 3853.408098 3813.920021 35 4096.0 7168.0 16384.0 8.0 3748.720169 5609.439850 5589.791775 36 8.0 7168.0 18432.0 8.0 2388.911963 2412.415981 2414.047956 37 16.0 7168.0 18432.0 8.0 2394.703865 2411.087990 2419.296026 38 32.0 7168.0 18432.0 8.0 2399.872065 2429.023981 2426.431894 39 64.0 7168.0 18432.0 8.0 2402.192116 2438.463926 2437.727928 40 128.0 7168.0 18432.0 8.0 2421.983957 2489.536047 2489.903927 41 256.0 7168.0 18432.0 8.0 2459.872007 2600.800037 2600.032091 42 1024.0 7168.0 18432.0 8.0 2841.232061 3394.383907 3390.416145 43 2048.0 7168.0 18432.0 8.0 3259.903908 4320.672035 4282.527924 44 4096.0 7168.0 18432.0 8.0 4199.071884 6304.543972 6224.607944 45 8.0 4608.0 7168.0 8.0 680.768013 689.055979 689.872026 46 16.0 4608.0 7168.0 8.0 684.480011 693.567991 692.095995 47 32.0 4608.0 7168.0 8.0 682.879984 693.104029 693.359971 48 64.0 4608.0 7168.0 8.0 681.952000 696.287990 696.928024 49 128.0 4608.0 7168.0 8.0 688.480020 715.071976 714.655995 50 256.0 4608.0 7168.0 8.0 703.552008 746.656001 743.903995 51 1024.0 4608.0 7168.0 8.0 820.864022 958.368003 955.376029 52 2048.0 4608.0 7168.0 8.0 1001.935959 1281.056046 1268.656015 53 4096.0 4608.0 7168.0 8.0 1275.552034 1807.456017 1790.607929 54 8.0 3072.0 7168.0 8.0 496.223986 499.055982 500.383973 55 16.0 3072.0 7168.0 8.0 496.623993 498.111993 501.631975 56 32.0 3072.0 7168.0 8.0 494.623989 503.407955 501.536012 57 64.0 3072.0 7168.0 8.0 492.671996 504.320025 504.224002 58 128.0 3072.0 7168.0 8.0 497.759998 513.023973 512.000024 59 256.0 3072.0 7168.0 8.0 512.000024 539.232016 538.319945 60 1024.0 3072.0 7168.0 8.0 613.759995 703.296006 703.136027 61 2048.0 3072.0 7168.0 8.0 755.584002 924.704015 922.288001 62 4096.0 3072.0 7168.0 8.0 1022.896051 1363.968015 1364.223957 63 8.0 4096.0 512.0 8.0 473.327994 363.263994 376.224011 64 16.0 4096.0 512.0 8.0 471.455991 362.240016 362.271994 65 32.0 4096.0 512.0 8.0 470.048010 360.799998 361.248016 66 64.0 4096.0 512.0 8.0 466.655999 360.064000 357.984006 67 128.0 4096.0 512.0 8.0 470.256001 376.320004 376.367986 68 256.0 4096.0 512.0 8.0 469.983995 382.400006 378.544003 69 1024.0 4096.0 512.0 8.0 476.639986 379.168004 380.095989 70 2048.0 4096.0 512.0 8.0 472.975999 382.959992 383.776009 71 4096.0 4096.0 512.0 8.0 474.496007 385.951996 385.744005 72 8.0 3072.0 1536.0 8.0 473.536015 374.480009 359.696001 73 16.0 3072.0 1536.0 8.0 472.223997 362.176001 358.047992 74 32.0 3072.0 1536.0 8.0 476.191998 370.415986 369.664013 75 64.0 3072.0 1536.0 8.0 476.352006 365.568012 373.360008 76 128.0 3072.0 1536.0 8.0 488.207996 379.648000 367.935985 77 256.0 3072.0 1536.0 8.0 479.360014 365.040004 378.688008 78 1024.0 3072.0 1536.0 8.0 475.152016 390.864015 383.616000 79 2048.0 3072.0 1536.0 8.0 481.391996 392.351985 391.968012 80 4096.0 3072.0 1536.0 8.0 487.088025 388.448000 386.976004 81 8.0 512.0 7168.0 8.0 478.944004 383.424014 372.047991 82 16.0 512.0 7168.0 8.0 475.232005 382.719994 377.552003 83 32.0 512.0 7168.0 8.0 475.695997 377.407998 385.679990 84 64.0 512.0 7168.0 8.0 476.368010 382.975996 377.631992 85 128.0 512.0 7168.0 8.0 482.656002 380.928010 384.496003 86 256.0 512.0 7168.0 8.0 476.159990 386.047989 382.192016 87 1024.0 512.0 7168.0 8.0 480.176002 385.199994 387.551993 88 2048.0 512.0 7168.0 8.0 481.743991 400.319993 426.144004 89 4096.0 512.0 7168.0 8.0 584.336042 629.440010 646.975994 90 8.0 7168.0 2304.0 8.0 479.312003 382.560015 461.791992 91 16.0 7168.0 2304.0 8.0 482.912004 383.935988 382.016003 92 32.0 7168.0 2304.0 8.0 479.360014 379.999995 380.335987 93 64.0 7168.0 2304.0 8.0 476.624012 380.383998 380.255997 94 128.0 7168.0 2304.0 8.0 479.743987 389.472008 393.599987 95 256.0 7168.0 2304.0 8.0 478.336006 407.375991 408.335984 96 1024.0 7168.0 2304.0 8.0 483.168006 504.688025 504.607975 97 2048.0 7168.0 2304.0 8.0 501.504004 623.488009 625.760019 98 4096.0 7168.0 2304.0 8.0 635.775983 886.687994 896.160007 99 8.0 7168.0 2048.0 8.0 480.463982 390.592009 392.560005 100 16.0 7168.0 2048.0 8.0 480.991989 392.336011 385.424018 101 32.0 7168.0 2048.0 8.0 478.703976 390.751988 386.896014 102 64.0 7168.0 2048.0 8.0 481.503993 390.959978 397.664011 103 128.0 7168.0 2048.0 8.0 484.239995 394.591987 398.111999 104 256.0 7168.0 2048.0 8.0 500.415981 408.383995 404.127985 105 1024.0 7168.0 2048.0 8.0 501.375973 462.399989 461.088002 106 2048.0 7168.0 2048.0 8.0 507.647991 566.383958 567.407966 107 4096.0 7168.0 2048.0 8.0 578.320026 805.904031 798.207998 108 8.0 7168.0 256.0 8.0 505.280018 397.215992 392.672002 109 16.0 7168.0 256.0 8.0 494.751990 395.040005 392.416000 110 32.0 7168.0 256.0 8.0 496.767998 393.279999 394.751996 111 64.0 7168.0 256.0 8.0 497.664005 393.984020 391.488016 112 128.0 7168.0 256.0 8.0 504.159987 395.904005 397.920012 113 256.0 7168.0 256.0 8.0 497.824013 399.711996 393.599987 114 1024.0 7168.0 256.0 8.0 499.552011 398.847997 375.871986 115 2048.0 7168.0 256.0 8.0 474.063993 369.343996 369.055986 116 4096.0 7168.0 256.0 8.0 478.015989 372.720003 370.527983 ``` In h200: - tp=8 ```shell fp8-gemm-performance-comparison-tp8: m n k tp_size DeepGEMM SGLang vLLM 0 8.0 576.0 7168.0 8.0 224.127993 170.111999 172.575995 1 16.0 576.0 7168.0 8.0 222.240001 171.008006 171.519995 2 32.0 576.0 7168.0 8.0 222.624004 171.455994 171.039999 3 64.0 576.0 7168.0 8.0 221.855998 170.368001 169.919997 4 128.0 576.0 7168.0 8.0 221.231997 179.104000 179.168001 5 256.0 576.0 7168.0 8.0 220.479995 190.975994 189.472005 6 1024.0 576.0 7168.0 8.0 253.311992 281.823993 281.599998 7 2048.0 576.0 7168.0 8.0 370.368004 405.503988 407.328010 8 4096.0 576.0 7168.0 8.0 579.487979 642.207980 639.567971 9 8.0 24576.0 7168.0 8.0 2901.792049 2917.887926 2919.071913 10 16.0 24576.0 7168.0 8.0 2891.839981 2910.655975 2909.183979 11 32.0 24576.0 7168.0 8.0 2891.007900 2910.207987 2910.239935 12 64.0 24576.0 7168.0 8.0 2889.663935 2909.984112 2911.231995 13 128.0 24576.0 7168.0 8.0 2898.240089 2984.416008 2986.080170 14 256.0 24576.0 7168.0 8.0 2934.623957 3105.184078 3103.264093 15 1024.0 24576.0 7168.0 8.0 3208.287954 3917.695999 3919.359922 16 2048.0 24576.0 7168.0 8.0 3597.023964 5002.848148 5005.407810 17 4096.0 24576.0 7168.0 8.0 4326.303959 7080.800056 7089.375973 18 8.0 32768.0 512.0 8.0 359.344006 354.431987 354.431987 19 16.0 32768.0 512.0 8.0 359.679997 356.543988 356.207997 20 32.0 32768.0 512.0 8.0 360.368013 354.672015 354.752004 21 64.0 32768.0 512.0 8.0 360.480011 356.943995 356.976002 22 128.0 32768.0 512.0 8.0 363.456011 366.207987 365.568012 23 256.0 32768.0 512.0 8.0 370.624006 382.048011 382.463992 24 1024.0 32768.0 512.0 8.0 420.320004 479.584008 479.167998 25 2048.0 32768.0 512.0 8.0 482.176006 602.815986 602.976024 26 4096.0 32768.0 512.0 8.0 607.743979 854.240000 852.096021 27 8.0 7168.0 16384.0 8.0 1983.152032 1996.079922 1996.160030 28 16.0 7168.0 16384.0 8.0 1981.760025 1998.687983 1998.208046 29 32.0 7168.0 16384.0 8.0 1976.079941 2001.136065 2002.592087 30 64.0 7168.0 16384.0 8.0 1968.703985 2006.432056 2007.040024 31 128.0 7168.0 16384.0 8.0 1986.239910 2054.944038 2054.175854 32 256.0 7168.0 16384.0 8.0 2024.544001 2151.535988 2153.055906 33 1024.0 7168.0 16384.0 8.0 2354.367971 2859.087944 2861.583948 34 2048.0 7168.0 16384.0 8.0 2713.248014 3651.167870 3646.016121 35 4096.0 7168.0 16384.0 8.0 3503.871918 5345.327854 5352.416039 36 8.0 7168.0 18432.0 8.0 2219.264030 2236.991882 2236.704111 37 16.0 7168.0 18432.0 8.0 2211.999893 2240.128040 2237.535954 38 32.0 7168.0 18432.0 8.0 2207.520008 2240.832090 2240.704060 39 64.0 7168.0 18432.0 8.0 2207.551956 2250.319958 2249.792099 40 128.0 7168.0 18432.0 8.0 2224.832058 2301.199913 2302.623987 41 256.0 7168.0 18432.0 8.0 2267.424107 2410.992146 2410.416126 42 1024.0 7168.0 18432.0 8.0 2636.623859 3206.624031 3205.568075 43 2048.0 7168.0 18432.0 8.0 3037.007809 4082.544327 4087.632179 44 4096.0 7168.0 18432.0 8.0 3923.712015 6007.535934 6005.424023 45 8.0 4608.0 7168.0 8.0 635.136008 637.983978 638.128042 46 16.0 4608.0 7168.0 8.0 630.079985 638.848007 638.912022 47 32.0 4608.0 7168.0 8.0 629.279971 640.511990 641.439974 48 64.0 4608.0 7168.0 8.0 626.655996 643.263996 644.320011 49 128.0 4608.0 7168.0 8.0 633.872032 661.311984 662.016034 50 256.0 4608.0 7168.0 8.0 647.871971 694.176018 693.343997 51 1024.0 4608.0 7168.0 8.0 761.568010 902.032018 902.480006 52 2048.0 4608.0 7168.0 8.0 931.904018 1210.719943 1212.543964 53 4096.0 4608.0 7168.0 8.0 1198.224068 1722.336054 1723.551989 54 8.0 3072.0 7168.0 8.0 463.663995 461.504012 462.336004 55 16.0 3072.0 7168.0 8.0 457.376003 463.808000 461.551994 56 32.0 3072.0 7168.0 8.0 456.831992 466.176003 466.048002 57 64.0 3072.0 7168.0 8.0 453.520000 468.511999 467.967987 58 128.0 3072.0 7168.0 8.0 460.960001 477.456003 477.551997 59 256.0 3072.0 7168.0 8.0 473.567992 502.879977 501.520038 60 1024.0 3072.0 7168.0 8.0 569.696009 662.495971 662.591994 61 2048.0 3072.0 7168.0 8.0 703.439951 875.584006 876.991987 62 4096.0 3072.0 7168.0 8.0 958.591998 1310.176015 1310.783982 63 8.0 4096.0 512.0 8.0 227.903992 170.368001 168.704003 64 16.0 4096.0 512.0 8.0 224.848002 170.527995 170.111999 65 32.0 4096.0 512.0 8.0 225.375995 170.320004 169.263989 66 64.0 4096.0 512.0 8.0 224.672005 170.175999 169.568002 67 128.0 4096.0 512.0 8.0 227.008000 171.488002 170.911998 68 256.0 4096.0 512.0 8.0 227.039993 172.031999 170.607999 69 1024.0 4096.0 512.0 8.0 227.520004 172.864005 172.800004 70 2048.0 4096.0 512.0 8.0 227.039993 175.264001 174.976006 71 4096.0 4096.0 512.0 8.0 227.487996 185.471997 186.751992 72 8.0 3072.0 1536.0 8.0 227.295995 170.847997 170.016006 73 16.0 3072.0 1536.0 8.0 226.480007 171.136007 170.240000 74 32.0 3072.0 1536.0 8.0 225.567997 171.296000 170.767993 75 64.0 3072.0 1536.0 8.0 226.815999 171.391994 170.975998 76 128.0 3072.0 1536.0 8.0 227.648005 171.519995 170.144007 77 256.0 3072.0 1536.0 8.0 225.295991 168.640003 168.768004 78 1024.0 3072.0 1536.0 8.0 224.240005 191.520005 193.279997 79 2048.0 3072.0 1536.0 8.0 228.992000 238.368005 239.071995 80 4096.0 3072.0 1536.0 8.0 270.399988 336.863995 340.063989 81 8.0 512.0 7168.0 8.0 226.159990 175.871998 175.056010 82 16.0 512.0 7168.0 8.0 228.720009 175.200000 173.951998 83 32.0 512.0 7168.0 8.0 228.799999 177.279994 176.159993 84 64.0 512.0 7168.0 8.0 229.552001 177.215993 177.088007 85 128.0 512.0 7168.0 8.0 228.480011 175.487995 175.136000 86 256.0 512.0 7168.0 8.0 228.031993 176.768005 176.512003 87 1024.0 512.0 7168.0 8.0 235.072002 251.040012 250.079989 88 2048.0 512.0 7168.0 8.0 354.752004 371.616006 374.112010 89 4096.0 512.0 7168.0 8.0 541.599989 589.119971 589.887977 90 8.0 7168.0 2304.0 8.0 357.951999 358.128011 357.919991 91 16.0 7168.0 2304.0 8.0 357.551992 356.992006 357.248008 92 32.0 7168.0 2304.0 8.0 357.823998 358.720005 358.687997 93 64.0 7168.0 2304.0 8.0 357.088000 358.080000 358.271986 94 128.0 7168.0 2304.0 8.0 358.880013 365.359992 364.111990 95 256.0 7168.0 2304.0 8.0 365.247995 377.247989 377.664000 96 1024.0 7168.0 2304.0 8.0 411.231995 476.767987 477.120012 97 2048.0 7168.0 2304.0 8.0 468.607992 599.168003 599.135995 98 4096.0 7168.0 2304.0 8.0 596.768022 854.687989 856.064022 99 8.0 7168.0 2048.0 8.0 324.864000 321.056008 320.767999 100 16.0 7168.0 2048.0 8.0 325.888008 323.552012 324.384004 101 32.0 7168.0 2048.0 8.0 324.784011 323.583990 324.256003 102 64.0 7168.0 2048.0 8.0 325.152010 325.152010 325.760007 103 128.0 7168.0 2048.0 8.0 326.687992 329.407990 331.328005 104 256.0 7168.0 2048.0 8.0 333.824009 343.952000 345.376015 105 1024.0 7168.0 2048.0 8.0 375.088006 434.080005 434.175998 106 2048.0 7168.0 2048.0 8.0 427.807987 537.984014 540.000021 107 4096.0 7168.0 2048.0 8.0 542.783976 768.224001 769.135952 108 8.0 7168.0 256.0 8.0 231.728002 174.879998 173.311993 109 16.0 7168.0 256.0 8.0 230.591998 173.087999 173.792005 110 32.0 7168.0 256.0 8.0 232.352003 176.512003 175.487995 111 64.0 7168.0 256.0 8.0 230.368003 175.648004 175.807998 112 128.0 7168.0 256.0 8.0 231.440008 175.280005 174.559996 113 256.0 7168.0 256.0 8.0 231.488004 175.280005 174.559996 114 1024.0 7168.0 256.0 8.0 230.031997 178.080007 178.640008 115 2048.0 7168.0 256.0 8.0 234.463990 180.160001 180.928007 116 4096.0 7168.0 256.0 8.0 231.840000 180.895999 180.99200 ``` - tp=16 ```shell fp8-gemm-performance-comparison-tp16: m n k tp_size DeepGEMM SGLang vLLM 0 8.0 576.0 7168.0 16.0 234.512001 179.104000 179.232001 1 16.0 576.0 7168.0 16.0 229.312003 179.376006 176.927999 2 32.0 576.0 7168.0 16.0 231.056005 179.872006 180.800006 3 64.0 576.0 7168.0 16.0 228.399992 178.479999 177.568004 4 128.0 576.0 7168.0 16.0 226.768002 177.151993 176.512003 5 256.0 576.0 7168.0 16.0 228.351995 189.088002 190.559998 6 1024.0 576.0 7168.0 16.0 250.800014 280.223995 280.128002 7 2048.0 576.0 7168.0 16.0 369.536012 404.352009 407.231987 8 4096.0 576.0 7168.0 16.0 578.256011 639.840007 638.656020 9 8.0 24576.0 7168.0 16.0 2900.063992 2914.688110 2917.311907 10 16.0 24576.0 7168.0 16.0 2890.399933 2907.840014 2908.159971 11 32.0 24576.0 7168.0 16.0 2889.823914 2907.648087 2907.007933 12 64.0 24576.0 7168.0 16.0 2889.568090 2909.696102 2908.351898 13 128.0 24576.0 7168.0 16.0 2897.088051 2983.824015 2984.224081 14 256.0 24576.0 7168.0 16.0 2934.079885 3103.647947 3103.552103 15 1024.0 24576.0 7168.0 16.0 3206.416130 3919.104099 3918.879986 16 2048.0 24576.0 7168.0 16.0 3596.384048 5003.456116 5001.311779 17 4096.0 24576.0 7168.0 16.0 4326.864243 7081.632137 7091.807842 18 8.0 32768.0 512.0 16.0 357.712001 353.280008 353.376001 19 16.0 32768.0 512.0 16.0 359.232008 354.416013 354.207993 20 32.0 32768.0 512.0 16.0 358.783990 353.408009 353.408009 21 64.0 32768.0 512.0 16.0 358.848006 353.599995 353.664011 22 128.0 32768.0 512.0 16.0 361.279994 364.800006 362.415999 23 256.0 32768.0 512.0 16.0 368.719995 379.487991 381.632000 24 1024.0 32768.0 512.0 16.0 418.352008 477.824003 477.631986 25 2048.0 32768.0 512.0 16.0 480.544001 603.519976 601.087987 26 4096.0 32768.0 512.0 16.0 605.823994 852.064013 851.391971 27 8.0 2304.0 7168.0 16.0 368.192017 370.463997 371.551991 28 16.0 2304.0 7168.0 16.0 364.127994 371.151984 373.600006 29 32.0 2304.0 7168.0 16.0 363.040000 374.464005 375.216007 30 64.0 2304.0 7168.0 16.0 362.848014 378.095984 377.664000 31 128.0 2304.0 7168.0 16.0 371.039987 386.079997 386.112005 32 256.0 2304.0 7168.0 16.0 384.095997 413.727999 414.463997 33 1024.0 2304.0 7168.0 16.0 474.624008 548.031986 549.983978 34 2048.0 2304.0 7168.0 16.0 606.912017 746.208012 748.799980 35 4096.0 2304.0 7168.0 16.0 864.160001 1134.799957 1137.199998 36 8.0 1536.0 7168.0 16.0 272.255987 278.816015 278.943986 37 16.0 1536.0 7168.0 16.0 269.760013 279.040009 281.280011 38 32.0 1536.0 7168.0 16.0 268.415987 283.199996 283.199996 39 64.0 1536.0 7168.0 16.0 267.967999 284.976006 285.376012 40 128.0 1536.0 7168.0 16.0 275.792003 292.127997 289.407998 41 256.0 1536.0 7168.0 16.0 289.887995 304.991990 305.023998 42 1024.0 1536.0 7168.0 16.0 369.311988 409.568012 411.455989 43 2048.0 1536.0 7168.0 16.0 494.271994 586.943984 587.296009 44 4096.0 1536.0 7168.0 16.0 708.384037 878.383994 879.487991 45 8.0 2048.0 512.0 16.0 229.983985 176.223993 174.303994 46 16.0 2048.0 512.0 16.0 228.095993 174.079999 173.472002 47 32.0 2048.0 512.0 16.0 227.968007 172.928005 173.615992 48 64.0 2048.0 512.0 16.0 229.248002 174.336001 174.048007 49 128.0 2048.0 512.0 16.0 230.463997 175.136000 177.199990 50 256.0 2048.0 512.0 16.0 229.279995 177.487999 175.999999 51 1024.0 2048.0 512.0 16.0 230.463997 173.375994 174.208000 52 2048.0 2048.0 512.0 16.0 228.735998 176.448002 175.840005 53 4096.0 2048.0 512.0 16.0 228.848010 180.127993 178.368002 54 8.0 1536.0 1536.0 16.0 227.712005 170.144007 172.160000 55 16.0 1536.0 1536.0 16.0 230.720013 171.792001 171.583995 56 32.0 1536.0 1536.0 16.0 229.727998 171.232000 170.767993 57 64.0 1536.0 1536.0 16.0 228.624001 171.488002 171.327993 58 128.0 1536.0 1536.0 16.0 229.376003 170.736000 171.263993 59 256.0 1536.0 1536.0 16.0 230.704010 171.232000 171.568006 60 1024.0 1536.0 1536.0 16.0 228.479996 173.503995 173.600003 61 2048.0 1536.0 1536.0 16.0 228.607997 181.600004 184.064001 62 4096.0 1536.0 1536.0 16.0 229.535997 245.407999 250.272006 63 8.0 256.0 7168.0 16.0 230.048001 173.759997 173.600003 64 16.0 256.0 7168.0 16.0 228.992000 173.951998 173.087999 65 32.0 256.0 7168.0 16.0 230.399996 173.728004 173.664004 66 64.0 256.0 7168.0 16.0 228.848010 174.848005 173.951998 67 128.0 256.0 7168.0 16.0 230.976000 173.216000 173.887998 68 256.0 256.0 7168.0 16.0 231.616005 174.383998 185.791999 69 1024.0 256.0 7168.0 16.0 229.824007 241.632000 269.632012 70 2048.0 256.0 7168.0 16.0 312.032014 348.639995 376.320004 71 4096.0 256.0 7168.0 16.0 517.184019 536.768019 565.999985 ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-26 21:14:53 +00:00

closed this issue
added the
pull-request
label

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

deepseek-ai/DeepGEMM#209

No description provided.

Rows
Columns