[GH-ISSUE #1176 ] [Bug] v0.2.4 启动模型失败 #810

Author

Owner

@kitchen7c commented on GitHub (Apr 21, 2025):

我也遇到同样的问题 CUDA12.4 torch2.6.0

@kitchen7c commented on GitHub (Apr 21, 2025): 我也遇到同样的问题 CUDA12.4 torch2.6.0

bsakaguchi commented

Author

Owner

@qiyuxinlin commented on GitHub (Apr 22, 2025):

重复日志打印在 main 上已经删掉了，但是 MLA 报错看起来是 flashinfer 出错，想问下您 flashinfer 如何安装的？

@qiyuxinlin commented on GitHub (Apr 22, 2025): 重复日志打印在 main 上已经删掉了，但是 MLA 报错看起来是 flashinfer 出错，想问下您 flashinfer 如何安装的？

bsakaguchi commented

Author

Owner

@Mewral commented on GitHub (Apr 22, 2025):

@qiyuxinlin 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，https://github.com/flashinfer-ai/flashinfer/issues/892

@Mewral commented on GitHub (Apr 22, 2025): @qiyuxinlin 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，https://github.com/flashinfer-ai/flashinfer/issues/892

bsakaguchi commented

Author

Owner

@kitchen7c commented on GitHub (Apr 22, 2025):

重复日志打印在 main 上已经删掉了，但是 MLA 报错看起来是 flashinfer 出错，想问下您 flashinfer 如何安装的？
我就是用脚本安装的。除了 pip 换成国内的源。

@kitchen7c commented on GitHub (Apr 22, 2025): > 重复日志打印在 main 上已经删掉了，但是 MLA 报错看起来是 flashinfer 出错，想问下您 flashinfer 如何安装的？我就是用脚本安装的。除了 pip 换成国内的源。

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (Apr 22, 2025):

@qiyuxinlin 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，flashinfer-ai/flashinfer#892

custom_flashinfer不是安装main分支，而是安装fix-precision-mla-merge-main或者GQA_var_batch分支

@Atream commented on GitHub (Apr 22, 2025): > [@qiyuxinlin](https://github.com/qiyuxinlin) 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，[flashinfer-ai/flashinfer#892](https://github.com/flashinfer-ai/flashinfer/issues/892) custom_flashinfer不是安装main分支，而是安装fix-precision-mla-merge-main或者GQA_var_batch分支

bsakaguchi commented

Author

Owner

@Mewral commented on GitHub (Apr 22, 2025):

@qiyuxinlin 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，flashinfer-ai/flashinfer#892

custom_flashinfer不是安装main分支，而是安装fix-precision-mla-merge-main或者GQA_var_batch分支

感谢，我之前安装的是v0.2.4 代码里连接的分支，我再试下

@Mewral commented on GitHub (Apr 22, 2025): > > [@qiyuxinlin](https://github.com/qiyuxinlin) 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，[flashinfer-ai/flashinfer#892](https://github.com/flashinfer-ai/flashinfer/issues/892) > > custom_flashinfer不是安装main分支，而是安装fix-precision-mla-merge-main或者GQA_var_batch分支感谢，我之前安装的是v0.2.4 代码里连接的分支，我再试下

bsakaguchi commented

Author

Owner

@kitchen7c commented on GitHub (Apr 22, 2025):

@qiyuxinlin 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，flashinfer-ai/flashinfer#892

custom_flashinfer不是安装main分支，而是安装fix-precision-mla-merge-main或者GQA_var_batch分支

我安装是从fix-precision-mla-merge-main 分支同步的。但还是启动是报下面的错误。

loading blk.58.attn_norm.weight to cuda
loading blk.58.ffn_norm.weight to cuda
loading blk.59.attn_q_a_norm.weight to cuda
loading blk.59.attn_kv_a_norm.weight to cuda
loading blk.59.attn_kv_b.weight to cuda:0
loading blk.59.attn_norm.weight to cuda
loading blk.59.ffn_norm.weight to cuda
loading blk.60.attn_q_a_norm.weight to cuda
loading blk.60.attn_kv_a_norm.weight to cuda
loading blk.60.attn_kv_b.weight to cuda:0
loading blk.60.attn_norm.weight to cuda
loading blk.60.ffn_norm.weight to cuda
loading output_norm.weight to cuda
Getting inference context from sched_client.
Got inference context, sending it to subscribers.
Rebuilding kvcache
kv_cache loaded successfully.
capturing cuda graph 1 1
Process SpawnProcess-1:
Traceback (most recent call last):
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2209, in _run_ninja_build
subprocess.run(
File "/data/miniconda3/envs/ktransformers/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data/miniconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/data/miniconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 253, in run_engine
engine.model_runner.warmup()
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 107, in warmup
self.model.flash_infer_attn_plan(self.input[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf,
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 145, in flash_infer_attn_plan
self.wrapper.plan(minibatch.q_indptr, minibatch.kv_indptr, minibatch.kv_indices,
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/mla.py", line 232, in plan
self._cached_module = get_batch_mla_module(self._backend)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/mla.py", line 43, in backend_module
modules_dict[args] = gen_batch_mla_module(backend, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/jit/attention/pytorch.py", line 185, in gen_batch_mla_module
return load_cuda_ops(
^^^^^^^^^^^^^^
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/jit/core.py", line 123, in load_cuda_ops
torch_cpp_ext.load(
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1380, in load
return _jit_compile(
^^^^^^^^^^^^^
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1798, in _jit_compile
_write_ninja_file_and_build_library(
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1926, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2225, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90': [1/4] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_plan.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu -o batch_mla_sm90_plan.cuda.o
FAILED: batch_mla_sm90_plan.cuda.o
/usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_plan.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu -o batch_mla_sm90_plan.cuda.o
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu:19:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu:19:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu:19:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu(44): error: no instance of function template "flashinfer::MLAPlan" matches the argument list
argument types are: (void *, size_t, void *, void *, size_t, flashinfer::MLAPlanInfo, IdType *, IdType *, IdType , int, int64_t, int64_t, bool, const cudaStream_t)
MLAPlan(float_workspace_buffer.data_ptr(), float_workspace_size_in_bytes,
^
/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include/flashinfer/attention/scheduler.cuh(1071): note #3322-D: number of parameters of function template "flashinfer::MLAPlan" does not match the call
inline cudaError_t MLAPlan(void float_buffer, size_t float_workspace_size_in_bytes,
^

1 error detected in the compilation of "/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu".
[2/4] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_run.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu -o batch_mla_sm90_run.cuda.o
FAILED: batch_mla_sm90_run.cuda.o
/usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_run.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu -o batch_mla_sm90_run.cuda.o
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu:21:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu:21:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu:21:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu(65): error: a value of type "DTypeO " cannot be assigned to an entity of type "float "
switch (mask_mode) { case MaskMode::kNone: { constexpr MaskMode MASK_MODE = MaskMode::kNone; { using Params = MLAParams<DTypeQ, DTypeKV, DTypeO, IdType>; [&] { Params params; params.q_nope = static_cast<DTypeQ>(q_nope.data_ptr()); params.q_pe = static_cast<DTypeQ>(q_pe.data_ptr()); params.ckv = static_cast<DTypeKV*>(ckv_cache.data_ptr()); params.kpe = static_cast<DTypeKV*>(kpe_cache.data_ptr()); params.q_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_indptr_offset); params.kv_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_indptr_offset); params.partial_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.partial_indptr_offset); params.kv_indices = static_cast<IdType*>(kv_indices.data_ptr()); params.q_len = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_len_offset); params.kv_len = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_len_offset); params.q_start = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_start_offset); params.kv_start = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_start_offset); params.kv_end = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_end_offset); params.work_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.work_indptr_offset); params.merge_packed_offset_start = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_packed_offset_start_offset); params.merge_packed_offset_end = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.merge_packed_offset_end_offset); params.merge_partial_packed_offset_start = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_partial_packed_offset_start_offset); params.merge_partial_packed_offset_end = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_partial_packed_offset_end_offset); params.merge_partial_stride = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.merge_partial_stride_offset); params.final_o = static_cast<DTypeO*>(o.data_ptr()); params.final_lse = maybe_lse.has_value() ? static_cast<float*>(maybe_lse->data_ptr()) : nullptr; params.partial_o = GetPtrFromBaseOffset(float_buffer_ptr, plan_info.partial_o_offset); params.partial_lse = GetPtrFromBaseOffset(float_buffer_ptr, plan_info.partial_lse_offset); params.num_heads = uint_fastdiv(num_heads); params.block_size = uint_fastdiv(page_size); params.q_nope_stride_n = q_nope_stride_n; params.q_nope_stride_h = q_nope_stride_h; params.q_pe_stride_n = q_pe_stride_n; params.q_pe_stride_h = q_pe_stride_h; params.ckv_stride_page = ckv_stride_page; params.ckv_stride_n = ckv_stride_n; params.kpe_stride_page = kpe_stride_page; params.kpe_stride_n = kpe_stride_n; params.o_stride_n = o_stride_n; params.o_stride_h = o_stride_h; params.sm_scale = sm_scale; cudaError_t status = mla::BatchMLAPageAttentionHopper<MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE>( params, plan_info.num_blks_x, plan_info.num_blks_y, stream);
^

/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu(65): error: a value of type "DTypeO " cannot be assigned to an entity of type "float "
; }(); } break; } case MaskMode::kCausal: { constexpr MaskMode MASK_MODE = MaskMode::kCausal; { using Params = MLAParams<DTypeQ, DTypeKV, DTypeO, IdType>; [&] { Params params; params.q_nope = static_cast<DTypeQ>(q_nope.data_ptr()); params.q_pe = static_cast<DTypeQ>(q_pe.data_ptr()); params.ckv = static_cast<DTypeKV*>(ckv_cache.data_ptr()); params.kpe = static_cast<DTypeKV*>(kpe_cache.data_ptr()); params.q_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_indptr_offset); params.kv_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_indptr_offset); params.partial_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.partial_indptr_offset); params.kv_indices = static_cast<IdType*>(kv_indices.data_ptr()); params.q_len = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_len_offset); params.kv_len = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_len_offset); params.q_start = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_start_offset); params.kv_start = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_start_offset); params.kv_end = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_end_offset); params.work_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.work_indptr_offset); params.merge_packed_offset_start = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_packed_offset_start_offset); params.merge_packed_offset_end = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.merge_packed_offset_end_offset); params.merge_partial_packed_offset_start = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_partial_packed_offset_start_offset); params.merge_partial_packed_offset_end = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_partial_packed_offset_end_offset); params.merge_partial_stride = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.merge_partial_stride_offset); params.final_o = static_cast<DTypeO*>(o.data_ptr()); params.final_lse = maybe_lse.has_value() ? static_cast<float*>(maybe_lse->data_ptr()) : nullptr; params.partial_o = GetPtrFromBaseOffset(float_buffer_ptr, plan_info.partial_o_offset); params.partial_lse = GetPtrFromBaseOffset(float_buffer_ptr, plan_info.partial_lse_offset); params.num_heads = uint_fastdiv(num_heads); params.block_size = uint_fastdiv(page_size); params.q_nope_stride_n = q_nope_stride_n; params.q_nope_stride_h = q_nope_stride_h; params.q_pe_stride_n = q_pe_stride_n; params.q_pe_stride_h = q_pe_stride_h; params.ckv_stride_page = ckv_stride_page; params.ckv_stride_n = ckv_stride_n; params.kpe_stride_page = kpe_stride_page; params.kpe_stride_n = kpe_stride_n; params.o_stride_n = o_stride_n; params.o_stride_h = o_stride_h; params.sm_scale = sm_scale; cudaError_t status = mla::BatchMLAPageAttentionHopper<MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE>( params, plan_info.num_blks_x, plan_info.num_blks_y, stream);
^

/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu(65): error: a value of type "DTypeO " cannot be assigned to an entity of type "float "
; }(); } break; } case MaskMode::kCustom: { constexpr MaskMode MASK_MODE = MaskMode::kCustom; { using Params = MLAParams<DTypeQ, DTypeKV, DTypeO, IdType>; [&] { Params params; params.q_nope = static_cast<DTypeQ>(q_nope.data_ptr()); params.q_pe = static_cast<DTypeQ>(q_pe.data_ptr()); params.ckv = static_cast<DTypeKV*>(ckv_cache.data_ptr()); params.kpe = static_cast<DTypeKV*>(kpe_cache.data_ptr()); params.q_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_indptr_offset); params.kv_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_indptr_offset); params.partial_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.partial_indptr_offset); params.kv_indices = static_cast<IdType*>(kv_indices.data_ptr()); params.q_len = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_len_offset); params.kv_len = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_len_offset); params.q_start = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.q_start_offset); params.kv_start = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_start_offset); params.kv_end = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.kv_end_offset); params.work_indptr = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.work_indptr_offset); params.merge_packed_offset_start = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_packed_offset_start_offset); params.merge_packed_offset_end = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.merge_packed_offset_end_offset); params.merge_partial_packed_offset_start = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_partial_packed_offset_start_offset); params.merge_partial_packed_offset_end = GetPtrFromBaseOffset( int_buffer_ptr, plan_info.merge_partial_packed_offset_end_offset); params.merge_partial_stride = GetPtrFromBaseOffset(int_buffer_ptr, plan_info.merge_partial_stride_offset); params.final_o = static_cast<DTypeO*>(o.data_ptr()); params.final_lse = maybe_lse.has_value() ? static_cast<float*>(maybe_lse->data_ptr()) : nullptr; params.partial_o = GetPtrFromBaseOffset(float_buffer_ptr, plan_info.partial_o_offset); params.partial_lse = GetPtrFromBaseOffset(float_buffer_ptr, plan_info.partial_lse_offset); params.num_heads = uint_fastdiv(num_heads); params.block_size = uint_fastdiv(page_size); params.q_nope_stride_n = q_nope_stride_n; params.q_nope_stride_h = q_nope_stride_h; params.q_pe_stride_n = q_pe_stride_n; params.q_pe_stride_h = q_pe_stride_h; params.ckv_stride_page = ckv_stride_page; params.ckv_stride_n = ckv_stride_n; params.kpe_stride_page = kpe_stride_page; params.kpe_stride_n = kpe_stride_n; params.o_stride_n = o_stride_n; params.o_stride_h = o_stride_h; params.sm_scale = sm_scale; cudaError_t status = mla::BatchMLAPageAttentionHopper<MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE>( params, plan_info.num_blks_x, plan_info.num_blks_y, stream);
^

3 errors detected in the compilation of "/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu".
[3/4] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_pybind.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu -o batch_mla_sm90_pybind.cuda.o
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu:16:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu:16:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu:16:
/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file
29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...)
|
ninja: build stopped: subcommand failed.

@kitchen7c commented on GitHub (Apr 22, 2025): > > [@qiyuxinlin](https://github.com/qiyuxinlin) 是使用install.sh脚本里安装的，git拉取的custom_flashinfer 0.2.3，我在原Flashinfer中找到了同样的问题，看起来是最近修复的bug，咱们的custom_flashinfer中好像没有同步更新，[flashinfer-ai/flashinfer#892](https://github.com/flashinfer-ai/flashinfer/issues/892) > > custom_flashinfer不是安装main分支，而是安装fix-precision-mla-merge-main或者GQA_var_batch分支我安装是从fix-precision-mla-merge-main 分支同步的。但还是启动是报下面的错误。 loading blk.58.attn_norm.weight to cuda loading blk.58.ffn_norm.weight to cuda loading blk.59.attn_q_a_norm.weight to cuda loading blk.59.attn_kv_a_norm.weight to cuda loading blk.59.attn_kv_b.weight to cuda:0 loading blk.59.attn_norm.weight to cuda loading blk.59.ffn_norm.weight to cuda loading blk.60.attn_q_a_norm.weight to cuda loading blk.60.attn_kv_a_norm.weight to cuda loading blk.60.attn_kv_b.weight to cuda:0 loading blk.60.attn_norm.weight to cuda loading blk.60.ffn_norm.weight to cuda loading output_norm.weight to cuda Getting inference context from sched_client. Got inference context, sending it to subscribers. Rebuilding kvcache kv_cache loaded successfully. capturing cuda graph 1 1 Process SpawnProcess-1: Traceback (most recent call last): File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2209, in _run_ninja_build subprocess.run( File "/data/miniconda3/envs/ktransformers/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/data/miniconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap self.run() File "/data/miniconda3/envs/ktransformers/lib/python3.11/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 253, in run_engine engine.model_runner.warmup() File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 107, in warmup self.model.flash_infer_attn_plan(self.input[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/custom_modeling_deepseek_v3.py", line 145, in flash_infer_attn_plan self.wrapper.plan(minibatch.q_indptr, minibatch.kv_indptr, minibatch.kv_indices, File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/mla.py", line 232, in plan self._cached_module = get_batch_mla_module(self._backend)( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/mla.py", line 43, in backend_module modules_dict[args] = gen_batch_mla_module(backend, *args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/jit/attention/pytorch.py", line 185, in gen_batch_mla_module return load_cuda_ops( ^^^^^^^^^^^^^^ File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/jit/core.py", line 123, in load_cuda_ops torch_cpp_ext.load( File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1380, in load return _jit_compile( ^^^^^^^^^^^^^ File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1798, in _jit_compile _write_ninja_file_and_build_library( File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1926, in _write_ninja_file_and_build_library _run_ninja_build( File "/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2225, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error building extension 'batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90': [1/4] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_plan.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu -o batch_mla_sm90_plan.cuda.o FAILED: batch_mla_sm90_plan.cuda.o /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_plan.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu -o batch_mla_sm90_plan.cuda.o In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu:19: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu:19: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu:19: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu(44): error: no instance of function template "flashinfer::MLAPlan" matches the argument list argument types are: (void *, size_t, void *, void *, size_t, flashinfer::MLAPlanInfo, IdType *, IdType *, IdType *, int, int64_t, int64_t, bool, const cudaStream_t) MLAPlan(float_workspace_buffer.data_ptr(), float_workspace_size_in_bytes, ^ /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include/flashinfer/attention/scheduler.cuh(1071): note #3322-D: number of parameters of function template "flashinfer::MLAPlan" does not match the call inline cudaError_t MLAPlan(void* float_buffer, size_t float_workspace_size_in_bytes, ^ 1 error detected in the compilation of "/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_plan.cu". [2/4] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_run.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu -o batch_mla_sm90_run.cuda.o FAILED: batch_mla_sm90_run.cuda.o /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_run.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu -o batch_mla_sm90_run.cuda.o In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu:21: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu:21: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu:21: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu(65): error: a value of type "DTypeO *" cannot be assigned to an entity of type "float *" switch (mask_mode) { case MaskMode::kNone: { constexpr MaskMode MASK_MODE = MaskMode::kNone; { using Params = MLAParams<DTypeQ, DTypeKV, DTypeO, IdType>; [&] { Params params; params.q_nope = static_cast<DTypeQ*>(q_nope.data_ptr()); params.q_pe = static_cast<DTypeQ*>(q_pe.data_ptr()); params.ckv = static_cast<DTypeKV*>(ckv_cache.data_ptr()); params.kpe = static_cast<DTypeKV*>(kpe_cache.data_ptr()); params.q_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_indptr_offset); params.kv_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_indptr_offset); params.partial_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.partial_indptr_offset); params.kv_indices = static_cast<IdType*>(kv_indices.data_ptr()); params.q_len = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_len_offset); params.kv_len = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_len_offset); params.q_start = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_start_offset); params.kv_start = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_start_offset); params.kv_end = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_end_offset); params.work_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.work_indptr_offset); params.merge_packed_offset_start = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_packed_offset_start_offset); params.merge_packed_offset_end = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.merge_packed_offset_end_offset); params.merge_partial_packed_offset_start = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_partial_packed_offset_start_offset); params.merge_partial_packed_offset_end = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_partial_packed_offset_end_offset); params.merge_partial_stride = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.merge_partial_stride_offset); params.final_o = static_cast<DTypeO*>(o.data_ptr()); params.final_lse = maybe_lse.has_value() ? static_cast<float*>(maybe_lse->data_ptr()) : nullptr; params.partial_o = GetPtrFromBaseOffset<DTypeO>(float_buffer_ptr, plan_info.partial_o_offset); params.partial_lse = GetPtrFromBaseOffset<float>(float_buffer_ptr, plan_info.partial_lse_offset); params.num_heads = uint_fastdiv(num_heads); params.block_size = uint_fastdiv(page_size); params.q_nope_stride_n = q_nope_stride_n; params.q_nope_stride_h = q_nope_stride_h; params.q_pe_stride_n = q_pe_stride_n; params.q_pe_stride_h = q_pe_stride_h; params.ckv_stride_page = ckv_stride_page; params.ckv_stride_n = ckv_stride_n; params.kpe_stride_page = kpe_stride_page; params.kpe_stride_n = kpe_stride_n; params.o_stride_n = o_stride_n; params.o_stride_h = o_stride_h; params.sm_scale = sm_scale; cudaError_t status = mla::BatchMLAPageAttentionHopper<MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE>( params, plan_info.num_blks_x, plan_info.num_blks_y, stream); ^ /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu(65): error: a value of type "DTypeO *" cannot be assigned to an entity of type "float *" ; }(); } break; } case MaskMode::kCausal: { constexpr MaskMode MASK_MODE = MaskMode::kCausal; { using Params = MLAParams<DTypeQ, DTypeKV, DTypeO, IdType>; [&] { Params params; params.q_nope = static_cast<DTypeQ*>(q_nope.data_ptr()); params.q_pe = static_cast<DTypeQ*>(q_pe.data_ptr()); params.ckv = static_cast<DTypeKV*>(ckv_cache.data_ptr()); params.kpe = static_cast<DTypeKV*>(kpe_cache.data_ptr()); params.q_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_indptr_offset); params.kv_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_indptr_offset); params.partial_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.partial_indptr_offset); params.kv_indices = static_cast<IdType*>(kv_indices.data_ptr()); params.q_len = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_len_offset); params.kv_len = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_len_offset); params.q_start = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_start_offset); params.kv_start = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_start_offset); params.kv_end = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_end_offset); params.work_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.work_indptr_offset); params.merge_packed_offset_start = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_packed_offset_start_offset); params.merge_packed_offset_end = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.merge_packed_offset_end_offset); params.merge_partial_packed_offset_start = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_partial_packed_offset_start_offset); params.merge_partial_packed_offset_end = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_partial_packed_offset_end_offset); params.merge_partial_stride = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.merge_partial_stride_offset); params.final_o = static_cast<DTypeO*>(o.data_ptr()); params.final_lse = maybe_lse.has_value() ? static_cast<float*>(maybe_lse->data_ptr()) : nullptr; params.partial_o = GetPtrFromBaseOffset<DTypeO>(float_buffer_ptr, plan_info.partial_o_offset); params.partial_lse = GetPtrFromBaseOffset<float>(float_buffer_ptr, plan_info.partial_lse_offset); params.num_heads = uint_fastdiv(num_heads); params.block_size = uint_fastdiv(page_size); params.q_nope_stride_n = q_nope_stride_n; params.q_nope_stride_h = q_nope_stride_h; params.q_pe_stride_n = q_pe_stride_n; params.q_pe_stride_h = q_pe_stride_h; params.ckv_stride_page = ckv_stride_page; params.ckv_stride_n = ckv_stride_n; params.kpe_stride_page = kpe_stride_page; params.kpe_stride_n = kpe_stride_n; params.o_stride_n = o_stride_n; params.o_stride_h = o_stride_h; params.sm_scale = sm_scale; cudaError_t status = mla::BatchMLAPageAttentionHopper<MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE>( params, plan_info.num_blks_x, plan_info.num_blks_y, stream); ^ /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu(65): error: a value of type "DTypeO *" cannot be assigned to an entity of type "float *" ; }(); } break; } case MaskMode::kCustom: { constexpr MaskMode MASK_MODE = MaskMode::kCustom; { using Params = MLAParams<DTypeQ, DTypeKV, DTypeO, IdType>; [&] { Params params; params.q_nope = static_cast<DTypeQ*>(q_nope.data_ptr()); params.q_pe = static_cast<DTypeQ*>(q_pe.data_ptr()); params.ckv = static_cast<DTypeKV*>(ckv_cache.data_ptr()); params.kpe = static_cast<DTypeKV*>(kpe_cache.data_ptr()); params.q_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_indptr_offset); params.kv_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_indptr_offset); params.partial_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.partial_indptr_offset); params.kv_indices = static_cast<IdType*>(kv_indices.data_ptr()); params.q_len = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_len_offset); params.kv_len = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_len_offset); params.q_start = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.q_start_offset); params.kv_start = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_start_offset); params.kv_end = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.kv_end_offset); params.work_indptr = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.work_indptr_offset); params.merge_packed_offset_start = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_packed_offset_start_offset); params.merge_packed_offset_end = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.merge_packed_offset_end_offset); params.merge_partial_packed_offset_start = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_partial_packed_offset_start_offset); params.merge_partial_packed_offset_end = GetPtrFromBaseOffset<IdType>( int_buffer_ptr, plan_info.merge_partial_packed_offset_end_offset); params.merge_partial_stride = GetPtrFromBaseOffset<IdType>(int_buffer_ptr, plan_info.merge_partial_stride_offset); params.final_o = static_cast<DTypeO*>(o.data_ptr()); params.final_lse = maybe_lse.has_value() ? static_cast<float*>(maybe_lse->data_ptr()) : nullptr; params.partial_o = GetPtrFromBaseOffset<DTypeO>(float_buffer_ptr, plan_info.partial_o_offset); params.partial_lse = GetPtrFromBaseOffset<float>(float_buffer_ptr, plan_info.partial_lse_offset); params.num_heads = uint_fastdiv(num_heads); params.block_size = uint_fastdiv(page_size); params.q_nope_stride_n = q_nope_stride_n; params.q_nope_stride_h = q_nope_stride_h; params.q_pe_stride_n = q_pe_stride_n; params.q_pe_stride_h = q_pe_stride_h; params.ckv_stride_page = ckv_stride_page; params.ckv_stride_n = ckv_stride_n; params.kpe_stride_page = kpe_stride_page; params.kpe_stride_n = kpe_stride_n; params.o_stride_n = o_stride_n; params.o_stride_h = o_stride_h; params.sm_scale = sm_scale; cudaError_t status = mla::BatchMLAPageAttentionHopper<MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE>( params, plan_info.num_blks_x, plan_info.num_blks_y, stream); ^ 3 errors detected in the compilation of "/data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_run.cu". [3/4] /usr/local/cuda/bin/nvcc --generate-dependencies-with-compile --dependency-output batch_mla_sm90_pybind.cuda.o.d -DTORCH_EXTENSION_NAME=batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/csrc -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/include -I/data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/flashinfer/data/cutlass/tools/util/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -isystem /data/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/miniconda3/envs/ktransformers/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 --expt-relaxed-constexpr -gencode=arch=compute_90,code=sm_90 --compiler-options '-fPIC' -O3 -std=c++17 --threads 4 -use_fast_math -DFLASHINFER_ENABLE_F16 -DFLASHINFER_ENABLE_BF16 -DFLASHINFER_ENABLE_FP8_E4M3 -DFLASHINFER_ENABLE_FP8_E5M2 -gencode=arch=compute_90a,code=sm_90a -c /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu -o batch_mla_sm90_pybind.cuda.o In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu:16: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu:16: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | In file included from /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_pybind.cu:16: /data/home/deepseek/.cache/flashinfer/90/generated/batch_mla_attention_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_ckv_512_head_dim_kpe_64_profiler_False_sm90/batch_mla_sm90_config.inc:29:111: warning: backslash-newline at end of file 29 | #define DISPATCH_context(DTypeQ, DTypeKV, DTypeO, IdType, MASK_MODE, HEAD_DIM_CKV, HEAD_DIM_KPE, Params, ...) \ | ninja: build stopped: subcommand failed.

bsakaguchi commented

Author

Owner

@Mewral commented on GitHub (Apr 22, 2025):

@kitchen7c 我也尝试了一下GQA_var_batch分支，也是一样的报错

@Mewral commented on GitHub (Apr 22, 2025): @kitchen7c 我也尝试了一下GQA_var_batch分支，也是一样的报错

bsakaguchi commented

Author

Owner

@999wwx commented on GitHub (Apr 23, 2025):

同遇到这个问题，相同的源代码，L20机器可以正常运行，H20机器就不行，不知道是机器的问题还是镜像的问题

@999wwx commented on GitHub (Apr 23, 2025): 同遇到这个问题，相同的源代码，L20机器可以正常运行，H20机器就不行，不知道是机器的问题还是镜像的问题

bsakaguchi commented

Author

Owner

@kitchen7c commented on GitHub (Apr 23, 2025):

同遇到这个问题，相同的源代码，L20机器可以正常运行，H20机器就不行，不知道是机器的问题还是镜像的问题

的确，我也是 H20

@kitchen7c commented on GitHub (Apr 23, 2025): > 同遇到这个问题，相同的源代码，L20机器可以正常运行，H20机器就不行，不知道是机器的问题还是镜像的问题的确，我也是 H20

bsakaguchi commented

Author

Owner

@PhzCode commented on GitHub (Apr 24, 2025):

@Mewral @999wwx @kitchen7c 在H20上同样的问题

@PhzCode commented on GitHub (Apr 24, 2025): @Mewral @999wwx @kitchen7c 在H20上同样的问题

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (Apr 29, 2025):

hopper架构上可能会遇到这个问题，是因为我们的custom flashinfer没有修改SM90的相关代码，我们目前手头没有5090的卡，所以暂时没法测试
我简单检查下代码，先让hopper及以后的架构使用非SM90的代码

@Atream commented on GitHub (Apr 29, 2025): hopper架构上可能会遇到这个问题，是因为我们的custom flashinfer没有修改SM90的相关代码，我们目前手头没有5090的卡，所以暂时没法测试我简单检查下代码，先让hopper及以后的架构使用非SM90的代码

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (Apr 29, 2025):

我暂时让hopper架构的卡用SM90之前的代码，但手头没有卡，能帮忙测试下吗？

@Atream commented on GitHub (Apr 29, 2025): 我暂时让hopper架构的卡用SM90之前的代码，但手头没有卡，能帮忙测试下吗？

bsakaguchi commented

Author

Owner

@Mewral commented on GitHub (Apr 29, 2025):

@Atream 可以，要怎么测试？

@Mewral commented on GitHub (Apr 29, 2025): @Atream 可以，要怎么测试？

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (Apr 29, 2025):

@Atream 可以，要怎么测试？

直接拉取最新代码编译即可

@Atream commented on GitHub (Apr 29, 2025): > [@Atream](https://github.com/Atream) 可以，要怎么测试？直接拉取最新代码编译即可

bsakaguchi commented

Author

Owner

@Mewral commented on GitHub (Apr 29, 2025):

@Atream 可以，要怎么测试？

直接拉取最新代码编译即可

加载是没有问题了，可以正常启动服务，但是请求报错，下面是报错信息，custom_flashinfer还用的是GQA_var_batch分支的，ktransformers用的是main分支最新的代码

Traceback (most recent call last):
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "code/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 436, in chat_completion
    async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
  File "code/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
    async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
  File "code/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 412, in inference
    for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "code/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
    logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "code/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
    logits = self.model(
             ^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "code/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1689, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "code/ktransformers/ktransformers/operators/models.py", line 729, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "code/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1206, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'

@Mewral commented on GitHub (Apr 29, 2025): > > [@Atream](https://github.com/Atream) 可以，要怎么测试？ > > 直接拉取最新代码编译即可加载是没有问题了，可以正常启动服务，但是请求报错，下面是报错信息，custom_flashinfer还用的是GQA_var_batch分支的，ktransformers用的是main分支最新的代码 ``` Traceback (most recent call last): File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__ await self.middleware_stack(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 73, in app response = await f(request) ^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "code/ktransformers/ktransformers/server/api/openai/endpoints/chat.py", line 436, in chat_completion async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens): File "code/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens): File "code/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 412, in inference for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens): File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "code/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "code/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill logits = self.model( ^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "code/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1689, in forward outputs = self.model( ^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "code/ktransformers/ktransformers/operators/models.py", line 729, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "code/ktransformers/ktransformers/models/modeling_deepseek_v3.py", line 1206, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask' ```

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (Apr 29, 2025):

记得加上--backend_type balance_serve，目前只为balance_serve添加了qwen3的支持

@Atream commented on GitHub (Apr 29, 2025): 记得加上--backend_type balance_serve，目前只为balance_serve添加了qwen3的支持

bsakaguchi commented

Author

Owner

@Atream commented on GitHub (May 6, 2025):

@Atream 用main分支最新的代码编译，执行localchat，报错如下。

python -m ktransformers.local_chat --model_path /data0_7T/Deepseek-R1-bf16_pt --gguf_path /data0_7T/Deepseek-R1-bf16_pt-GGUF-bf16 --backend_type balance_serve

2025-05-06 18:36:23,589 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/p00838136/ktransformers/ktransformers/local_chat.py", line 61, in <module>
    cpu_infer: int = Config().cpu_infer,
                     ^^^^^^^^
  File "/home/p00838136/ktransformers/ktransformers/server/config/singleton.py", line 26, in __call__
    cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwds)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/p00838136/ktransformers/ktransformers/server/config/config.py", line 194, in __init__
    self.sched_strategy = cfg["async_server"]["sched_strategy"]
                          ~~~^^^^^^^^^^^^^^^^
KeyError: 'async_server'

尝试rm ~/.ktransformers/config.yaml

@Atream commented on GitHub (May 6, 2025): > [@Atream](https://github.com/Atream) 用main分支最新的代码编译，执行localchat，报错如下。 > > ``` > python -m ktransformers.local_chat --model_path /data0_7T/Deepseek-R1-bf16_pt --gguf_path /data0_7T/Deepseek-R1-bf16_pt-GGUF-bf16 --backend_type balance_serve > > 2025-05-06 18:36:23,589 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend > found flashinfer > Traceback (most recent call last): > File "<frozen runpy>", line 198, in _run_module_as_main > File "<frozen runpy>", line 88, in _run_code > File "/home/p00838136/ktransformers/ktransformers/local_chat.py", line 61, in <module> > cpu_infer: int = Config().cpu_infer, > ^^^^^^^^ > File "/home/p00838136/ktransformers/ktransformers/server/config/singleton.py", line 26, in __call__ > cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwds) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/p00838136/ktransformers/ktransformers/server/config/config.py", line 194, in __init__ > self.sched_strategy = cfg["async_server"]["sched_strategy"] > ~~~^^^^^^^^^^^^^^^^ > KeyError: 'async_server' > ``` 尝试rm ~/.ktransformers/config.yaml

bsakaguchi commented

Author

Owner

@PhzCode commented on GitHub (May 6, 2025):

这个是transformers版本的问题吗？我看issue里这个报错都是关于qwen的，没有关于deepseek的 @Atream

Traceback (most recent call last):
  File "/home/p00838136/ktransformers/ktransformers/util/utils.py", line 206, in prefill_and_generate
    model._get_logits_warper(generation_config,device=inputs.device)
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__
    raise AttributeError(
AttributeError: 'DeepseekV3ForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/p00838136/ktransformers/ktransformers/local_chat.py", line 185, in <module>
    fire.Fire(local_chat)
  File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/p00838136/ktransformers/ktransformers/local_chat.py", line 174, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/p00838136/ktransformers/ktransformers/util/utils.py", line 210, in prefill_and_generate
    model._get_logits_warper(generation_config)
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__
    raise AttributeError(
AttributeError: 'DeepseekV3ForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'?

@PhzCode commented on GitHub (May 6, 2025): 这个是transformers版本的问题吗？我看issue里这个报错都是关于qwen的，没有关于deepseek的 @Atream ``` Traceback (most recent call last): File "/home/p00838136/ktransformers/ktransformers/util/utils.py", line 206, in prefill_and_generate model._get_logits_warper(generation_config,device=inputs.device) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__ raise AttributeError( AttributeError: 'DeepseekV3ForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'? During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/p00838136/ktransformers/ktransformers/local_chat.py", line 185, in <module> fire.Fire(local_chat) File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/p00838136/ktransformers/ktransformers/local_chat.py", line 174, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/home/p00838136/ktransformers/ktransformers/util/utils.py", line 210, in prefill_and_generate model._get_logits_warper(generation_config) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miniconda3/envs/ds/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__ raise AttributeError( AttributeError: 'DeepseekV3ForCausalLM' object has no attribute '_get_logits_warper'. Did you mean: '_get_logits_processor'? ```

bsakaguchi commented