kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #900] Rocm build error #648

New issue
Closed
opened 2026-10-29 14:42:21 +00:00 by bsakaguchi · 15 comments
bsakaguchi commented 2026-10-29 14:42:21 +00:00
Owner
Copy link

Originally created by @gengchaogit on GitHub (Mar 15, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/900

Originally assigned to: @Azure-Tang on GitHub.

(ktransformers) root@epyc:~/ktransformers# rocm-smi

Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU%
(DID, GUID) (Edge) (Socket) (Mem, Compute, ID)
0 1 0x66a1, 52718 33.0°C 22.0W N/A, N/A, 0 860Mhz 350Mhz 9.41% auto 190.0W 0% 0%
1 2 0x66a1, 46486 35.0°C 20.0W N/A, N/A, 0 860Mhz 350Mhz 9.41% auto 190.0W 0% 0%

(ktransformers) root@epyc:~/ktransformers#

Originally created by @gengchaogit on GitHub (Mar 15, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/900 Originally assigned to: @Azure-Tang on GitHub. (ktransformers) root@epyc:~/ktransformers# rocm-smi Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% (DID, GUID) (Edge) (Socket) (Mem, Compute, ID) 0 1 0x66a1, 52718 33.0°C 22.0W N/A, N/A, 0 860Mhz 350Mhz 9.41% auto 190.0W 0% 0% 1 2 0x66a1, 46486 35.0°C 20.0W N/A, N/A, 0 860Mhz 350Mhz 9.41% auto 190.0W 0% 0% (ktransformers) root@epyc:~/ktransformers#
bsakaguchi commented 2026-10-29 14:42:22 +00:00
Author
Owner
Copy link

@gengchaogit commented on GitHub (Mar 15, 2025):

running build_ext
[1/54] /usr/bin/c++ -DKTRANSFORMERS_USE_CUDA=1 -Dcpuinfer_ext_EXPORTS -I/root/ktransformers/ktransformers/ktransformers_ext/../../third_party -I/usr/local/cuda-12.8/include -I/root/ktransformers/third_party/llama.cpp/. -isystem /root/ktransformers/third_party/pybind11/include -isystem /root/miniconda3/envs/ktransformers/include/python3.11 -O3 -ffast-math -O3 -DNDEBUG -fPIC -fvisibility=hidden -march=native -flto -fno-fat-lto-objects -MD -MT CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/flags.cpp.o -MF CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/flags.cpp.o.d -o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/flags.cpp.o -c /root/ktransformers/third_party/llamafile/flags.cpp
.........................................................
.........................................................
.........................................................
Standard output: b'-- The C compiler identification is GNU 11.4.0\n-- The CXX compiler identification is GNU 11.4.0\n-- Detecting C compiler ABI info\n-- Detecting C compiler ABI info - done\n-- Check for working C compiler: /usr/bin/cc - skipped\n-- Detecting C compile features\n-- Detecting C compile features - done\n-- Detecting CXX compiler ABI info\n-- Detecting CXX compiler ABI info - done\n-- Check for working CXX compiler: /usr/bin/c++ - skipped\n-- Detecting CXX compile features\n-- Detecting CXX compile features - done\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- pybind11 v2.14.0 dev1\n-- Found PythonInterp: /root/miniconda3/envs/ktransformers/bin/python (found suitable version "3.11.11", minimum required is "3.7") \n-- Found PythonLibs: /root/miniconda3/envs/ktransformers/lib/libpython3.11.so\n-- Performing Test HAS_FLTO\n-- Performing Test HAS_FLTO - Success\n-- Found Git: /usr/bin/git (found version "2.34.1") \n-- Looking for pthread.h\n-- Looking for pthread.h - found\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success\n-- Found Threads: TRUE \n-- Found OpenMP_C: -fopenmp (found version "4.5") \n-- Found OpenMP_CXX: -fopenmp (found version "4.5") \n-- Found OpenMP: TRUE (found version "4.5") \n-- OpenMP found\n-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with LLAMA_CCACHE=OFF\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- Found CUDA: /usr/local/cuda-12.8 (found version "12.8") \n-- ALL_SOURCES: /root/ktransformers/ktransformers/ktransformers_ext/ext_bindings.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/backend.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/task_queue.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/linear.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/mlp.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/moe.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/shared_mem_buffer.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/flags.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/sgemm.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_unsupported.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_attn.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_load_dump.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_read_write.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_utils.cpp\n-- NUMA support is disabled\n-- NUMA library not found or user not set USE_NUMA - disabling NUMA support\n-- Configuring done\n-- Generating done\n-- Build files have been written to: /root/ktransformers/ktransformers/ktransformers_ext/build\n'
Standard error: b'CMake Warning:\n Manually-specified variables were not used by the project:\n\n EXAMPLE_VERSION_INFO\n\n\n'
building 'KTransformersOps' extension
creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip
creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf
creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/gptq_marlin
Emitting ninja build file /root/ktransformers/build/temp.linux-x86_64-cpython-311/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /opt/rocm-6.3.3/bin/hipcc -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o -fPIC -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -fno-gpu-rdc -std=c++17
FAILED: /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o
/opt/rocm-6.3.3/bin/hipcc -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o -fPIC -D__HIP_PLATFORM_AMD=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS_=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -fno-gpu-rdc -std=c++17
clang++: warning: argument unused during compilation: '-Xcompiler' [-Wunused-command-line-argument]
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:50:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
50 | global void dequantize_q8_0_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) {
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:53:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
53 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:53:49: error: use of undeclared identifier 'nv_bfloat16'
53 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:53:61: error: expected expression
53 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:141:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
141 | global void dequantize_q2_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) {
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:144:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
144 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:144:49: error: use of undeclared identifier 'nv_bfloat16'
144 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:144:61: error: expected expression
144 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:287:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
287 | global void dequantize_q3_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) {
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:293:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
293 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:293:49: error: use of undeclared identifier 'nv_bfloat16'
293 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:293:61: error: expected expression
293 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:392:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
392 | global void dequantize_q4_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) {
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:395:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
395 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:395:49: error: use of undeclared identifier 'nv_bfloat16'
395 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:395:61: error: expected expression
395 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:474:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
474 | global void dequantize_q5_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) {
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:477:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'?
477 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^~~~~~~~~~~
| hip_bfloat16
/opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here
57 | struct hip_bfloat16
| ^
/root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:477:49: error: use of undeclared identifier 'nv_bfloat16'
477 | nv_bfloat16* restrict output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk);
| ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated when compiling for gfx1030.
failed to execute:/opt/rocm-6.3.3/lib/llvm/bin/clang++ --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c -x hip /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip -o "/root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o" -fPIC -D__HIP_PLATFORM_AMD=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS_=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1016" -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 -fno-gpu-rdc -std=c++17
[2/3] c++ -MMD -MF /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/binding.o.d -pthread -B /root/miniconda3/envs/ktransformers/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/ktransformers/include -fPIC -O2 -isystem /root/miniconda3/envs/ktransformers/include -fPIC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/binding.cpp -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/binding.o -fPIC -D__HIP_PLATFORM_AMD=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -D__HIP_PLATFORM_AMD_=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -O3 -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++17
[3/3] /opt/rocm-6.3.3/bin/hipcc -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/gptq_marlin/gptq_marlin.hip -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/gptq_marlin/gptq_marlin.o -fPIC -D__HIP_PLATFORM_AMD=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS_=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -fno-gpu-rdc -std=c++17
clang++: warning: argument unused during compilation: '-Xcompiler' [-Wunused-command-line-argument]
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2209, in _run_ninja_build
subprocess.run(
File "/root/miniconda3/envs/ktransformers/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
      main()
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
      json_out["return_val"] = hook(**hook_input["kwargs"])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
      return _build_backend().build_wheel(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 438, in build_wheel
      return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 426, in _build
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
      self.run_setup()
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup
      exec(code, locals())
    File "<string>", line 443, in <module>
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup
      return run_commands(dist)
             ^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
      dist.run_commands()
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 983, in run_commands
      self.run_command(cmd)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
      super().run_command(command)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
      cmd_obj.run()
    File "<string>", line 243, in run
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 379, in run
      self.run_command("build")
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
      self.distribution.run_command(command)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
      super().run_command(command)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
      cmd_obj.run()
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 136, in run
      self.run_command(cmd_name)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command
      self.distribution.run_command(command)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command
      super().run_command(command)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command
      cmd_obj.run()
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
      _build_ext.run(self)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 365, in run
      self.build_extensions()
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 900, in build_extensions
      build_ext.build_extensions(self)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 481, in build_extensions
      self._build_extensions_serial()
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 507, in _build_extensions_serial
      self.build_extension(ext)
    File "<string>", line 287, in build_extension
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
      _build_ext.build_extension(self, ext)
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 562, in build_extension
      objects = self.compiler.compile(
                ^^^^^^^^^^^^^^^^^^^^^^
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 713, in unix_wrap_ninja_compile
      _write_ninja_file_and_compile_objects(
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1869, in _write_ninja_file_and_compile_objects
      _run_ninja_build(
    File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2225, in _run_ninja_build
      raise RuntimeError(message) from e
  RuntimeError: Error compiling objects for extension
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for ktransformers
Failed to build ktransformers
ERROR: Failed to build installable wheels for some pyproject.toml based projects (ktransformers)

<!-- gh-comment-id:2726470794 --> @gengchaogit commented on GitHub (Mar 15, 2025): running build_ext [1/54] /usr/bin/c++ -DKTRANSFORMERS_USE_CUDA=1 -Dcpuinfer_ext_EXPORTS -I/root/ktransformers/ktransformers/ktransformers_ext/../../third_party -I/usr/local/cuda-12.8/include -I/root/ktransformers/third_party/llama.cpp/. -isystem /root/ktransformers/third_party/pybind11/include -isystem /root/miniconda3/envs/ktransformers/include/python3.11 -O3 -ffast-math -O3 -DNDEBUG -fPIC -fvisibility=hidden -march=native -flto -fno-fat-lto-objects -MD -MT CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/flags.cpp.o -MF CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/flags.cpp.o.d -o CMakeFiles/cpuinfer_ext.dir/root/ktransformers/third_party/llamafile/flags.cpp.o -c /root/ktransformers/third_party/llamafile/flags.cpp ......................................................... ......................................................... ......................................................... Standard output: b'-- The C compiler identification is GNU 11.4.0\n-- The CXX compiler identification is GNU 11.4.0\n-- Detecting C compiler ABI info\n-- Detecting C compiler ABI info - done\n-- Check for working C compiler: /usr/bin/cc - skipped\n-- Detecting C compile features\n-- Detecting C compile features - done\n-- Detecting CXX compiler ABI info\n-- Detecting CXX compiler ABI info - done\n-- Check for working CXX compiler: /usr/bin/c++ - skipped\n-- Detecting CXX compile features\n-- Detecting CXX compile features - done\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- pybind11 v2.14.0 dev1\n-- Found PythonInterp: /root/miniconda3/envs/ktransformers/bin/python (found suitable version "3.11.11", minimum required is "3.7") \n-- Found PythonLibs: /root/miniconda3/envs/ktransformers/lib/libpython3.11.so\n-- Performing Test HAS_FLTO\n-- Performing Test HAS_FLTO - Success\n-- Found Git: /usr/bin/git (found version "2.34.1") \n-- Looking for pthread.h\n-- Looking for pthread.h - found\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success\n-- Found Threads: TRUE \n-- Found OpenMP_C: -fopenmp (found version "4.5") \n-- Found OpenMP_CXX: -fopenmp (found version "4.5") \n-- Found OpenMP: TRUE (found version "4.5") \n-- OpenMP found\n-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with LLAMA_CCACHE=OFF\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- Found CUDA: /usr/local/cuda-12.8 (found version "12.8") \n-- ALL_SOURCES: /root/ktransformers/ktransformers/ktransformers_ext/ext_bindings.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/backend.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/task_queue.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/linear.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/mlp.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/moe.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/shared_mem_buffer.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/flags.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/sgemm.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_unsupported.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_attn.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_load_dump.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_read_write.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_utils.cpp\n-- NUMA support is disabled\n-- NUMA library not found or user not set USE_NUMA - disabling NUMA support\n-- Configuring done\n-- Generating done\n-- Build files have been written to: /root/ktransformers/ktransformers/ktransformers_ext/build\n' Standard error: b'CMake Warning:\n Manually-specified variables were not used by the project:\n\n EXAMPLE_VERSION_INFO\n\n\n' building 'KTransformersOps' extension creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf creating /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/gptq_marlin Emitting ninja build file /root/ktransformers/build/temp.linux-x86_64-cpython-311/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/3] /opt/rocm-6.3.3/bin/hipcc -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o -fPIC -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -fno-gpu-rdc -std=c++17 FAILED: /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o /opt/rocm-6.3.3/bin/hipcc -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o -fPIC -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -fno-gpu-rdc -std=c++17 clang++: warning: argument unused during compilation: '-Xcompiler' [-Wunused-command-line-argument] /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:50:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 50 | __global__ void dequantize_q8_0_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) { | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:53:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 53 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:53:49: error: use of undeclared identifier 'nv_bfloat16' 53 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:53:61: error: expected expression 53 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:141:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 141 | __global__ void dequantize_q2_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) { | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:144:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 144 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:144:49: error: use of undeclared identifier 'nv_bfloat16' 144 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:144:61: error: expected expression 144 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:287:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 287 | __global__ void dequantize_q3_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) { | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:293:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 293 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:293:49: error: use of undeclared identifier 'nv_bfloat16' 293 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:293:61: error: expected expression 293 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:392:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 392 | __global__ void dequantize_q4_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) { | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:395:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 395 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:395:49: error: use of undeclared identifier 'nv_bfloat16' 395 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:395:61: error: expected expression 395 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:474:65: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 474 | __global__ void dequantize_q5_k_bf16_kernel(const int8_t* data, nv_bfloat16* output, const int blk_size, const int ele_per_blk, const int num_blocks) { | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:477:9: error: unknown type name 'nv_bfloat16'; did you mean 'hip_bfloat16'? 477 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^~~~~~~~~~~ | hip_bfloat16 /opt/rocm-6.3.3/lib/llvm/bin/../../../include/hip/amd_detail/amd_hip_bfloat16.h:57:8: note: 'hip_bfloat16' declared here 57 | struct hip_bfloat16 | ^ /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip:477:49: error: use of undeclared identifier 'nv_bfloat16' 477 | nv_bfloat16* __restrict__ output_blk = (nv_bfloat16*)(output + block_id * ele_per_blk); | ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated when compiling for gfx1030. failed to execute:/opt/rocm-6.3.3/lib/llvm/bin/clang++ --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c -x hip /root/ktransformers/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.hip -o "/root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/custom_gguf/dequant.o" -fPIC -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1016\" -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 -fno-gpu-rdc -std=c++17 [2/3] c++ -MMD -MF /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/binding.o.d -pthread -B /root/miniconda3/envs/ktransformers/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /root/miniconda3/envs/ktransformers/include -fPIC -O2 -isystem /root/miniconda3/envs/ktransformers/include -fPIC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/binding.cpp -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/binding.o -fPIC -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -O3 -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++17 [3/3] /opt/rocm-6.3.3/bin/hipcc -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/TH -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THC -I/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/include/THH -I/opt/rocm-6.3.3/include -I/root/miniconda3/envs/ktransformers/include/python3.11 -c -c /root/ktransformers/ktransformers/ktransformers_ext/hip/gptq_marlin/gptq_marlin.hip -o /root/ktransformers/build/temp.linux-x86_64-cpython-311/ktransformers/ktransformers_ext/hip/gptq_marlin/gptq_marlin.o -fPIC -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -Xcompiler -fPIC -DKTRANSFORMERS_USE_CUDA -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1016"' -DTORCH_EXTENSION_NAME=KTransformersOps -D_GLIBCXX_USE_CXX11_ABI=1 --offload-arch=gfx900 --offload-arch=gfx906 --offload-arch=gfx908 --offload-arch=gfx90a --offload-arch=gfx942 --offload-arch=gfx1030 --offload-arch=gfx1100 --offload-arch=gfx1101 -fno-gpu-rdc -std=c++17 clang++: warning: argument unused during compilation: '-Xcompiler' [-Wunused-command-line-argument] ninja: build stopped: subcommand failed. Traceback (most recent call last): File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2209, in _run_ninja_build subprocess.run( File "/root/miniconda3/envs/ktransformers/lib/python3.11/subprocess.py", line 571, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in <module> main() File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main json_out["return_val"] = hook(**hook_input["kwargs"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel return _build_backend().build_wheel( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 438, in build_wheel return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 426, in _build return self._build_with_temp_dir( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir self.run_setup() File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup exec(code, locals()) File "<string>", line 443, in <module> File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup return distutils.core.setup(**attrs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 186, in setup return run_commands(dist) ^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 202, in run_commands dist.run_commands() File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 983, in run_commands self.run_command(cmd) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command super().run_command(command) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command cmd_obj.run() File "<string>", line 243, in run File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 379, in run self.run_command("build") File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command self.distribution.run_command(command) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command super().run_command(command) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command cmd_obj.run() File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 136, in run self.run_command(cmd_name) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 339, in run_command self.distribution.run_command(command) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/dist.py", line 999, in run_command super().run_command(command) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 1002, in run_command cmd_obj.run() File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run _build_ext.run(self) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 365, in run self.build_extensions() File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 900, in build_extensions build_ext.build_extensions(self) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 481, in build_extensions self._build_extensions_serial() File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 507, in _build_extensions_serial self.build_extension(ext) File "<string>", line 287, in build_extension File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 264, in build_extension _build_ext.build_extension(self, ext) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 562, in build_extension objects = self.compiler.compile( ^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 713, in unix_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1869, in _write_ninja_file_and_compile_objects _run_ninja_build( File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2225, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for ktransformers Failed to build ktransformers ERROR: Failed to build installable wheels for some pyproject.toml based projects (ktransformers)
bsakaguchi commented 2026-10-29 14:42:22 +00:00
Author
Owner
Copy link

@gengchaogit commented on GitHub (Mar 15, 2025):

When compiling kt, I encountered the following error

nv_bfloat16/amd_hip_bfloat16

Will kt detect a dual graphics card environment?
I have the nvidia driver in my environment, but no physical nvidia graphics card.

<!-- gh-comment-id:2726472596 --> @gengchaogit commented on GitHub (Mar 15, 2025): When compiling kt, I encountered the following error nv_bfloat16/amd_hip_bfloat16 Will kt detect a dual graphics card environment? I have the nvidia driver in my environment, but no physical nvidia graphics card.
bsakaguchi commented 2026-10-29 14:42:22 +00:00
Author
Owner
Copy link

@gengchaogit commented on GitHub (Mar 15, 2025):

(ktransformers) root@epyc:/ktransformers# ninja --version
1.11.1.git.kitware.jobserver-1
(ktransformers) root@epyc:/ktransformers# hipcc --version
HIP version: 6.3.42134-a9a80e791
AMD clang version 18.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-6.3.3 25012 e5bf7e55c91490b07c49d8960fa7983d864936c4)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-6.3.3/lib/llvm/bin
Configuration file: /opt/rocm-6.3.3/lib/llvm/bin/clang++.cfg
(ktransformers) root@epyc:~/ktransformers#

<!-- gh-comment-id:2726473117 --> @gengchaogit commented on GitHub (Mar 15, 2025): (ktransformers) root@epyc:~/ktransformers# ninja --version 1.11.1.git.kitware.jobserver-1 (ktransformers) root@epyc:~/ktransformers# hipcc --version HIP version: 6.3.42134-a9a80e791 AMD clang version 18.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-6.3.3 25012 e5bf7e55c91490b07c49d8960fa7983d864936c4) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /opt/rocm-6.3.3/lib/llvm/bin Configuration file: /opt/rocm-6.3.3/lib/llvm/bin/clang++.cfg (ktransformers) root@epyc:~/ktransformers#
bsakaguchi commented 2026-10-29 14:42:22 +00:00
Author
Owner
Copy link

@gengchaogit commented on GitHub (Mar 15, 2025):

(ktransformers) root@epyc:~/ktransformers# rocminfo
ROCk module version 6.10.5 is loaded

HSA System Attributes

Runtime Version: 1.14
Runtime Ext Version: 1.6
System Timestamp Freq.: 1000.000000MHz
Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model: LARGE
System Endianness: LITTLE
Mwaitx: DISABLED
DMAbuf Support: YES

==========
HSA Agents

Agent 1

Name: AMD Eng Sample: 100-000000897-03
Uuid: CPU-XX
Marketing Name: AMD Eng Sample: 100-000000897-03
Vendor Name: CPU
Feature: None specified
Profile: FULL_PROFILE
Float Round Mode: NEAR
Max Queue Number: 0(0x0)
Queue Min Size: 0(0x0)
Queue Max Size: 0(0x0)
Queue Type: MULTI
Node: 0
Device Type: CPU
Cache Info:
L1: 32768(0x8000) KB
Chip ID: 0(0x0)
ASIC Revision: 0(0x0)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 2550
BDFID: 0
Internal Node ID: 0
Compute Unit: 64
SIMDs per CU: 0
Shader Engines: 0
Shader Arrs. per Eng.: 0
WatchPts on Addr. Ranges:1
Memory Properties:
Features: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: FINE GRAINED
Size: 1188270360(0x46d39118) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 1188270360(0x46d39118) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 3
Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED
Size: 1188270360(0x46d39118) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
Pool 4
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 1188270360(0x46d39118) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:4KB
Alloc Alignment: 4KB
Accessible by all: TRUE
ISA Info:

Agent 2

Name: gfx906
Uuid: GPU-abd208e17348c2f5
Marketing Name: AMD Radeon (TM) Pro VII
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 1
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
L2: 8192(0x2000) KB
Chip ID: 26273(0x66a1)
ASIC Revision: 1(0x1)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1700
BDFID: 768
Internal Node ID: 1
Compute Unit: 60
SIMDs per CU: 4
Shader Engines: 4
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Memory Properties:
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 40(0x28)
Max Work-item Per CU: 2560(0xa00)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 472
SDMA engine uCode:: 145
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16760832(0xffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 16760832(0xffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32

Agent 3

Name: gfx906
Uuid: GPU-2150604173497dd3
Marketing Name: AMD Radeon (TM) Pro VII
Vendor Name: AMD
Feature: KERNEL_DISPATCH
Profile: BASE_PROFILE
Float Round Mode: NEAR
Max Queue Number: 128(0x80)
Queue Min Size: 64(0x40)
Queue Max Size: 131072(0x20000)
Queue Type: MULTI
Node: 2
Device Type: GPU
Cache Info:
L1: 16(0x10) KB
L2: 8192(0x2000) KB
Chip ID: 26273(0x66a1)
ASIC Revision: 1(0x1)
Cacheline Size: 64(0x40)
Max Clock Freq. (MHz): 1700
BDFID: 41728
Internal Node ID: 2
Compute Unit: 60
SIMDs per CU: 4
Shader Engines: 4
Shader Arrs. per Eng.: 1
WatchPts on Addr. Ranges:4
Coherent Host Access: FALSE
Memory Properties:
Features: KERNEL_DISPATCH
Fast F16 Operation: TRUE
Wavefront Size: 64(0x40)
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Max Waves Per CU: 40(0x28)
Max Work-item Per CU: 2560(0xa00)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
Max fbarriers/Workgrp: 32
Packet Processor uCode:: 472
SDMA engine uCode:: 145
IOMMU Support:: None
Pool Info:
Pool 1
Segment: GLOBAL; FLAGS: COARSE GRAINED
Size: 16760832(0xffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 2
Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED
Size: 16760832(0xffc000) KB
Allocatable: TRUE
Alloc Granule: 4KB
Alloc Recommended Granule:2048KB
Alloc Alignment: 4KB
Accessible by all: FALSE
Pool 3
Segment: GROUP
Size: 64(0x40) KB
Allocatable: FALSE
Alloc Granule: 0KB
Alloc Recommended Granule:0KB
Alloc Alignment: 0KB
Accessible by all: FALSE
ISA Info:
ISA 1
Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack-
Machine Models: HSA_MACHINE_MODEL_LARGE
Profiles: HSA_PROFILE_BASE
Default Rounding Mode: NEAR
Default Rounding Mode: NEAR
Fast f16: TRUE
Workgroup Max Size: 1024(0x400)
Workgroup Max Size per Dimension:
x 1024(0x400)
y 1024(0x400)
z 1024(0x400)
Grid Max Size: 4294967295(0xffffffff)
Grid Max Size per Dimension:
x 4294967295(0xffffffff)
y 4294967295(0xffffffff)
z 4294967295(0xffffffff)
FBarrier Max Size: 32
*** Done ***
(ktransformers) root@epyc:~/ktransformers#

<!-- gh-comment-id:2726473194 --> @gengchaogit commented on GitHub (Mar 15, 2025): (ktransformers) root@epyc:~/ktransformers# rocminfo ROCk module version 6.10.5 is loaded ===================== HSA System Attributes ===================== Runtime Version: 1.14 Runtime Ext Version: 1.6 System Timestamp Freq.: 1000.000000MHz Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) Machine Model: LARGE System Endianness: LITTLE Mwaitx: DISABLED DMAbuf Support: YES ========== HSA Agents ========== ******* Agent 1 ******* Name: AMD Eng Sample: 100-000000897-03 Uuid: CPU-XX Marketing Name: AMD Eng Sample: 100-000000897-03 Vendor Name: CPU Feature: None specified Profile: FULL_PROFILE Float Round Mode: NEAR Max Queue Number: 0(0x0) Queue Min Size: 0(0x0) Queue Max Size: 0(0x0) Queue Type: MULTI Node: 0 Device Type: CPU Cache Info: L1: 32768(0x8000) KB Chip ID: 0(0x0) ASIC Revision: 0(0x0) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 2550 BDFID: 0 Internal Node ID: 0 Compute Unit: 64 SIMDs per CU: 0 Shader Engines: 0 Shader Arrs. per Eng.: 0 WatchPts on Addr. Ranges:1 Memory Properties: Features: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: FINE GRAINED Size: 1188270360(0x46d39118) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 1188270360(0x46d39118) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 3 Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED Size: 1188270360(0x46d39118) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE Pool 4 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 1188270360(0x46d39118) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:4KB Alloc Alignment: 4KB Accessible by all: TRUE ISA Info: ******* Agent 2 ******* Name: gfx906 Uuid: GPU-abd208e17348c2f5 Marketing Name: AMD Radeon (TM) Pro VII Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 1 Device Type: GPU Cache Info: L1: 16(0x10) KB L2: 8192(0x2000) KB Chip ID: 26273(0x66a1) ASIC Revision: 1(0x1) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 1700 BDFID: 768 Internal Node ID: 1 Compute Unit: 60 SIMDs per CU: 4 Shader Engines: 4 Shader Arrs. per Eng.: 1 WatchPts on Addr. Ranges:4 Coherent Host Access: FALSE Memory Properties: Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 64(0x40) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 40(0x28) Max Work-item Per CU: 2560(0xa00) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) Max fbarriers/Workgrp: 32 Packet Processor uCode:: 472 SDMA engine uCode:: 145 IOMMU Support:: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 16760832(0xffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 16760832(0xffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 3 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Recommended Granule:0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) FBarrier Max Size: 32 ******* Agent 3 ******* Name: gfx906 Uuid: GPU-2150604173497dd3 Marketing Name: AMD Radeon (TM) Pro VII Vendor Name: AMD Feature: KERNEL_DISPATCH Profile: BASE_PROFILE Float Round Mode: NEAR Max Queue Number: 128(0x80) Queue Min Size: 64(0x40) Queue Max Size: 131072(0x20000) Queue Type: MULTI Node: 2 Device Type: GPU Cache Info: L1: 16(0x10) KB L2: 8192(0x2000) KB Chip ID: 26273(0x66a1) ASIC Revision: 1(0x1) Cacheline Size: 64(0x40) Max Clock Freq. (MHz): 1700 BDFID: 41728 Internal Node ID: 2 Compute Unit: 60 SIMDs per CU: 4 Shader Engines: 4 Shader Arrs. per Eng.: 1 WatchPts on Addr. Ranges:4 Coherent Host Access: FALSE Memory Properties: Features: KERNEL_DISPATCH Fast F16 Operation: TRUE Wavefront Size: 64(0x40) Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Max Waves Per CU: 40(0x28) Max Work-item Per CU: 2560(0xa00) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) Max fbarriers/Workgrp: 32 Packet Processor uCode:: 472 SDMA engine uCode:: 145 IOMMU Support:: None Pool Info: Pool 1 Segment: GLOBAL; FLAGS: COARSE GRAINED Size: 16760832(0xffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 2 Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED Size: 16760832(0xffc000) KB Allocatable: TRUE Alloc Granule: 4KB Alloc Recommended Granule:2048KB Alloc Alignment: 4KB Accessible by all: FALSE Pool 3 Segment: GROUP Size: 64(0x40) KB Allocatable: FALSE Alloc Granule: 0KB Alloc Recommended Granule:0KB Alloc Alignment: 0KB Accessible by all: FALSE ISA Info: ISA 1 Name: amdgcn-amd-amdhsa--gfx906:sramecc+:xnack- Machine Models: HSA_MACHINE_MODEL_LARGE Profiles: HSA_PROFILE_BASE Default Rounding Mode: NEAR Default Rounding Mode: NEAR Fast f16: TRUE Workgroup Max Size: 1024(0x400) Workgroup Max Size per Dimension: x 1024(0x400) y 1024(0x400) z 1024(0x400) Grid Max Size: 4294967295(0xffffffff) Grid Max Size per Dimension: x 4294967295(0xffffffff) y 4294967295(0xffffffff) z 4294967295(0xffffffff) FBarrier Max Size: 32 *** Done *** (ktransformers) root@epyc:~/ktransformers#
bsakaguchi commented 2026-10-29 14:42:22 +00:00
Author
Owner
Copy link

@gengchaogit commented on GitHub (Mar 15, 2025):

Dear developers, thank you for your support for rocm.
I have compiled and installed it as soon as possible.
Regarding the error message during the installation process, please confirm whether it is a dual graphics card problem？a bug？ or my personal operation problem.

<!-- gh-comment-id:2726473870 --> @gengchaogit commented on GitHub (Mar 15, 2025): Dear developers, thank you for your support for rocm. I have compiled and installed it as soon as possible. Regarding the error message during the installation process, please confirm whether it is a dual graphics card problem？a bug？ or my personal operation problem.
bsakaguchi commented 2026-10-29 14:42:22 +00:00
Author
Owner
Copy link

@gengchaogit commented on GitHub (Mar 15, 2025):

dual mi50 gfx906

<!-- gh-comment-id:2726474908 --> @gengchaogit commented on GitHub (Mar 15, 2025): dual mi50 gfx906
bsakaguchi commented 2026-10-29 14:42:23 +00:00
Author
Owner
Copy link

@tsdcz commented on GitHub (Apr 12, 2025):

你好，我也遇到了和你一样的问题，只是我配置了单片mi50，请问后来解决了吗？

<!-- gh-comment-id:2798849626 --> @tsdcz commented on GitHub (Apr 12, 2025): 你好，我也遇到了和你一样的问题，只是我配置了单片mi50，请问后来解决了吗？
bsakaguchi commented 2026-10-29 14:42:23 +00:00
Author
Owner
Copy link

@Azure-Tang commented on GitHub (Apr 13, 2025):

For Users Requiring MI50 Support:

Through collaborative testing with @gengchaogit, we've determined that MI50 compatibility requires two specific workarounds:

  1. Disabling CUDA graph support
  2. Using TorchLinear instead of Marlin kernels

And The MI50 lacks support for hipblaslt, which results in significantly degraded decoding performance. Due to this limitation, we strongly discourage using MI50 GPUs for running KTransformers.

<!-- gh-comment-id:2799945213 --> @Azure-Tang commented on GitHub (Apr 13, 2025): For Users Requiring MI50 Support: Through collaborative testing with @gengchaogit, we've determined that MI50 compatibility requires two specific workarounds: 1. Disabling CUDA graph support 2. Using TorchLinear instead of Marlin kernels And The MI50 lacks support for hipblaslt, which results in significantly degraded decoding performance. Due to this limitation, we strongly discourage using MI50 GPUs for running KTransformers.
bsakaguchi commented 2026-10-29 14:42:23 +00:00
Author
Owner
Copy link

@gengchaogit commented on GitHub (Apr 13, 2025):

@Azure-Tang Thanks for your help! Please close this issue.

<!-- gh-comment-id:2799955058 --> @gengchaogit commented on GitHub (Apr 13, 2025): @Azure-Tang Thanks for your help! Please close this issue.
bsakaguchi commented 2026-10-29 14:42:23 +00:00
Author
Owner
Copy link

@Len-Li commented on GitHub (Apr 23, 2025):

Hi,

May i ask how to Using TorchLinear instead of Marlin kernels during installation?

Thanks!

<!-- gh-comment-id:2822864241 --> @Len-Li commented on GitHub (Apr 23, 2025): Hi, May i ask how to Using TorchLinear instead of Marlin kernels during installation? Thanks!
bsakaguchi commented 2026-10-29 14:42:24 +00:00
Author
Owner
Copy link

@Azure-Tang commented on GitHub (Apr 23, 2025):

Hi,

May i ask how to Using TorchLinear instead of Marlin kernels during installation?

Thanks!

So your marlin ops cannot pass compilation?

<!-- gh-comment-id:2823261535 --> @Azure-Tang commented on GitHub (Apr 23, 2025): > Hi, > > May i ask how to Using TorchLinear instead of Marlin kernels during installation? > > Thanks! So your marlin ops cannot pass compilation?
bsakaguchi commented 2026-10-29 14:42:24 +00:00
Author
Owner
Copy link

@Len-Li commented on GitHub (Apr 24, 2025):

Hi,
May i ask how to Using TorchLinear instead of Marlin kernels during installation?
Thanks!

So your marlin ops cannot pass compilation?

Yes. Same error in this issue.

<!-- gh-comment-id:2826315092 --> @Len-Li commented on GitHub (Apr 24, 2025): > > Hi, > > May i ask how to Using TorchLinear instead of Marlin kernels during installation? > > Thanks! > > So your marlin ops cannot pass compilation? Yes. Same error in this issue.
bsakaguchi commented 2026-10-29 14:42:24 +00:00
Author
Owner
Copy link

@Len-Li commented on GitHub (Apr 27, 2025):

Hi, i successfully build kt for mi50 gpu. Just following https://github.com/kvcache-ai/ktransformers/issues/374. And comment all gptq_marlin line in setup.py.

<!-- gh-comment-id:2832902602 --> @Len-Li commented on GitHub (Apr 27, 2025): Hi, i successfully build kt for mi50 gpu. Just following https://github.com/kvcache-ai/ktransformers/issues/374. And comment all gptq_marlin line in setup.py.
bsakaguchi commented 2026-10-29 14:42:24 +00:00
Author
Owner
Copy link

@mj520 commented on GitHub (May 1, 2025):

python ./ktransformers/local_chat.py \

--model_path /root/DeepSeek-R1
--gguf_path /root/DeepSeek-R1-UD-IQ1_S
--max_new_tokens 2048
--force_think true
Traceback (most recent call last):
File "/root/ktransformers/./ktransformers/local_chat.py", line 25, in
from ktransformers.optimize.optimize import optimize_and_load_gguf
File "/root/ktransformers/./ktransformers/optimize/optimize.py", line 15, in
from ktransformers.util.custom_gguf import GGUFLoader, translate_name_to_gguf
File "/root/ktransformers/./ktransformers/util/custom_gguf.py", line 27, in
import KTransformersOps
ImportError: /root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so: undefined symbol: _Z16gptq_marlin_gemmRN2at6TensorES1_S1_S1_S1_S1_llllb

<!-- gh-comment-id:2844141204 --> @mj520 commented on GitHub (May 1, 2025): python ./ktransformers/local_chat.py \ > --model_path /root/DeepSeek-R1 \ > --gguf_path /root/DeepSeek-R1-UD-IQ1_S \ > --max_new_tokens 2048 \ > --force_think true Traceback (most recent call last): File "/root/ktransformers/./ktransformers/local_chat.py", line 25, in <module> from ktransformers.optimize.optimize import optimize_and_load_gguf File "/root/ktransformers/./ktransformers/optimize/optimize.py", line 15, in <module> from ktransformers.util.custom_gguf import GGUFLoader, translate_name_to_gguf File "/root/ktransformers/./ktransformers/util/custom_gguf.py", line 27, in <module> import KTransformersOps ImportError: /root/anaconda3/envs/ktransformers/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so: undefined symbol: _Z16gptq_marlin_gemmRN2at6TensorES1_S1_S1_S1_S1_llllb
bsakaguchi commented 2026-10-29 14:42:24 +00:00
Author
Owner
Copy link

@azjam78910 commented on GitHub (May 7, 2025):

Traceback (most recent call last):
File "/home/rocm/ktransformers/ktransformers/local_chat.py", line 25, in
from ktransformers.optimize.optimize import optimize_and_load_gguf
File "/home/rocm/ktransformers/ktransformers/optimize/optimize.py", line 15, in
from ktransformers.util.custom_gguf import GGUFLoader, translate_name_to_gguf
File "/home/rocm/ktransformers/ktransformers/util/custom_gguf.py", line 27, in
import KTransformersOps
ImportError: /home/rocm/anaconda3/envs/ktransformers/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so: undefined symbol: _Z16gptq_marlin_gemmRN2at6TensorES1_S1_S1_S1_S1_llllb , I have same error .. I also did this "comment all gptq_marlin line in setup.py."

<!-- gh-comment-id:2857143836 --> @azjam78910 commented on GitHub (May 7, 2025): Traceback (most recent call last): File "/home/rocm/ktransformers/ktransformers/local_chat.py", line 25, in <module> from ktransformers.optimize.optimize import optimize_and_load_gguf File "/home/rocm/ktransformers/ktransformers/optimize/optimize.py", line 15, in <module> from ktransformers.util.custom_gguf import GGUFLoader, translate_name_to_gguf File "/home/rocm/ktransformers/ktransformers/util/custom_gguf.py", line 27, in <module> import KTransformersOps ImportError: /home/rocm/anaconda3/envs/ktransformers/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so: undefined symbol: _Z16gptq_marlin_gemmRN2at6TensorES1_S1_S1_S1_S1_llllb , I have same error .. I also did this "comment all gptq_marlin line in setup.py."
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#648
No description provided.