mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #634] 无法开启numa，内存足够时，仍然只占用380GB内存，均匀的分散在两个numa node，如何使其将全部内存在numa节点间复制一份？使内存占用达到760GB以上 #471
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#471
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @chi2liu on GitHub (Feb 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/634
开启了numa，编译最新分支的源代码后启动
启动命令如下
python ./ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path DeepSeek-R1-Q4_K_M --cpu_infer 47 --max_new_tokens 1000 --optimize_rule_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml内存总占用380GB，占用如下，均匀的分布在两个numa节点，而不是复制一份
机器配置：
48物理核双通路CPU，1360GB双通路内存，1张A100-80GB
processor : 47
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) CPU @ 2.20GHz
stepping : 7
microcode : 0xffffffff
cpu MHz : 2200.218
cache size : 39424 KB
physical id : 1
siblings : 24
core id : 23
cpu cores : 24
apicid : 110
initial apicid : 110
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
bugs : spectre_v1 spectre_v2 spec_store_bypass swapgs taa mmio_stale_data retbleed eibrs_pbrsb bhi
bogomips : 4400.43
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
MemTotal: 1403646380 kB
MemFree: 594507192 kB
MemAvailable: 1391112312 kB
Buffers: 170512 kB
Cached: 792418808 kB
SwapCached: 0 kB
Active: 396242196 kB
Inactive: 396562128 kB
Active(anon): 225172 kB
Inactive(anon): 0 kB
Active(file): 396017024 kB
Inactive(file): 396562128 kB
Unevictable: 27832 kB
Mlocked: 27832 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Zswap: 0 kB
Zswapped: 0 kB
Dirty: 112 kB
Writeback: 0 kB
AnonPages: 241948 kB
Mapped: 1094300 kB
Shmem: 1084 kB
KReclaimable: 12034072 kB
Slab: 13600168 kB
SReclaimable: 12034072 kB
SUnreclaim: 1566096 kB
KernelStack: 12832 kB
PageTables: 10708 kB
SecPageTables: 0 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 701823188 kB
Committed_AS: 1617136 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 642412 kB
VmallocChunk: 0 kB
Percpu: 42816 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
Unaccepted: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 1037112 kB
DirectMap2M: 68165632 kB
DirectMap1G: 1358954496 kB
@wkgcass commented on GitHub (Feb 24, 2025):
@chi2liu commented on GitHub (Feb 24, 2025):
make install_numa 这个命令在ktransformer中执行么？
跟我打开了 numa balancing有关系么？关闭numa balancing后，memory被分配到单台节点了。但是仍然不会复制。
@VinayaLee commented on GitHub (Feb 25, 2025):
是需要重新安装一个USE_NUMA版本，底层有些C++代码分支是需要重编译才能生效的
@yeungtuzi commented on GitHub (Feb 25, 2025):
我有同样的问题，现在试试
@yeungtuzi commented on GitHub (Feb 25, 2025):
操作如下，没有效果
rm -rf ktransformers/ktransformers_ext/build
rm -rf ktransformers/ktransformers_ext/cuda/build
rm -rf ktransformers/ktransformers_ext/cuda/dist
rm -rf ktransformers/ktransformers_ext/cuda/*.egg-info
apt install libnuma-dev
export USE_NUMA=1
make install_numa
然后运行，仍然照旧
Per-node process memory usage (in MBs) for PID 1062946 (ktransformers)
Node 0 Node 1 Total
--------------- --------------- ---------------
Huge 0.00 0.00 0.00
Heap 57.44 1171.15 1228.59
Stack 0.06 0.07 0.13
Private 141094.95 247136.80 388231.75
Total 141152.45 248308.02 389460.47
@yeungtuzi commented on GitHub (Feb 25, 2025):
build log:
CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/tmp/tmp4964nybc.build-lib/', '-DPYTHON_EXECUTABLE=/root/anaconda3/bin/python', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_CUDA=ON', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.2.1.post1+cu128torch26fancy']
Standard output: b'-- The C compiler identification is GNU 11.4.0\n-- The CXX compiler identification is GNU 11.4.0\n-- Detecting C compiler ABI info\n-- Detecting C compiler ABI info - done\n-- Check for working C compiler: /usr/bin/cc - skipped\n-- Detecting C compile features\n-- Detecting C compile features - done\n-- Detecting CXX compiler ABI info\n-- Detecting CXX compiler ABI info - done\n-- Check for working CXX compiler: /usr/bin/c++ - skipped\n-- Detecting CXX compile features\n-- Detecting CXX compile features - done\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- pybind11 v2.14.0 dev1\n-- Found PythonInterp: /root/anaconda3/bin/python (found suitable version "3.12.7", minimum required is "3.7") \n-- Found PythonLibs: /root/anaconda3/lib/libpython3.12.so\n-- Performing Test HAS_FLTO\n-- Performing Test HAS_FLTO - Success\n-- Found Git: /usr/bin/git (found version "2.34.1") \n-- Looking for pthread.h\n-- Looking for pthread.h - found\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD\n-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success\n-- Found Threads: TRUE \n-- Found OpenMP_C: -fopenmp (found version "4.5") \n-- Found OpenMP_CXX: -fopenmp (found version "4.5") \n-- Found OpenMP: TRUE (found version "4.5") \n-- OpenMP found\n-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with LLAMA_CCACHE=OFF\n-- CMAKE_SYSTEM_PROCESSOR: x86_64\n-- x86 detected\n-- Found CUDA: /usr/local/cuda (found version "12.8") \n-- ALL_SOURCES: /root/ktransformers/ktransformers/ktransformers_ext/ext_bindings.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/backend.cpp;/root/ktransformers/ktransformers/ktransformers_ext/cpu_backend/task_queue.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/linear.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/mlp.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/moe.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/llamafile/shared_mem_buffer.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/flags.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/iqk_mul_mat_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/sgemm.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp;/root/ktransformers/ktransformers/ktransformers_ext/../../third_party/llamafile/tinyblas_cpu_unsupported.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_attn.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_load_dump.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_read_write.cpp;/root/ktransformers/ktransformers/ktransformers_ext/operators/kvcache/kvcache_utils.cpp\n-- NUMA support is enabled\n-- NUMA library found: /usr/lib/x86_64-linux-gnu/libnuma.so - enabling NUMA support\n-- Configuring done\n-- Generating done\n-- Build files have been written to: /root/ktransformers/ktransformers/ktransformers_ext/build\n'
Standard error: b'\x1b[33mCMake Warning:\n Manually-specified variables were not used by the project:\n\n EXAMPLE_VERSION_INFO\n\n\x1b[0m\n'
building 'KTransformersOps' extension
creating /tmp/tmp1bjbftyb.build-temp/ktransformers/ktransformers_ext/cuda
creating /tmp/tmp1bjbftyb.build-temp/ktransformers/ktransformers_ext/cuda/custom_gguf
creating /tmp/tmp1bjbftyb.build-temp/ktransformers/ktransformers_ext/cuda/gptq_marlin
/root/anaconda3/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2059: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
@yeungtuzi commented on GitHub (Feb 25, 2025):
use
numactl --cpubind=0 --membind=0 ktransformers --gguf_path
also no luck.
Per-node process memory usage (in MBs) for PID 10218 (ktransformers)
Node 0 Node 1 Total
--------------- --------------- ---------------
Huge 0.00 0.00 0.00
Heap 1122.02 0.00 1122.02
Stack 0.13 0.00 0.13
Private 201466.04 186783.53 388249.57
Total 202588.19 186783.53 389371.71
@yeungtuzi commented on GitHub (Feb 26, 2025):
aftter today's update, it works but slower than before.
@wkgcass commented on GitHub (Feb 26, 2025):
@yeungtuzi 试试这个pr #685
@Atream commented on GitHub (Mar 15, 2025):
感谢反馈，我们会尽快尝试复现并解决此问题。
一些可能的检查：export USE_NUMA=1需要在编译前设置，同时编译前建议删除之前编译的结果以防cache导致没能成功重新编译。
@yeungtuzi commented on GitHub (Mar 18, 2025):
I think this had been fixed.