mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #683] 16G显存无法启动 #498
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#498
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @wxtt522 on GitHub (Feb 26, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/683
loading blk.60.ffn_norm.weight to cuda:0
loading output_norm.weight to cuda:0
loading output.weight to cuda:0
Traceback (most recent call last):
File "/mnt/pve/h3200/model/r1/./ktransformers/ktransformers/local_chat.py", line 187, in
fire.Fire(local_chat)
File "/mnt/pve/h3200/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/pve/h3200/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/mnt/pve/h3200/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/pve/h3200/model/r1/./ktransformers/ktransformers/local_chat.py", line 111, in local_chat
optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
File "/mnt/pve/h3200/model/r1/./ktransformers/ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf
load_weights(module, gguf_loader)
File "/mnt/pve/h3200/model/r1/./ktransformers/ktransformers/util/utils.py", line 86, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/mnt/pve/h3200/model/r1/./ktransformers/ktransformers/util/utils.py", line 84, in load_weights
load_cur_state_dict(module, gguf_loader, prefix)
File "/mnt/pve/h3200/model/r1/./ktransformers/ktransformers/util/utils.py", line 74, in load_cur_state_dict
weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.73 GiB. GPU 0 has a total capacity of 15.70 GiB of which 1.61 GiB is free. Including non-PyTorch memory, this process has 14.08 GiB memory in use. Of the allocated memory 13.24 GiB is allocated by PyTorch, and 725.79 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
(ktransformers) root@pve:~/model/r1# nvidia-smi && free -g
Wed Feb 26 14:44:42 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03 Driver Version: 550.144.03 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4060 Ti Off | 00000000:03:00.0 Off | N/A |
| 0% 30C P8 10W / 165W | 4MiB / 16380MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
total used free shared buff/cache available
Mem: 251 4 33 0 215 246
Swap: 0 0 0
(ktransformers) root@pve:~/model/r1# pip list|grep torch
ktransformers 0.2.1.post1+cu124torch24avx2
torch 2.4.0+cu124
看文档Q4只需要14G显存，但似乎测试都在用x090，我跑DeepSeek-R1-Q2_K_XS用256g内存ollama能跑，ktransformers跑不了吗？
@wxtt522 commented on GitHub (Feb 26, 2025):
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True 确实有效