mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #419] 通过ktransformers --type transformers --model_path --port 10002 --web True的方式运行32B 在24G 的 4096上 提示显存不足 #285
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#285
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @andrewliu1345 on GitHub (Feb 17, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/419
ktransformers --type transformers --model_path /vol1/1000/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B --port 10002 --web True
2025-02-17 19:18:42,984 INFO /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-17 19:18:42,985 INFO /root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
Loading checkpoint shards: 12%|████████████████████▋ | 1/8 [00:03<00:23, 3.29s/it]
Traceback (most recent call last):
File "/root/miniconda3/envs/ktransformers/bin/ktransformers", line 8, in
sys.exit(main())
^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/main.py", line 114, in main
create_interface(config=cfg, default_args=cfg)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
GlobalInterface.interface = BackendInterface(default_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 137, in init
self.model = AutoModelForCausalLM.from_pretrained(args.model_dir, device_map=args.device, use_safetensors=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
return model_class.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3916, in from_pretrained
) = cls._load_pretrained_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/modeling_utils.py", line 4390, in _load_pretrained_model
new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/transformers/modeling_utils.py", line 936, in _load_state_dict_into_meta_model
set_module_tensor_to_device(model, param_name, param_device, **set_module_kwargs)
File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/accelerate/utils/modeling.py", line 330, in set_module_tensor_to_device
new_value = value.to(device)
^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 540.00 MiB. GPU 0 has a total capacity of 23.55 GiB of which 276.69 MiB is free. Including non-PyTorch memory, this process has 23.27 GiB memory in use. Of the allocated memory 22.88 GiB is allocated by PyTorch, and 11.29 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
@ShinoharaHare commented on GitHub (Feb 17, 2025):
半精度的模型理論上要約 32x2 = 64G 以上的 VRAM 才夠，你應該要找量化過的版本
@andrewliu1345 commented on GitHub (Feb 18, 2025):
嗯嗯~
@ErvinXie commented on GitHub (Dec 11, 2025):
This issue has been automatically closed due to inactivity for 60 days.
If you believe this issue is still relevant, please feel free to reopen it with additional information or context.
Thank you for your contribution to KTransformers!
此 issue 因超过 60 天无活动而被自动关闭。
如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。
感谢您对 KTransformers 的贡献！