mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #629] 如何指定显卡运行 #465
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#465
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @wzj-ll on GitHub (Feb 24, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/629
我想让整个模型运行在cuda:1上，现在是在0上，我修改了所有位置为0的配置和代码之后，直接runtimeError了：
Traceback (most recent call last):
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/local_chat.py", line 187, in
fire.Fire(local_chat)
File "/home/ubuntu/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
component, remaining_args = _CallAndUpdateTrace(
^^^^^^^^^^^^^^^^^^^^
File "/home/ubuntu/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/local_chat.py", line 111, in local_chat
optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/optimize/optimize.py", line 129, in optimize_and_load_gguf
load_weights(module, gguf_loader)
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 86, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 88, in load_weights
module.load()
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 86, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 86, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 86, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 88, in load_weights
module.load()
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/operators/base_operator.py", line 63, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 86, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 84, in load_weights
load_cur_state_dict(module, gguf_loader, prefix)
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/utils.py", line 74, in load_cur_state_dict
weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/custom_gguf.py", line 332, in load_gguf_tensor
values = GGML_DEQUANTIZE_GPU[ggml_name](data, device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/k8s/deepseek/ktransformers/./ktransformers/util/custom_gguf.py", line 755, in dequantize_f32_gpu
res_gpu = torch.empty_like(res, device=device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device ordinal
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions各位有遇到过这个问题吗？求解答
@Microcarft commented on GitHub (Feb 24, 2025):
不妨换个思路，我是在lxc环境下运行的，可以方便的控制GPU实例以及numa等分配规则，还可以做更多精细化的控制。可以尝试一下
@moneyisallyouneed commented on GitHub (Feb 24, 2025):
直接CUDA_VISIBLE_DEVICES=1
@yileld commented on GitHub (Feb 24, 2025):
配置文件里的cuda:1是第二张卡的意思，而程序仍然会获取所有可用的显卡，你需要在命令行里增加CUDA_VISIBLE_DEVICES=1，然后配置文件仍然是cuda:0不变
@snailfrying commented on GitHub (Feb 25, 2025):
你可以直接配置环境试试：export CUDA_VISIBLE_DEVICES=1 # 在运行前设置
@ccp123456789 commented on GitHub (Mar 5, 2025):
不行，还是会启用第0张卡
@better-er commented on GitHub (Mar 20, 2025):
先运行export CUDA_VISIBLE_DEVICES=2，再启动后虽然输出都是cuda:0，但是只有cuda:0占用不到400M显存，cuda:2占用10G多，而且运行时使用率也是只有cuda:2有，Gold 6326+4090 1.58b速度4.75token/s。不过测试时输出一直在重复一个字，明天再测试一下性能