mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #225] ubuntu 4080 显卡部署 deepseek-r1模型，报错RuntimeError: Tensor on device meta is not on the expected device cuda:0! #138
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#138
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @qingqinghu1026 on GitHub (Feb 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/225
python ../tools/ktransformers/ktransformers/local_chat.py --model_path ./DeepSeek-R1-Distill-Qwen-7B/ --gguf_path ./deepseek-r1-gguf/ --optimize_rule_path ./ktransformers-yaml/DeepSeek-r1-1.yaml
yaml配置文件
`- match:
class: ktransformers.models.modeling_qwen2_moe.Qwen2MoeRotaryEmbedding
replace:
class: ktransformers.operators.RoPE.YarnRotaryEmbedding
kwargs:
generate_device: "cuda:0"
prefill_device: "cuda:0"
match:
name: "^model\.layers\.(?!.self_attn\.kv_b_proj).$" # regular expression
class: torch.nn.Linear # only match modules matching name and class simultaneously
replace:
class: ktransformers.operators.linear.KTransformersLinear # optimized Kernel on quantized data types
kwargs:
generate_device: "cuda:0"
prefill_device: "cuda:0"
generate_op: "KLinearMarlin"
prefill_op: "KLinearTorch"
match:
name: "^model\.layers\..*\.mlp\.experts$"
replace:
class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert parallelism
kwargs:
prefill_device: "cuda:0"
prefill_op: "KExpertsTorch"
generate_device: "cpu"
generate_op: "KExpertsCPU"
out_device: "cuda:0"
recursive: False # don't pip install --upgrade transformersqrecursively inject submodules of this module
match:
name: "^model.embed_tokens"
replace:
class: "default"
kwargs:
generate_device: "cpu"
prefill_device: "cpu"`
部分运行日志：
loading token_embd.weight to cpu
loading blk.0.attn_norm.weight to cuda:0
loading blk.0.ffn_norm.weight to cuda:0
loading blk.1.attn_norm.weight to cuda:0
loading blk.1.ffn_norm.weight to cuda:0
loading blk.2.attn_norm.weight to cuda:0
loading blk.2.ffn_norm.weight to cuda:0
loading blk.3.attn_norm.weight to cuda:0
loading blk.3.ffn_norm.weight to cuda:0
loading blk.4.attn_norm.weight to cuda:0
loading blk.4.ffn_norm.weight to cuda:0
loading blk.5.attn_norm.weight to cuda:0
loading blk.5.ffn_norm.weight to cuda:0
loading blk.6.attn_norm.weight to cuda:0
loading blk.6.ffn_norm.weight to cuda:0
loading blk.7.attn_norm.weight to cuda:0
loading blk.7.ffn_norm.weight to cuda:0
loading blk.8.attn_norm.weight to cuda:0
loading blk.8.ffn_norm.weight to cuda:0
loading blk.9.attn_norm.weight to cuda:0
loading blk.9.ffn_norm.weight to cuda:0
loading blk.10.attn_norm.weight to cuda:0
loading blk.10.ffn_norm.weight to cuda:0
loading blk.11.attn_norm.weight to cuda:0
loading blk.11.ffn_norm.weight to cuda:0
loading blk.12.attn_norm.weight to cuda:0
loading blk.12.ffn_norm.weight to cuda:0
loading blk.13.attn_norm.weight to cuda:0
loading blk.13.ffn_norm.weight to cuda:0
loading blk.14.attn_norm.weight to cuda:0
loading blk.14.ffn_norm.weight to cuda:0
loading blk.15.attn_norm.weight to cuda:0
loading blk.15.ffn_norm.weight to cuda:0
loading blk.16.attn_norm.weight to cuda:0
loading blk.16.ffn_norm.weight to cuda:0
loading blk.17.attn_norm.weight to cuda:0
loading blk.17.ffn_norm.weight to cuda:0
loading blk.18.attn_norm.weight to cuda:0
loading blk.18.ffn_norm.weight to cuda:0
loading blk.19.attn_norm.weight to cuda:0
loading blk.19.ffn_norm.weight to cuda:0
loading blk.20.attn_norm.weight to cuda:0
loading blk.20.ffn_norm.weight to cuda:0
loading blk.21.attn_norm.weight to cuda:0
loading blk.21.ffn_norm.weight to cuda:0
loading blk.22.attn_norm.weight to cuda:0
loading blk.22.ffn_norm.weight to cuda:0
loading blk.23.attn_norm.weight to cuda:0
loading blk.23.ffn_norm.weight to cuda:0
loading blk.24.attn_norm.weight to cuda:0
loading blk.24.ffn_norm.weight to cuda:0
loading blk.25.attn_norm.weight to cuda:0
loading blk.25.ffn_norm.weight to cuda:0
loading blk.26.attn_norm.weight to cuda:0
loading blk.26.ffn_norm.weight to cuda:0
loading blk.27.attn_norm.weight to cuda:0
loading blk.27.ffn_norm.weight to cuda:0
loading output_norm.weight to cuda:0
loading output.weight to cuda:0
Chat:
@qingqinghu1026 commented on GitHub (Feb 13, 2025):
运行日志显示是在cuda上，为什么报错tensor在meta上呢？我的设备是ubuntu系统，显卡是两张RTX4080
@jhscut commented on GitHub (Feb 14, 2025):
可以分享一下R1的配置文件吗
@eeeeeeeason commented on GitHub (Feb 23, 2025):
后来跑起来了吗？我也是只有一条4080，想看看是不是买多一条能工作
@ErvinXie commented on GitHub (Dec 11, 2025):
This issue has been automatically closed due to inactivity for 60 days.
If you believe this issue is still relevant, please feel free to reopen it with additional information or context.
Thank you for your contribution to KTransformers!
此 issue 因超过 60 天无活动而被自动关闭。
如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。
感谢您对 KTransformers 的贡献！