mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #712] Does the official plan support Turing Nvidia GPU ？ #521
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#521
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jason-ji-227 on GitHub (Feb 27, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/712
Firstly, I am pleased to see the author working so hard to maintain this project v0.2.2rc1, which was just released two days ago, has been updated and is now being used normally on Ampere architecture's Nvidia GPU. However, I believe that the original intention of the ktransformer project is to enable individual enthusiasts to experience private deployment of local large model projects at the lowest cost. Therefore, there is a small request for the author to ensure compatibility in the lower left corner of future versions, which is whether it is possible to officially update that support the Turing Nvidia GPU architecture. I have also tried to modify some of the inference code according to issues #374 and #452 , but when KTransformers started, the video memory load exceeded 22GB, causing OOM in my Nvidia Quadro RTX 6000 environment and rendering it unusable.
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --optimize_rule_path /workspace/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --gguf_path /workspace/download/DeepSeek-R1-GGUF-Q2_K_XS ...... ...... loading blk.41.attn_kv_a_norm.weight to cuda:0 loading blk.41.attn_kv_b.weight to cuda:0 Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/root/workspace/ktransformers/ktransformers/local_chat.py", line 183, in <module> fire.Fire(local_chat) File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/root/workspace/ktransformers/ktransformers/local_chat.py", line 110, in local_chat optimize_and_load_gguf(model, optimize_config_path, gguf_path, config) File "/root/workspace/ktransformers/ktransformers/optimize/optimize.py", line 131, in optimize_and_load_gguf load_weights(module, gguf_loader) File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 109, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 111, in load_weights module.load() File "/root/workspace/ktransformers/ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 109, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 109, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 109, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 111, in load_weights module.load() File "/root/workspace/ktransformers/ktransformers/operators/base_operator.py", line 63, in load utils.load_weights(child, self.gguf_loader, self.key+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 109, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 109, in load_weights load_weights(child, gguf_loader, prefix+name+".") File "/root/workspace/ktransformers/ktransformers/util/utils.py", line 111, in load_weights module.load() File "/root/workspace/ktransformers/ktransformers/operators/linear.py", line 522, in load self.generate_linear.load(w=w) File "/root/workspace/ktransformers/ktransformers/operators/linear.py", line 149, in load if w is None: w = self.load_weight(device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/workspace/ktransformers/ktransformers/operators/linear.py", line 97, in load_weight tensors = self.load_multi(key, ["weight"], device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/workspace/ktransformers/ktransformers/operators/linear.py", line 107, in load_multi tensors[k] = self.gguf_loader.load_gguf_tensor(key + "." + k, device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/workspace/ktransformers/ktransformers/util/custom_gguf.py", line 368, in load_gguf_tensor cur_values = GGML_DEQUANTIZE_GPU[ggml_name](data[blocks_begin*block_size : blocks_end*block_size], device, target_dtype) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/workspace/ktransformers/ktransformers/util/custom_gguf.py", line 494, in dequantize_q2_k_gpu return KTransformersOps.dequantize_q2_k(c_pointer, data.size, block_size, ele_per_blk, device, target_dtype) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 22.15 GiB of which 7.19 MiB is free.Finally, I wish the author a smooth work and happy life, and I wish the KTransformers project continued success, allowing more enthusiasts with limited funds to experience the joy of deploying large-scale models privately.
@Azure-Tang commented on GitHub (Feb 27, 2025):
Thank you for your interest in KTransformers and for raising this important question about Turing architecture GPU support. We appreciate your engagement with our project.
Technical Context:
The current limitation with Turing-series GPUs (e.g., NVIDIA T4/Turing GPUs) stems from their incompatibility with the Marlin operator optimization. This requires fallback to standard PyTorch operators, which increases VRAM consumption by approximately 400% compared to Ampere/Ada Lovelace architectures.
Implementation Challenges:
While integrating alternative quantization operators like GGML could theoretically alleviate this, we face significant technical hurdles:
So i'm sorry that this may not be achieved in the near future. Thank you for understanding, and please don't hesitate to reach out with other technical inquiries.
Wish you a good day.
@ErvinXie commented on GitHub (Dec 11, 2025):
This issue has been automatically closed due to inactivity for 60 days.
If you believe this issue is still relevant, please feel free to reopen it with additional information or context.
Thank you for your contribution to KTransformers!
此 issue 因超过 60 天无活动而被自动关闭。
如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。
感谢您对 KTransformers 的贡献！