kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #240] Has anyone tried to run on 4060ti 16G ? #144

New issue

Closed

opened 2026-10-29 14:34:54 +00:00 by bsakaguchi · 8 comments

bsakaguchi commented

2026-10-29 14:34:54 +00:00

Owner

Originally created by @wkgcass on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/240

The doc says vram usage: at least 14GB VRAM, may I deduce that a 4060ti 16G could also fulfill the tutorial requirements?
文档说“至少14GB显存”，我能否推断，4060ti 16G也可以满足教程的要求？

Has anyone already tried to play with 4060ti 16G yet?
目前有人试过4060ti 16G吗？

Originally created by @wkgcass on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/240 The doc says vram usage: `at least 14GB VRAM`, may I deduce that a `4060ti 16G` could also fulfill the tutorial requirements? 文档说“至少14GB显存”，我能否推断，`4060ti 16G`也可以满足教程的要求？ Has anyone already tried to play with `4060ti 16G` yet? 目前有人试过`4060ti 16G`吗？

bsakaguchi closed this issue

2026-10-29 14:34:54 +00:00

bsakaguchi commented

2026-10-29 14:34:55 +00:00

Author

Owner

@vc5409ftu commented on GitHub (Feb 16, 2025):

那内存要512

@vc5409ftu commented on GitHub (Feb 16, 2025): 那内存要512

bsakaguchi commented

2026-10-29 14:34:55 +00:00

Author

Owner

@wkgcass commented on GitHub (Feb 19, 2025):

@vc5409ftu 需要修改什么配置或者代码吗？按文档操作16G刚好加载不了。内存管够，随便用～

@wkgcass commented on GitHub (Feb 19, 2025): @vc5409ftu 需要修改什么配置或者代码吗？按文档操作16G刚好加载不了。内存管够，随便用～

bsakaguchi commented

2026-10-29 14:34:56 +00:00

Author

Owner

@wkgcass commented on GitHub (Feb 19, 2025):

运行起来了：

基于我pull到的最新提交 b4fb633991bae29a26a81dc486a0b1920ff18519

diff --git a/ktransformers/util/utils.py b/ktransformers/util/utils.py
index 7034ac9..c7d9aac 100644
--- a/ktransformers/util/utils.py
+++ b/ktransformers/util/utils.py
@@ -71,6 +71,7 @@ def load_cur_state_dict(module: nn.Module, gguf_loader: GGUFLoader, prefix: str
             device = get_device(translated_key[:translated_key.rfind(".")], gguf_loader.tensor_device_map)
             print(f"loading {translated_key} to {device}")
             # device = "cpu" if "embd" in translated_key else "cuda"
+            torch.cuda.empty_cache()
             weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype)
             set_param(module, name, weights)
             del weights

不过速度嘛。。。我再试试跨numa的吧，不知道为啥刚看内存不像是USE_NUMA=1生效的

@wkgcass commented on GitHub (Feb 19, 2025): 运行起来了：基于我pull到的最新提交 `b4fb633991bae29a26a81dc486a0b1920ff18519` ``` diff --git a/ktransformers/util/utils.py b/ktransformers/util/utils.py index 7034ac9..c7d9aac 100644 --- a/ktransformers/util/utils.py +++ b/ktransformers/util/utils.py @@ -71,6 +71,7 @@ def load_cur_state_dict(module: nn.Module, gguf_loader: GGUFLoader, prefix: str device = get_device(translated_key[:translated_key.rfind(".")], gguf_loader.tensor_device_map) print(f"loading {translated_key} to {device}") # device = "cpu" if "embd" in translated_key else "cuda" + torch.cuda.empty_cache() weights = gguf_loader.load_gguf_tensor(translated_key, device = device).to(dtype = target_dtype) set_param(module, name, weights) del weights ``` 不过速度嘛。。。我再试试跨numa的吧，不知道为啥刚看内存不像是USE_NUMA=1生效的

bsakaguchi commented

2026-10-29 14:34:56 +00:00

Author

Owner

@wkgcass commented on GitHub (Feb 19, 2025):

没事了，我忘了装libnuma-dev。。。

但是USE_NUMA=1版本比不带NUMA的要差很多。而ktransformers版本比纯cpu推理也要差很多。。。

@wkgcass commented on GitHub (Feb 19, 2025): 没事了，我忘了装libnuma-dev。。。但是`USE_NUMA=1`版本比不带NUMA的要差很多。而ktransformers版本比纯cpu推理也要差很多。。。

bsakaguchi commented

2026-10-29 14:34:56 +00:00

Author

Owner

@wkgcass commented on GitHub (Feb 21, 2025):

性能问题已解决。
线程数高于某个值时，性能会骤降至1token/s，猜测是调度问题或者软件bug。
而最高性能的线程数比较玄学，试了很多次试出来了。

平台：
9275f * 2 + (2 * 12 * 64g ddr5 6400MHz(运行于6000MHz下)) + 4060ti 16G
--max_new_tokens 8192，稳定12+token/s

@wkgcass commented on GitHub (Feb 21, 2025): 性能问题已解决。线程数高于某个值时，性能会骤降至1token/s，猜测是调度问题或者软件bug。而最高性能的线程数比较玄学，试了很多次试出来了。平台： `9275f * 2 + (2 * 12 * 64g ddr5 6400MHz(运行于6000MHz下)) + 4060ti 16G` `--max_new_tokens 8192`，稳定`12+token/s`

bsakaguchi commented

2026-10-29 14:34:56 +00:00

Author

Owner

@wkgcass commented on GitHub (Feb 21, 2025):

顺带提一下，我试下来的最高性能线程数是32，即每个cpu16线程，32-36都差不多，42线程开始直接从11掉到1
我是关闭超线程的，numa数配的是2个numa

@wkgcass commented on GitHub (Feb 21, 2025): 顺带提一下，我试下来的最高性能线程数是`32`，即每个cpu16线程，32-36都差不多，42线程开始直接从11掉到1 我是关闭超线程的，numa数配的是2个numa

bsakaguchi commented

2026-10-29 14:34:56 +00:00

Author

Owner

@zhengjx-ai commented on GitHub (Feb 23, 2025):

你的CPU是什么配置？参考一下呗

@zhengjx-ai commented on GitHub (Feb 23, 2025): 你的CPU是什么配置？参考一下呗

bsakaguchi commented

2026-10-29 14:34:56 +00:00

Author

Owner

@wkgcass commented on GitHub (Feb 24, 2025):